1

Sorry if this might be off topic, but I’d like to have other sysadmins opinions on that.

We are planning to upgrade our backup solution because the one we have right now is somewhat hard to manage and we’re at the point where we are almost out of storage space…

What we have right now:

  • two HP DDS3 (c1537a) SCSI tapes (12/24Go) in linux boxes connected to Adaptec AHA-2930cu SCSI adapters

  • two Seagate/Quantum STT20000A travan drives (10/20Go)

The two HP drives are controlled through Amanda on linux servers, the two Seagate drives are controlled by NTBackup (I know, I know… not really good, but works for now)

We’re a relatively small company so we don’t want to spend money on very expensive hardware, but we don’t want to buy the cheapest solution if it doesn’t work either.

Currently, I have a hard time fitting all the data on the 4 tapes.

What would be your suggestions (hardware, software) for new solution?

3 Spice ups
2

We did bite the bullet (upon my recommendation) and went with LTO-3 from a Quantum DLT IV. It was overkill for what we needed but I wanted to ensure that we had future compatibility as much as possible as well as LTO is a multi-vendor standard. This way I’m not locked to a vendor for my hardware. You may want to look at lower capacity systems, however, I think you should go the highest you can to carry your company’s data and future business needs forward. I figured there was no sense in using LTO-2 when we could potentially outgrow it in three or four years. Your needs may differ but I would suggest LTO at the moment if you’re still looking at tape.

3

For sure, tapes are still an option, we’re opened to other solutions too.

I’ll have a look at the LTO, and I’m looking for other recommendations too.

Thanks Mainstrike

4

Another company I work for resells for Mozy ( Domain Registered at Safenames ) which is an online backup storage company. I’m sure your (business class) Internet service provider may have a similar offering. And with online storage, you get off-site storage for disaster recovery. I don’t know if you have certain laws you need to comply and if this is a viable solution. I do believe the Mozy backup software sends block changes rather than full files to quicken your transfer. I don’t see Linux clients for Mozy.

5

I would suggest using a few external harddrives and backing up the data on to them. The larger and faster your tape drives get the more unreliable they become overtime, not to mention the cost of the tape drives. Removable harddrives don’t cost much more then large media tapes. They are more reliable in the long run can be recycled to other uses if you want to.

But the key to any backup solution is to get that backup off site. Series of tapes, or several portable harddrives are the key. The only other real alternative is an offsite backup provider, but they are going to be much more expensive.

6

I would like to add my own dilemma into this post. I currently have been asked by my boss to come up with a viable solution to our ever growing problem with backups.

We are currently using an Ultrium 460 drive with tapes going at a compression rate of 400 GB. This allows us to backup approximately 225 GB per tape. We have about 430 gigs of space to backup and we swap tapes every night as well as what we backup. We have two servers, each carrying a capacity of about 200 GB.

So our backup schedule is like this…
Mon: Server1 + Exchange
Tue: Server2 + Exchange
Wed: Server1 + Exchange
Thu: Server2 + Exchange
Fri: Server1/2 + Exchange (we alternate every week)
End of Month (We have 24 tapes for this… one for each server + exchange) - is ran usually on a Saturday or in place of two regular backup nights.

We are looking at purchasing another server for our new ERP system and with that we are figuring that our capacity will be running out soon. In fact, we’ve already had that problem… I had to remove about 20 gigs from the backup queue (although it wasn’t important so it didn’t need to be backed up). We’re constantly pushing our Tape backup limits and I personally would like to see us be able to go six months without a problem with our backups which includes either errors or space-issues.

We use BackupExec 10d currently but I am not impressed with it. Is it normal to go a week or so without having any problems with backups and then have a few weeks of problems every night? It’s really become a game… As I’m typing in my username and password to get into the system in the mornings I am guessing in my head whether or not BackupExec was successful. Most of the time it can get Exchange (except this week), but it seems to have random problems with the servers. I would think it was my fault if the error was the same every time, but unfortunately it’s random and if I give it a few days then it seems to go back to normal or if I pursue it, it leads usually to no where.

My boss would like to be able to have two tape backups going… keep our current tape backup system and get a bigger one. Then we could use this smaller one for the new server and the bigger one to backup the other two servers including exchange.

My addition to this would be that I want something that we won’t have to purchase anything extra for a very very long time.

What would you guys possibly recommend in my situation? What do you guys that are in my current setup doing right now? We have 79 users and growing every few weeks.

Oh, and one more mention… I am curious what you think on backing up desktops. It would be great to be able to backup all the desktops in a folder or something and then back that up to a tape. I don’t know how hugely beneficial this could be but we all know that while everyone is supposed to use the server to put important files, that does not happen… Does anyone with over fifty computers even bother with backing up your workstations and if so, how do you do it?

Thank you!

7

If backing up desktops is important you have a few options. 1st most Enterprise Backup software has a client that will install to a desktop and pull the data off it. You can also use built in windows software (ntbackup) to backup the desktop, or you could (with some fiddling) use 3rd party tools, rsync for example to backup the desktops. However I suggest skipping that headache and chance for error. Invest in a file server (if you don’t have one) and use folder redirection on every desktop so that their ‘my documents’ folder is actually located on the file server. That way their documents are guaranteed to be backed up.

As for the rest. Tape drive technolgy gets bigger and faster and always gets more unreliable the more data they try to compress onto a tape. If you are fine with the risks then you’re at a size now where you should be going to an LTO, which will run you around $10,000 for the drive plus 2 weeks of tapes.

Alternately you could use an external harddrive backup method like I suggested in a previous post in this thread. The other option is offsite backup. The costs very greatly and depending on how much data, what kind of data and how frequent you want to backup will run you from about $100 to $5000 a month.

8

What could something like this: http://www.superwarehouse.com/HP_StorageWorks_MSL2024_Ultrium_1840_Tape_Library/AJ033A/ps/1504246 do for us? We want to stick with HP and this is a StorageWorks that mentions something about multi-terabyte backups…

Maybe I didn’t understand it much but is this something would backup everything to this item and then could be transferred to tapes? Maybe I don’t understand LTO tape backup methods?

I’m not sure how the boss would like having things backed up through a remote backup facility but it is definitely an option I have sitting here in my list.

As for an external HDD method… Would you recommended the fastest, largest external hard drive out there to be reliable? What do you suppose… we get something in the lines of a 500 GB external hard drive and then just get the amount we would like? Or are there hard drives out there (say, 1 TB) that would be more beneficial?

Onto backing up desktops… we do already have a file server set up for the My Documents destination but my boss seems set to, within reasonable pricing, try and back up most or all of our 79+ computers. What would your take be? Should we go ahead and look into this, possibly using the options provided, or would there be a waste of time in this and if so, how am I to get this message across? I think it wouldn’t be a bad idea because there are always things, such as throwing stuff on the desktop or what-not that would be lost due to a crash. And also we do have a computer of databases for UPS and our credit card system that is located on workstations which would be beneficial to back up.

Thanks for the feedback!

9

With some more research I have found a solution that holds promise, although it is a wee bit expensive.

Sonicwall CDP http://www.nobackuptapes.com/Home/tabid/83/Default.aspx

Just a few questions…
1.) Has anyone used this?
2.) What are your overall thoughts on it?
3.) Does it work well? As in can you restore quickly as they say?
4.) What kinds of problems do you have?
5.) Do you know where the offsite backups are? Is it possible to do an ‘offsite’ backup of your own with a separate unit? How about backup up this information from the unit to either a tape or DVDs? (For example… if there was hugely important stuff we wanted to move out of our building but wanted to keep with one of our top employees?)
6.) Anything else I missed here to ask that you would want to offer your inquiry on?

I’ve been looking around and it looks hopeful. As I said, it seems relatively cheap but it does definitely appear more expensive than my boss is probably willing to pay. I would need to find a way to convince him that just adding more tapes and a new tape drive would be a bad idea.

Anyway… thanks for the responses?

10

Ok lots to discuss so I’ll try and break it down.

First the device you linked to is a tape archive device. Which means it’s basically a huge tape drive that uses multiple tapes, in rotation internally to store the information. So it replaces the human need to swap a tape if one tape is full and then moving onto the next one. Only this supports 24 tapes. I don’t know how much data you are moving around, but I would guess that’s extreme overkill as it supports u pto 38TB (3,800gb) of data that’s a lot. I don’t know your company but I’d be surprised if you had 1TB of data to back up unless you are doing a lot of graphic/cad work.

As far as LTO is concerned LTO is just another form of Tape Drive. It’s very fast and very dense so it’s a fantastic medium for storing a lot of data. The downside to any tape drive is that the tapes, the more dense they are the more susceptible they are to data loss from climate changes or physical damage. For example an LTO tape can be damaged (According to Backup Companies) from the climate change of going from work, into a cold car, heating up on the way home and then leaving it at home and back again. Those climate changes could wreck it. It’s also potentially damaged from any drop over I (don’t quote me) 2". So in terms of what LTO is capable of, they are great. You just have to baby them. Because remember a backup solution is only good if you can retrieve the data. One other consideration. The density of LTO is such that you can run into situations where (over time) only the LTO drive that is writing to the tapes is the one that can read them because of slight alignment issues that develop. So they aren’t recommended as a reliable disaster recovery medium if that’s important.

External HDD method. By the biggest drives you need for now with some room for growth. Drive prices are always coming down so there is little advantage to going big when it comes to harddrive space.

Now desktop backup. What I suggest is this. I’m assuming you have a file server that has a network share for people to share data, but their actual ‘My Documents’ folder is still stored locally on their computers. What we do is redirect the ‘My Documents’ folder to a network drive. This was done globally throughout the company by making a very simple Group Policy Object change on the Domain Controller. So now if you right click on ‘My Documents’ and check properties instead of saying ‘C:\Documents and Settings\username\My Documents’ for example. It now says ‘\172.22.23.3\homes\My Documents’ The same with their desktop folder. On our file server there is a seperate directory for every user by their username. The directories are all private but the key is they are all on the server. That way I don’t have to worry about backing up the data on PC’s it’s already on the server, the server always has live data and the server gets backed up regularly. The only exception to this are laptop users which we don’t use folder redirection on because the offline file synchronization in windows is laughable. As for how much space you need, I don’t know. Our file server has 300gb of storage and it’s only got 20% full. But plenty of room for expansion as you can see.

I’m personally not familiar with the Sonicwall solution, though I’ve researched other similar solutions and they are solid solutions that work well and are effectively 100% reliable. They do cost more then any local backup solution guaranteed. But keep in mind that your backup solution is like any other insurance policy for your company. It’s money being spent in hopes that nothing will go wrong. But if it does you damn well bet you’ll be thankful you have it. So in terms of price it really comes down to a simple discussion with your boss/owner. If all your digital data was to be utterly destroyed today. How much money would that cost him in lost business, downtime etc…etc…etc… $1000 a month, could be easily $1000 a day. I don’t know the answer for your business. But that’s really what it comes down to when it comes down to backup solutions. You need something that is effective (collects all the data in a reliable manor and can retrieve it) and reliable (the data needs to be accessible should the worst case happen).

It may sound like a sales pitch. But that’s just the way it is.

If you want something that is fairly in expensive I’d personally say External HDD’s over tape and then rotate those drives every year or 2 out of circulation because they will be moved around physically and are under wear and tare. If you want something that is just there always working and very secure then remote, offsite backup is king. But it’s going to cost the $$$

11

1.) As mentioned previously we do already backup the My Documents drive but we do not Backup Desktops… I’m not 100% sure why not the desktops but that’s just how it is at the moment.
2.) That Sonicwall CDP provides a unit on-site as well as an OPTIONAL off-site item. The problem with the Sonicwall CDP is that their largest capacity is a unit that allows for 1.2 TB compressed (600GB native) and is 8 grand… the 8 grand is pushy, but not a huge deal… the problem with it is I know that at our current rate of growth we will exceed that compression rate in 2 - 4 years. If we were to backup everything on our two servers, which is what the boss would like to do, and according to their DRP we would have to in order to do a quick restore of entire servers, then we would be looking at around 600GB just for the servers. On top of the 600 GB my boss, once again, still would like to look at desktops… would this require buying two SonicWall units and if so, how well do they play nice together? I have sent an email to SonicWALL addressing this issue but maybe someone else has dealt with it?
3.) You say external HDDs… That sounds great and everything but first of all… what do you suppose… that we replace all of our tapes with HDDs? Meaning we would have the 8 tapes for our weekly backups (4 for Mon-Thu and 4 for Fri,wk1-Fri,wk4) and 12 Monthly backups? And if so, what types of External hard drives would you recommend? Noting the 600 GB and the want for backing up Desktops, how big of HDDs would you think be necessary for our constantly growing company?

We have three things needing large spaces… our huge library of DWG and solidworks files as we are a manufacturing company. The other two items is customer-related information (the ERP system as well as tons of different PDF files and the likes) and our Exchange server. In the end this does equal to about 600GB and currently engineering is undergoing a project which revamps pretty much all of our current CAD files and of course they will want to keep all the old drawings for legacy support.

I truly appreciate your input on this matter Groffer.

12

Based on that amount of data and the potential sensitivity of data I’d suggest not to mess around with external HDD’s. They do work, I’ve employed them in the past on many occasion but they were for companies under 500gb of data and where digital medium wasn’t as essential to keep the company going. You have some significant information there and I would imagine any loss is not acceptable. I’d urge you to get in touch with some firms in your local that specialize in that sort of service. I think you need to settle down with them to get some real facts and figures together in order to make a better informed decision into this mater. IBM business solutions provides that level of service (not cheap) many major telcoms also provide it (verying degrees) then there are individual data vault style companies.

I would no longer suggest going the tape method as flipping tapes for a single day becomes less practical and drastically increase the chance of data loss. You need to start thinking big here and beyond my expertise (other then me saying GO BIG). Congrats, you should be happy, you are hitting the big time :slight_smile:

13

Haha thanks a lot. I’m certain this will take several weeks to several months to figure out but based on the idea that we are in a very crucial point in time, (we are in preparations to implement a new, much more expansive, ERP system as well as in the middle of moving most of our AutoCAD programs into Solidworks Programs), it is apparent that the sooner we can do this, the better… I believe I know more about backing stuff up from today tenfold more than I have ever learned before!

I will nix the External hard drive idea and am trying to find viable solutions to remove the Tape drives… my boss is currently set on using tape drives and just simply getting a bigger tape drive to suffice. I’m trying to gather all the information I can so I can go into his office and do my best at convincing him why we need to spend the extra money to protect what we need to back up.

Thanks again for all your help on this matter. I will be certain to report back what I learn… maybe after all my information is gathered I will create a project documentation based on this information. Speaking of which, I wonder if we already have any projects relating to backups… off to search

14

Groffer, based on your inputs previously I’m an curious to know what you think of the following article: http://edge.networkworld.com/whitepapers/nww/pdf/SonyC601SB.pdf

If this is true, then we are currently doing the right thing, as we use Suresync to replicate the data here for short-term and then we use tape for our long-term. I think our biggest issue is the idea of ‘what’ to backup for the long-term. Also the idea of getting a bigger, more reliable tape drive and tapes should be looked at as well. We have had the same tapes right now for about 2 years and while a lot of them have only been used twice, a lot of them get used once a week.

Hmm… I suppose I’m just thinking out loud but maybe we do need to get another tape drive and our monthly tape backups need to be backed up on a tape and never overwritten… I don’t know. I’ve still got more researching to do but I’ve been at this all day yesterday and today so far… pain in the ass!

15

keep it simple, get a few copys of backup exec system recovery and a
a few lacie drives

backup exec 80 dollars
lacie nas 1tb - 400 dollars.
If your a small company your not going to need tapes.

for your exchange server or mission critical servers
have an additional nas taken off site once a week,

its cheap and can restore your systems back to the minute in under an hour.

16

We already have Backup Exec but we’re upgrading to 11d (we had 10d)…

My boss and I have decided the best thing for us to do is go ahead and continue to use tapes, at least for the next five or so years. The reason is because we already do replication between each server (which, btw, I suppose we’ll have to be a third license), using Suresync. So that would be considered our short-term backup. Then we use the tapes for our more long-term backup. Unfortunately our current way of doing it is as such…

We have 12 monthly tapes, which are obviously overwritten once a year

We have 8 weekly tapes, 1 ea. for Monday - Thursday and 4 for Friday… so four get overwritten once a week and the other four gets overwritten once a month.

So that means anything we want to keep, even “Archive” will need to remain on the server throughout time. I’m thinking maybe having an archive server for the legacy information that is there purely for old records-sake (Y’know… the really old drawings and terminated employees, etc) and then we’ll back that up on DVD’s / or get another handful of tapes and back it up once a month or so. I’m not certain what we’ll do with long-term information yet, but I’m sure it’ll be in the works in 2008.

So for those who are curious… here is our plan:
1.) Keep our current LTO-2 Ultrium 460 which has a 200native/400compressed rate… we’ll use that for the new server which will host our ERP system. That costs nothing…
2.) Purchase (we already have) a new LTO-3 Ultrium 960 which has 400native/800compressed rate… We’ll use this for our two old servers which also hosts exchange. This costs a grand total of about $3,000.
3.) Upgrade our BE licensing… I don’t imagine that will cost more than a few hundred… unless the boss decides to purchase the extra agents and even then I’m not sure if one license will suffice or if we have to buy three licenses for three servers.
4.) As for the desktops I plan to locate a free GPL Licensed backup software and try it out on a few of the more important machines and with time I will probably have all of the machines backing up to folder on one of the servers so that we at least have a copy… we will probably not add this to our tape queue however due to the lack of importance. It’ll be there only for Disaster Recovery so that I can pull any important information from, say, the desktop or so that I can know how that computer was configured (as in what was installed on it even if the computer has crashed).

Feel free to comment… hopefully that may help someone. But yes, to get a decent Tape backup solution that allows you to back up 400GB Native on one tape (800 Compressed) and runs about 576GB/hour and also to get a good amount of tapes (20pk) you will be looking at around 3,000 if you do the research… if you buy direct (like from HP), our cheapest was going to be about 3,500 and that’s the lowest they would go with only a 5pk of tapes. Without trying to talk them down the grand total would’ve been 4,500 (again with only 5pk of tapes)… so looking for this stuff elsewhere could save you potentially thousands.

17

That’s a pretty solid solution overall. Just make sure that your periodically test to see if you can actually recover from your tapes.

As for backing up your desktops. We used to use rsync (gpl option) and every desktop would backup through rsync at login and then there was a scheduled event that ran anytime the machine was idle for an hour that would backup the system again. This works ok. But I found that it just wasn’t efficient enough. Which is why I recommend using folder redirection for their documents folder and their desktop. That way you don’t have to run any software on the individual machines and you know that their documents are always going to be backed up since they only exist on the file server. It really makes things much, much easier to manage.

Not to mention if you want to upgrade a new machine you don’t have to deal with backing up any of their stuff first. Create the new machine and as soon as they sign in they automatically get their desktops and documents back.

18

As i have mentioned we do currently redirect their My Documents but I might go ahead and talk with my boss about redirecting their Desktop as well and then you’re right…that would pretty much take care of it. Only backup a few machines then, as in those who have UPS as their local database.

But I’m curious… wouldn’t it be a pain in the ass to have shortcuts to software programs that may not be installed on a particular computer or, for some reason, installed in a different directory?

As for the tapes you are absolutely right. If we had a better budget I would’ve strongly encouraged my boss a different route but it was hard to beat 3 grand. Yes we could’ve done the external HDDs but we want to be able to take some of the tapes off-site… at least the monthly backups. External HDDs would probably only be cost effective if we purchased only a small amount, requiring that for the most part they be left here in the building and regardless of that they would be bulkier to take home… although I should recommend possibly getting a case to put the tapes in as well… something with a handle… I’m sure that would help defend against any possible corruption.

19

Hmmm, I would like to throw my knowledge in here. We are also a manufacturing company using Exchange, Solidworks, an older CAD product, and a lot of video and photo files.

I ran into similar issues a few years back with multiple tape drives, missed backups when I was out, reaching capacity on internal server storage and tapes. We ended up putting in a SAN and moving all storage to this, except we had to move the Solidworks files back to local on a server for performance issues. We also put in a PowerLoader with 15 tapes.

The SAN has multiple volumes for the servers, we snapshot most of the volumes daily and keep 3 snapshots. This allows for faster file recovery. Daily I mount the latest snapshots on a backup server and back them up to tape using Backup Exec. The SAN units are replicated between our two buildings for DR purposes. The tapes are cycled through and I remove one each month and send it off-site.

Now when I want to add more storage I just need to add more SAN units. They are not cheap at $10k each, need to add two for redundancy, but it ties in with my goal of server virtualisation, too. I see us booting virtual servers from the SAN, accessing the SAN for data and being able to move virtual servers between buildings for maintenance and fail-over. Again, not cheap but we run 24x7 and the cost of data entry or not being able to ship product adds up real quick.

As for desktop backups I switched a lot of my users to Roaming Profiles which accomplishes what you were discussing above. I also run, at my consultants suggestion, ASR’s on the laptops and certain execs machines in case they crash. Just had my first recovery as a matter of fact using files pulled from an ASR.

One thing to think of with regards to the ERP program is how you will back that up. Does the database need to be offline, does it need to be quiesed, will you need an agent? Currently I shut the db down, run an export, bring the database back up and then back up the export file. It’s a kludge but it’s working right now. Pretty soon, though I need a different solution, I’m hoping to add more SAN boxes and be able to snapshot the ERP databases every hour.

Something else to ponder… how long do you need to keep your data around? We need to keep some of our QA records for 7 years. I still have the data on tape but the server that has that tape drive in it has not been powered up in (2,3,4,7) years. Your choice of tape technology should be done with an eye to the future but also half an eye to the past as well.

20

Now that this has been brought back up with new information I figure I will inquire about something slightly different…

In my case we have two servers (Yea, I read 30 and I was like “WOW”… although by the end of the year we will be up to four! oh boy!!) and on these two servers we have MS Exchange on one and then we have both act as file servers.

We used to do tape backups via the three jobs: SRVR1, SRVR2 and Exchange. SRVR1 is where exchange is on, but we would only backup the other files on the first job. Since we moved over to a new device so that everything could fit on one tape I decided to consolidate it because I was lazy, (hey! I work hard, leave me alone!!), and also because I thought it would be nice to keep it clean and tidy (Yea, I know… they’re just jobs… I told you, leave me alone!)

My problem? I’m not sure there is one. We have had this thing implemented about a week and a half now and I have yet to get a backup job that says “successful.” If I fix one thing, the next night something else is wrong either from SRVR1, SRVR2 or mostly Exchange. I may have an “Access is Denied” error one night on SRVR1 and so the entire “SRVR1 and SRVR2” job I have created will fail while the next night after I have figured out the problem I will get Exchange to say some stupid consistency check failed and then fail the whole job.

Now this would be a huge deal to me but I noticed that if I go under the restore jobs wizard I can see the jobs from all these nights and the different files and it appears (I haven’t tried), that I can simply select what I want and then boom, it’ll restore it, regardless of the fail. My question is really a two-parter.

1.) Is it a good idea to consolidate all jobs of a single tape backup into one or should they be separated out? Does it matter? If it does, why?

2.) If a job is deemed to have failed (I am running BackupExec 11d but I imagine this question would possibly pertain to any tape backup program), then does that mean the entire thing is corrupt or only the files in which the failure occurred?

—> If #2 is what I think it is and that is that only files are affected if they are directly related to what the failure was, then I would feel much more at ease because most of the errors (or possibly all after I fixed the Access is Denied errors I was receiving), are of files we don’t truly care about anyway. For example I will get Access is Denied on the exchange server because a porn junk mail is corrupt! Don’t think we care about that anyway!!!

Anyway… I just ran back into the office to switch out a tape and got overly pissed because a job I tried running today failed yet again… hope my boss doesn’t see the dent in the wall!! Haha, j/k, but… well, ouch now that I look I’m not kidding.

I’m going home now…