Backing Up a Large Unraid Server

[gbdesai]

I am up to 13.5GB of storage with about 50% utilization.  I am not very concerned about Unraid losing my data due to some type of failure, but I always worry about natural and unatural disasters (i.e. fire, theft, etc.).  A lot of the content is not replaceable and the rest is costly and time consuming to replace. 

 

Does anyone out there have a good suggestion on how to backup this volume of data for offsite storage?  I guess I could get a bunch of external 2TB drives... Is that the optimal way today?  Any thoughts would be very much appreciated.

 

G

[NAS]

There are lots of way. Online commercial storage is doable but will be very expensive. You also have to take into account you internet bandwidth, If you have 2mbit lots of options become impractical.

 

Personally i think the most practical way in terms of ease of ongoing management would be to use rsync to an offsite server perhaps at a friends house. This obviously has some serious logistic hurdles.

 

The rsync method is what i use but i dont backup my whole server.

[RobJ]

I'm assuming you meant 13.5TB.

 

I don't think backing up that much is easily practicable.  A second, or perhaps a second and third, unRAID server(s) that are built to be as portable as possible, could be one option.  Trying to use any backup solution such as rsync over a network wire will take far too long to be useful, in my opinion.  Especially if you have a high rate of data change or new file creation ...  You could try connecting to a neighbor's house through a direct connection, using a second NIC ...

[gbdesai]

Yeah I did mean TB, I'll fix that.  And yes, Rsync looks cool, but it'll be a slow process over standard internet protocols.  Maybe I can just setup a second server in my home and use Rsync and take it offsite and bring it back every so often to sync again.

[NAS]

Keep in mind you would rsync locally then move the server to the end location. You wouldn't try 13.5TB remotely only the diffs to that.

 

There no solution that will easily scratch this itch but there are things that will help

[SSD]

About 400 16G thumbdrives should about take care of it.  

 

Most people here are selective about what they backup.  Backing up TBs of Movie RIPs is just not practical IMO.  Personal photos, home movies, scanned financial documents and the like are not nearly as large for most users.

[WeeboTech]

Keep in mind you would rsync locally then move the server to the end location. You wouldn't try 13.5TB remotely only the diffs to that.

 

There no solution that will easily scratch this itch but there are things that will help

 

The beauty of this is... if you put it at a family members house, you can designate drives which go one way.. and other drives which go the other way.

You can also throttle rsync so it does not consume all your bandwidth.

 

If you have this much data to back up..

rsync it locally. get the jobs working.. then consider installing the server remotely.

[kapperz]

I have been thinking of a good way to backup my 4TB unRaid (almost full). I consider all data worth backing up. Storage is fairly cheap and peace of mind is priceless. I made flac rips of my CDs and sold them and plan to ISO all my DVDs soon (currently all h.264/AAC). I shouldn't need to re-rip my movies. Spending ~$120 on a 1.5TB is better than weeks re-ripping. The really important stuff is backed up in two locations.

 

Anyhow, after reading though the forums, carrying spare drive back home to sync once a month is still the best way with so much data. I can't afford to build a second unRaid and bandwidth is so limited in the USA. Plus, it would be very difficult to carry a server living in a city without a car. With my old windows setup I had a 1:1 copy of each drive and that made things simple. With my unRaid setup, I made the split level on user shares equal 99. This makes the whole array look like one big 4TB space that I can simply throw files on. The problem with this is that the data needs to be split up on the off-site drives. I can no longer have all DVD rips on off-site disc 1 because eventually it will go over the backup discs total space. Where the unRaid will just put a DVD directory on another disc for the over flow files.

 

Since all my unRaid drives are derivatives of 500 (6x 500GB + 1TB) and I have an equal amount of space (in fewer drives (2x1.5TB + 1TB) for my off-site I will simply follow the same logic. I bought a few of thse Anti-Static Drive Case and a Hard Drive Dock. I will simply carry the drives home, rsync and bring them back.

 

unRaid |__________________| Off-Site

==============================

Disc 1 (1TB)..........................1TB

Disc 2-4 (each 500GB)............1.5TB

Disc 5-7 (each 500GB)............1.5TB

 

I knew by setting my split level to 99, the individual discs were going to have scattered files. The off-site discs will be an exact mirror of this. Not a big deal as restoring will require me to copy the files from the backup drive to a new server if the worse happens (fire, theft).

[vca]

Does anyone out there have a good suggestion on how to backup this volume of data for offsite storage?  I guess I could get a bunch of external 2TB drives... Is that the optimal way today?  Any thoughts would be very much appreciated.

 

G

 

That's pretty much what I do, I have two sets of backup disks and occasionally I swap the set I have on site with the other off site set.  I am using a backup package that writes a very long set of incremental backups ( http://arcvback.com/arcvback.html ) that makes this quite efficient time-wise.  Essentially one starts the process by doing an initial full backup (which takes a lot of time since we are talking a few TB of data), but after that is done only incremental backups are done and none of them are overwritten.  For me, turning on the external backup drive and then running the incremental backup takes place about once a week and typically is done in less than an hour (it runs at about 30-40GB/hr). I cache a copy of the backup data files on a spare drive in another machine so that when I swap the backup disk set with the off site copy I can quickly bring the stale copy up to date, then I can purge the cache and I'm ready to continue with the next few weeks on the updated set.  Then I again swap, update and purge... You can add additional disks to the backup sets as needed if you run out of space.

 

Regards,

 

Stephen

[guiri]

Here's what I do with my NAS (don't have unraid yet).

 

I got tired as hell of backing stuff up manually because Laplink which has a great interface kept screwing up, FOR YEARS.

 

I finally found this software that is pretty much bulletproof so far called ViceVersa Pro. It works like laplink and only moves the newer files or whatever parameters you set.

 

Here was the problem I had before. I had too many directories and since there was no good way for me to just copy all my stuff in one go, I had to do them one folder at the time and of course, I would forget what I'd copied, ESPECIALLY since laplink kept having problems with this, or that or whatever and I might have all the files moved except a few but wouldn't know which ones were NOT moved/copied.

 

So, I finally cleaned house and put all my files on the NAS which has a 5.5TB usable capacity in raid 5 so they look like one big partition.

 

I have a folder called FILES and then three subfolders called music, work and movies.

 

On my workstation I have three drives since I don't have raid there (long story) and one is called music, one movies and one work.

 

So, I have two options. Should I want to move or copy the WHOLE thing to something else with storage equal to the NAS, I can just copy/move the folder called FILES and everything comes along nicely or if I move stuff to smaller drives (ie, 1.5 or 2TB) then I just move ONE of the sub folders.

 

Works the same way if you're updating your  backup. You just tell viceversa pro which folder to move to the nas and it simply moves the newer files only and it can clean out files that are NOT on the source but are still on the backup, etc.

 

Works very well for me and I am now a happy camper.

 

Proggy is well worth the bucks.. http://www.tgrmn.com/

 

George

[purko]

Storage is fairly cheap and peace of mind is priceless. I made flac rips of my CDs and sold them and plan to ISO all my DVDs soon...

He used to live just a block away from the county jail...  

Now he lives just a block away from his old house.    

 

[queeg]

I have been thinking of a good way to backup my 4TB unRaid (almost full). I consider all data worth backing up. Storage is fairly cheap and peace of mind is priceless. I made flac rips of my CDs and sold them and plan to ISO all my DVDs soon (currently all h.264/AAC). I shouldn't need to re-rip my movies. Spending ~$120 on a 1.5TB is better than weeks re-ripping. The really important stuff is backed up in two locations.

 

Anyhow, after reading though the forums, carrying spare drive back home to sync once a month is still the best way with so much data. I can't afford to build a second unRaid and bandwidth is so limited in the USA. Plus, it would be very difficult to carry a server living in a city without a car. With my old windows setup I had a 1:1 copy of each drive and that made things simple. With my unRaid setup, I made the split level on user shares equal 99. This makes the whole array look like one big 4TB space that I can simply throw files on. The problem with this is that the data needs to be split up on the off-site drives. I can no longer have all DVD rips on off-site disc 1 because eventually it will go over the backup discs total space. Where the unRaid will just put a DVD directory on another disc for the over flow files.

 

Since all my unRaid drives are derivatives of 500 (6x 500GB + 1TB) and I have an equal amount of space (in fewer drives (2x1.5TB + 1TB) for my off-site I will simply follow the same logic. I bought a few of thse Anti-Static Drive Case and a Hard Drive Dock. I will simply carry the drives home, rsync and bring them back.

 

unRaid |__________________| Off-Site

==============================

Disc 1 (1TB)..........................1TB

Disc 2-4 (each 500GB)............1.5TB

Disc 5-7 (each 500GB)............1.5TB

 

I knew by setting my split level to 99, the individual discs were going to have scattered files. The off-site discs will be an exact mirror of this. Not a big deal as restoring will require me to copy the files from the backup drive to a new server if the worse happens (fire, theft).

 

I don't know anything about rsync so if I'm intruding just let me know.  Maybe the add-on application I wrote called SNAP could help a bit.  Using SNAP you could configure the unRAID server to detect a hotplugged drive (eSata or USB) and start up your backup software automatically.  You could probably configure what data gets backed up onto which disks.  Just stick the drive in and come back later (whenever the backup should finish) and remove it.  Just a thought.

 

http://lime-technology.com/forum/index.php?topic=5904.0