Backup Plan - Need some help!

Hi All,

I am planning my backup strategy for my new backup server will be ready shortly. My scenario is:

Main: Unraid Server v5.04 with 12TB of Data (Parity Protected) over 5 3 TB disks.

Backup: Unraid Server (will install latest beta) with 16TB of free space over 2 8TB disks (Not initially Parity Protected due to type of disk I am using and dont want it randomly writing Parity as I copy - but I will do this when I've done the initial copy and verified that data on both arrays is the same).

What I would like: A back-up Utility or script that would mirror data to the backup server and ensure integrity (e.g. CRC or Hash Check). As an option: Something that is resume-able would be nice in case of network error or interruption.

I've considered CrashPlan or Rsync but cant find an absolute step by step guide. This is not my area! :-(

Can someone point me to a guide or product or suggest something to help me please?

EDIT: I'd also like to run all this from the Unraid OS without the need for another PC, OS or VM etc.

Daniel

If you have a Windows box that's running all the time, I'd simply install the free version of SyncBack on it; set up the appropriate profiles to backup your primary server to your backup one [if it takes multiple profiles ... e.g. one per share ... you can create a "Group" profile that runs all of the necessary profiles];  set a schedule for that profile; and you're done with it.    There are LOTS of options you can set for the profiles ... I think the only one you'd actually need is "validate" so it will validate the copies.

My syncback profiles all run automatically at 4:00 every morning.  I have another scheduled task that runs at 3:38am and sends magic packets to the server to wake it up, so it's always on and ready when Syncback starts running.

Works like a charm 

I'm sure there are similar linux-based ways to do this (using rsync), but I'm not a "linux guy) and prefer to do everything "outside" of the server when possible.

Thanks for the reply Gary. I don't have an always on PC but I guess I could run a very lightweight Windows 8 VM just for doing this. Hmmmm. I really like the Wake Up Packets idea though!!

Well ==> I was going to post the link I forgot [ http://www.2brightsparks.com/freeware/freeware-hub.html ]  ... but I also just saw your Edit that says you don't want to involve another PC.

I'm confident this can easily be done using RSync ... but you'll need one of the "linux guys" to post the appropriate script to have this run at scheduled times and show the appropriate rsync parameters for it to do what you've asked.

A simple VM (Windows 7, 8, or even XP) on your main server that had a schedule task to (a) send the magic packets and then (b) execute a scheduled SyncBack task  would work very nicely -- and wouldn't require an additional PC 

What version do you use, Free, SE or Pro?

I use SE, but the free version is fine for what you have in mind.

I bought SE because it had far better error reporting when profiles had any issues ... but the newer freeware version now has almost the same error reporting capabilities.    For what you have in mind, it should be fine [You can always buy one of the paid versions if you try it and decide you want some of the extra features].

Hi All,

 

I am planning my backup strategy for my new backup server will be ready shortly. My scenario is:

 

Main: Unraid Server v5.04 with 12TB of Data (Parity Protected) over 5 3 TB disks.

Backup: Unraid Server (will install latest beta) with 16TB of free space over 2 8TB disks (Not initially Parity Protected due to type of disk I am using and dont want it randomly writing Parity as I copy - but I will do this when I've done the initial copy and verified that data on both arrays is the same).

 

What I would like: A back-up Utility or script that would mirror data to the backup server and ensure integrity (e.g. CRC or Hash Check). As an option: Something that is resume-able would be nice in case of network error or interruption.

 

I've considered CrashPlan or Rsync but cant find an absolute step by step guide. This is not my area! :-(

 

Can someone point me to a guide or product or suggest something to help me please?

 

EDIT: I'd also like to run all this from the Unraid OS without the need for another PC, OS or VM etc.

 

Daniel

My setup is that I do a scheduled backup locally on my main server every night with Rsync (since I have enough space).

I also do a scheduled backup every night to my backup sever, also with the help of Rsync.

In addition to that I am running CrashPlan on the main server. Once it has been configured from my Windows PC it runs unattended.

Pls let me know if you need some Rsync examples …

@leifgg I would be interested in the rsync examples if you have the time, thanks

@leifgg I would be interested in the rsync examples if you have the time, thanks

Ok… first the script “CompleteRemoteBackup.txt” (rename to .sh) contains a full backup of shares from my main server to the backup server. You need to make this executable, chmod +x file.name

I have made a folder on the USB flash drive where I store all my scripts

/boot/scripts

To have the script to run every night you need to create a cron job (task). The file “CronEntryCompleteRemoteBackup.txt” (remove .txt) contains info on when to run the job and what to do.

To make this persistent (so it survives a reboot of the main server) you need an entry that creates this job at boot. The following lines should be placed in your go file:

# Setup cron job for backup to remote share
cat /boot/scripts/CronEntryCompleteRemoteBackup >> /var/spool/cron/crontabs/root

I am running the backups as root user so the cron jobs are stored in the directory for that user.

As you can see in the backup script I am also using user root on my backup server that is on 192.168.1.102.

On difficulty here is that connecting to the backup server should be done via SSH. This means that you need to generate the keys and copy them to the remote server.

I am running unRAID6 so I don’t know how this (SSH) would work on unRAID5 (if you are using that).

Here is a tutorial, follow step 1, 2 and 3:

http://www.thegeekstuff.com/2008/11/3-steps-to-perform-ssh-login-without-password-using-ssh-keygen-ssh-copy-id/

You will find the keys stored in the home folder for user root in

/root/.ssh

Sadly this location doesn’t survive a reboot either so you now need to copy the content of this folder to your USB drive. I have chosen to make a folder

/boot/scripts/my_ssh

and copied them there.

The script “SSHInstall.txt” (rename to .sh and make executable) copies the files back again after reboot if you add one more entry in the go file:

# Install my ssh keys and known remote hosts
/boot/scripts/SSHInstall.sh

When playing with Rsync remember that this is a power full tool! Always test first with test data and also use the option dry run until you are sure on what is going on.

There is a lot of guides on Rsync and it has a tremendous amount of options to play with. Also a warning on the option --delete. This will delete files on the destination if they no longer exists on the source. That’s normally ok but what Rsync does is that it first builds a file list on what to delete and what to copy (and what to leave untouched, not needed to be copied) then it starts the actual job by first deleting files and then copying new files. This means that if you interrupt the script you can end up with missing files on the destination since it was interrupted before the new once got copied.

Have fun!

Leif

CompleteRemoteBackup.txt

CronEntryCompleteRemoteBackup.txt

SSHInstall.txt

@Leifgg - does Rsync verify the copied file in anyway against the source?

@Leifgg - does Rsync verify the copied file in anyway against the source?

Never checked that before but found a link for you indicating that some checking is (or can be) done:

http://unix.stackexchange.com/questions/30970/does-rsync-verify-files-copied-between-two-local-drives

@Leifgg - does Rsync verify the copied file in anyway against the source?

Only if the -c option is used.

Many times while 'moving' files, I will do a regular rsync -vaX source destination.

After I would do a second verification rsync -vrcX source destination.

The normal -a does a time/size comparison.

The -rcX option does a recursive checksum comparison.

I might use the -n on the -vrc option for a dry run then check the reports.

It depends if you want to overwrite corruption directly or decide to deal with it manually.

I.E. as in the recent reiserFS corruption in a recent beta

Had I been using -rc for the backups, the corruption would have been propagated to the backups where as using the meta data with -a it would not get propagated unless the actual file changed.

FWIW, If speed is a goal and the two machines are co-located on the same Lan you may want to use rsync in server mode vs ssh.

I get some amazing speed out of rsync to rsync server with special socket options.

http://lime-technology.com/forum/index.php?topic=38429.msg356906#msg356906

One word of caution with the --delete option. I wouldn't use it for a while until you have the hang of everything.

After that Maybe I would use it once a week on the lightest server change days.

Point is your backup should protect you from accidental deletes, therefore once you do the --delete your recovery option is gone.

Now, s'more food for thought.

If you are backing up movies and music that doesn't change, it probably doesn't matter what direction you go in.

However if you wanted to have a running archival of important documents and pictures, then pulling the files from server onto the backup provides another option. I.E. The link-dest option.

Some articles that help.

http://jimmyg.org/blog/2007/incremental-backups-using-rsync.html

http://goodcode.io/blog/easy-backups-using-rsync/

I use rsync an incremental archival backup for many hosts.

With it, the first rsync synchronizes everything.

Subsequent rsync's with the proper link-dest parameter only synchronize the changes into the current directory.

This lets me have a running set of changes depending on how often I run the backups.

I can elaborate more later if interested, but it requires the destination directory to be named some archival name.

Then that directory is used in the --link-dest= paramter while a new destination directory will be created.

All files form the archival directory are linked to the new directory.

Then all changes are copied over, (removing links) and creating a new copy.

In my case I use dated directories as in YYYYMMDD

When I need to prune, I remove the dates I do not want to keep and move on.

I keep 6 weeklys, 6 monthly, a few yearly.

The method using hard links looks interesting but honestly I never got that far understanding what it actually meant and what it could be used for.... I guess I have some reading to do now that I get a better feel for how it can be used. Thanks!

I like the idea of an incremental back up (or Mirror) of my main Media Server. I guess I could from weekly Rsync without deletes and cron with deletes once every 6months.

I'm liking the idea of Rsync at the moment!!!

The method using hard links looks interesting but honestly I never got that far understanding what it actually meant and what it could be used for.... I guess I have some reading to do now that I get a better feel for how it can be used. Thanks!

A hard link is two different file names pointing to the same blocks of data.

Not symbolically, but as a direct inode defintion.

In the case of rsync, it provides a way if having 1 copy of the data linked to multiple directories and saving space.

In some of our data distribution FTP servers, we stage 1 copy of the data and hardlink it to the customer directories.

This has the benefit of decreased disk space 1 copy of a 500gb file is seen by many customers yet only takes up 500GB of space.

I have an example method of doing this with dated directories here.

https://code.google.com/p/unraid-weebotech/downloads/detail?name=rsync_linked_backup

While not complete it's a good model, albeit a lil more compex then the prior example.

I used it for years to backup my source and home folders.