CrashPlan


agw

Recommended Posts

  • Replies 533
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Posted Images

Wow, this looks like a great service, thanks for the heads up!  I will test it out (backup my desktop and laptop to my unRAID server) and report back.  I'm going to try to convince some friends to start a backup ring with me as well, this seems like the cheapest and easiest way to achieve offsite backups that I've come across.

Link to comment

I did quite a bit of reading there, and it IS impressive.  Mentions the ability to backup to numerous devices, local and remote, including your own or a friend's Drobo, never mentions unRAID!  If you have non-personal data or non-personal usage, then upfront cost goes from FREE to $60.  Otherwise, costs for personal and business use seem reasonable.

 

It would be nice to have a few vocal unRAID users there, so we get the same free advertising as Drobo does.

Link to comment

Well, no luck so far.  I have installed CrashPlan (64 bit version) on my Vista x64 desktop and attempted to set my unRAID server as the backup destination.  I tried adding a network location and mapping a network drive, but neither show up in CrashPlan as a potential backup destination, only my local disks do.  Any ideas?

 

Edit: I'm trying to add unRAID as a 'folder' backup, as if it were an external hard drive.  Since (I assume) the CrashPlan software can't be installed on my unRAID server, I can't assign it as a 'computer' backup.

Link to comment

Why not just install the crashplan client/server on unraid?

 

Then you can choose unraid as a backup target as 'another computer' within crashplan - sticking to how they like things to work.

 

Crashplan installs ok into the ramfs - I need to revisit to put it persistently on one of the array disks somewhere - and the server runs.

 

I couldn't run the client interface natively to register it to my crashplan account - even X forwarding over ssh didn't work. However the docs provided with the linux release provide details on how to make the client software on another machine look towards any crashplan server - rather than the one on localhost by default.

 

A bit nasty, but for unraid you'd only need to do it once.

 

I don't see any reason this won't work - although as above I've only completed 50% of the task.

Link to comment

I've managed to install it and lnk the backup folder onto the array.

 

I then connected to the server from my windows desktop by changing the client config and configured the unraid crashplan install to login to my account.

 

Backups to it then worked as you'd expect and files were only opened on the array when backups were in progress - so idle disk spindown and the ability to stop the arrary was not impeded.

 

I also hacked their install script to remove all interactivity and hardcode the install paths to allow it to install from the go script at boot time. Copying the config files generated across during the intial install as well to, in theory, avoid the step of having to login and setup the account again.

 

BUt this didn't work. It worked in isolation - copying the configs and then uninstalling crashplan before reinstalling and copying the archived configs back and then starting the server made the whole thing seamless. But as soon as I actually rebooted although crashplan reinstalled ok and came back up its internal id seemed to have changed.

 

I had to completely remove it from my crashplan account and re add as a new machine before my clients would see and connect to it again. Unfortunately this was a manual operation and it also meant any backups were removed as part of the process.

 

I'm not sure how it generates this id or where it stores it. I had thought I'd archived all the files necessary to contain this but it would appear I've missed something or it doesn't behave how I think.

 

So more investigation required! Not quite as easy as I'd hoped. I am, in general, a bit alarmed that it' so fragile to a reboot.

Link to comment

It works just fine between my Vista x64 Desktop and my Vista x86 Laptop, just not with my unRAID server.  I'm backing up just under 10 GBs from each (mainly pictures and documents), and after using the 'compact' feature, the used disk space dropped to about 7 GBs each.  The backup was originally going to take several hours over wireless, so I plugged in my laptop to my LAN and it dropped to about half an hour.  I haven't tried to restore any files yet, since I haven't needed to.  Perhaps I'll simulate a crash and make sure the restore function works.

 

So in short, CrashPlan works perfectly as far as I can tell.  It just doesn't work with unRAID (yet).  My only difficulty thus far is convincing my friends to start a backup ring with me - for some reason they are very hesitant (they don't want to feel obligated to leave their computers on one night per week to make sure the backups are up to date).  I'm not giving my parents a choice about it, though, I'm just going to set it up for them and make them use it (well, it runs automatically so they don't really have to do anything).

 

What would be really nice is to pair CrashPlan with a script that turns all the computers on for several hours during the middle of the night to update and verify backups, then turns them all off again once it is complete.  I'm sure someone here knows how to accomplish this.

Link to comment

I'm stuck at the moment.

 

I went down the simpler route of rather than trying to be clever and reinstall crashplan each boot and copy configs over - simply install it once, configure it to login to my account and then tarball the entire /usr/local/crashplan directory up and untar it each boot.

 

Again this works as long as theres no reboot.

 

I can stop the crashplan engine, tar up the folder, remove /usr/local/crashplan, untar my tarball back into /usr/local/crashplan and then start the engine again. Works fine and clients only notice it go away for a minute or so.

 

However if I add a reboot into that my login settings are 'forgotten' and the computer id according to crashplan of my unraid server has changed.

 

I have no idea why. All the files held open by the process whilst running are in the tarball. I might install itnotify and track what files it puts elsewhere on the filesystem during install in case I'm missing something.

 

I've posted on their support forums but as this is a wildly left of center config and they don't profess to offering much support for the free version I don't hold out much hope!

Link to comment

I might install itnotify and track what files it puts elsewhere on the filesystem during install in case I'm missing something.

 

You could also pipe [pre]find / -cmin -5[/pre] to a couple of files before and after installing, then compare the output - a little kludgey, but saves having to install another package.

Link to comment

I appreciate your efforts, boof, I hope you get it figured out.  If it ends up involving anything more than a one-time configuration, I probably won't bother with it.  I'll just keep using unRAID for manual backups, and CrashPlan for automatic backups between all my non-unRAID computers.

 

Also, just a thought, I wonder if it is possible to manually backup the CrashPlan files to the unRAID server...this page details the default locations for the backup files.  Maybe something could be set up to backup that folder to the unRAID server (rsync maybe? I hesitate to use that term without fully understanding it).  So unRAID still wouldn't show up in the CrashPlan interface and it would be far more of a pain to restore from the unRAID server (since the files would have to be copied back to the desktop/laptop, then recognized by CrashPlan, then restored), but at least the data will be safe.

Link to comment

I might install itnotify and track what files it puts elsewhere on the filesystem during install in case I'm missing something.

 

You could also pipe [pre]find / -cmin -5[/pre] to a couple of files before and after installing, then compare the output - a little kludgey, but saves having to install another package.

 

Good plan - thanks. You're right much simpler. I can ensure the array is quiet so shouldn't see many spurious files.

Link to comment

Well other than the init scripts which I don't bother putting in the tarball as they're quite superfluos we have this file sneaking outwith the directory hierarchy :

 

/var/lib/crashplan/.identity

 

Which rings some pretty big alarm bells!

 

The contents of this file contain the private and public key, the email you use to sign into your account and guid on the machine.

 

Looks like we've hit the nail on the head.

 

I'll reconstruct the tar and try a reboot again.

 

Not sure if I'll get a chance to poke this much today or tomorrow so might be sunday before I report back!

 

Thanks all for the help.

Link to comment

okdokay - this seemed to work fine!

 

Here is a very quick and rough guide to getting this going.

 

Some caveats:

 

- this hasn't been tested thoroughly in terms of clients being happy

- You do need a couple of packages installed on unraid to satisfy

- my unraid is quite tweaked so you might need *more* packages than I  list below.

- consider this a first pass we can improve it with feedback and more testing when time allows

 

Okdokay.

 

Prerequisites :

 

- Access to a windows machine with crashplan installed (we just want the client!)

- cpio installed on unraid

- sshd and openssl installed

- both can be found from any slackware repository. Be sure to grab the 12.x stream or if you want to play safe the exact files I have below.

- there are guides on installing ssh elsewhere on the forum / wiki. I'll give a breif overview but you'd be better looking there for the moment if you don't find it intuitive.

 

cpio I'm using the cpio-2.5-i486-3.tgz and install using installpkg /boot/packages/cpio-2.5-i486-3.tgz in the go script.

 

ssh requires openssl.

 

Packages I'm using are openssl-0.9.8h-i486-3_slack12.1.tgz and openssh-5.1p1-i486-1_slack12.1.tgz.

 

My go script for these looks like :

 

# Install sshd
installpkg /boot/packages/openssl-0.9.8h-i486-3_slack12.1.tgz
installpkg /boot/packages/openssh-5.1p1-i486-1_slack12.1.tgz
cp --preserve=timestamps /boot/custom/etc/ssh/ssh*key* /etc/ssh
chmod 600 /etc/ssh/ssh*key*
/etc/rc.d/rc.sshd start

 

WHen you first install it and start it, it will generate your server key files. You need to immediately do the reverse of what I've done above to make your keys persistent across boot. You don't *have* to do this but your ssh client will bug you each initial connect after reboot if not and you lose the signature verification security benefit.

 

so

cp --preserve=timestamps /etc/ssh/* /boot/custom/etc/ssh

 

Alter for your own config.

 

Right thats that out the way!

 

For crashplan :

 

- Download the linux .tgz installer from their website

- Untar (tar -zxfv)

- run install.sh

- If all goes well you can just follow through the prompts here. Things to be wary of :

                      - If it asks you about Java tell it you would like it to go fetch it and install it dedicated to crashplan. It does this quite neatly and tucks it under /usr/local/crashplan

                      - Enter anything for the rc scripts, we won't be using them and they'll go bye bye on your next reboot.

                      - Your manifest directory is where your backups will live. You *really* want this on your unraid array ;) . I used /mnt/disk4/Private/Backups but of course tailor to suite.

                      - generally accept the defaults for everything else

- This should install crashplan. You might notice wget popping off for Java which will be the longest part of the process.

- Crash plan is now installed and running! pop off a quick ps aux | grep java and make sure theres something crashplan-esque running.

 

Now we need to connect to the engine on your unraid server to add it to your account.

 

- On your windows machine go to your crashplan install directorys conf folder.

- for me this was C:\Program Files\CrashPlan\conf

- edit the file (in wordpad) ui.properties

- this is all commented out on my default windows install, don't worry about that.

- add the following to the bottom :

 

servicePort=4200

 

- This will point the client to this arbitrary port instead of the one running the crashplanengine on your actual windows machine (bear with me!).

- If you run the client at this point you will not connect to anything and will receive an error.

- We need to port forward using ssh now. This is, again, slightly beyond the scope of this but I'll do my best!

- I'm assuming your ssh install is working - you tested it right? :)

- I'm running putty. You need to portforward port 4200 on your local machine to 4243 on the unraid server. (this is the actual unraid server engine port!)

- In putty this is under the connection -> ssh -> tunnels menu.

- Add 4200 as the source port then in the destination enter localhost:4243

- make sure you save this to your unraid server session so you don't have to do it again next time!

- login to your unraid server via ssh using this newly updated connection

- your tunnel is now open!

- Run the crashplan client again on your windows machine. This should now popup the box to allow you to create a new account or login with an existing one!

- At this point you're connected to the engine on your unraid server.

- Login using your existing credentials and make sure things proceed as you'd expect.

- Thats it! just close the client now.

- This clearly generates the /var/lib/crashplan/.identity file which is so key

 

I think this is the most fiddly step. Remember to undo the changes in the windows client config file to be able to go back to editing the windows crashplan instance. You can re add that line at any time to connect to and manage the unraid instance.

 

Now to make crashplan persist across a reboot. By this point you shoudl have crash plan running and have configured it to add your unraid machine to your account. All your other clients should see your unraid server and you can now, if you want, send backups to the unraid server.

 

To make persistent across a reboot :

 

- be neat and shutdown the crashplan engine cleanly

/usr/local/crashplan/bin/CrashPlanEngine stop

 

- tar up the crashplan folder including the .identify file thats been causing us greif!

 

tar -cvf /boot/packages/crashplan.tar /usr/local/crashplan /var/lib/crashplan

 

- add the following to your go script :

 

# install crashplan
tar -C / -xvf /boot/packages/crashplan.tar
/usr/local/crashplan/bin/CrashPlanEngine start

 

- This untars the tar ball we've just created relative to the / folder. Thus putting everything back nicely in place. Including that pesky identity file!

- It then fires up crashplan

- This is quite verbose, drop the v flag in tar to run quietly if you like watching your boot up messages.

 

And, I think, that's it!

 

The only caveats I can think of :

 

- I believe the crashplan client autoupdates? oviously this will never stick across reboots. It may be an exercise to crontab the creation of a new tarball periodically based on your current install

- When backups are running your array will have open files so shutting down will be difficult

 

Other than that, please try the above. Reboot and test things in anger!

 

Probably the simplest, and the best, test would be to :

 

- Install / reboot etc and make sure it looks good.

- Kick off a backup from a client to the unraid server and let it finish.

- Reboot the server

- Do a restore / another backup from the same client and make sure it propogates seamlessly.

 

I hope this helps. I've written it off the topof my head so things might not tie up nicely or I may have skipped a step. I honestly think the hardest part of this is getting your pre requisites up and running and the windows client botch / ssh tunnel part. Other than that the original install of crashplan is straightforward (thanks crashplan devs!) and then we're just creating a simple tar ball. All my issues stemmed from not being aware of the /var/lib directory.

 

So comments / abuse / improvements welcomed :)

Link to comment

Wow.  Impressive boof.

 

My head is spinning.

 

Your instructions seem very thorough to my linux-challenged brain.  Maybe just enough to get me to try it - but it is certainly beyond my comfort level. 

 

What are the chances that someday I can just have a 'CrashPlan' button on unMenu?!

 

 

Link to comment

Wow.  Impressive boof.

 

My head is spinning.

 

Your instructions seem very thorough to my linux-challenged brain.  Maybe just enough to get me to try it - but it is certainly beyond my comfort level. 

 

What are the chances that someday I can just have a 'CrashPlan' button on unMenu?!

 

 

 

Probably quite good. openssh and cpio would be easy (probably) to add to unmenu.

 

The installation of crashplan at the unraid side is a two stage process (initial install, import of user details) but could be done. All the settings for where things live and where your backups go are in a config file in plain text.

 

The only downer to the plan is that you need it to generate the .identity file by logging in manually.

 

So the best unmenu could do would be to take you to that point, mark it as such and say 'click this button when you've done the rest'.

 

The rest being setting up the ssh tunnel, logging in via ssh, changing the windows client settings and connecting to the unraid crashplan engine to do the initial login to generate said values in .identity.

 

In my mind thats still the trickiest part and its something unmenu or unraid side scripting can't help with.

 

I guess template windows client config files and putty registry files could be created to allow easy creation of the right files and putty configs with the port forwarding already in place. But it's still not very slick.

 

Or if we know how the .identity is generated we could perhaps do it manually (unlikely though given it contains encryption keys?) or if there is an option now or in the future to do a command line initial login direct from unraid rather than having to set up the windows bit (i.e /usr/local/bin/crashplan/bin/CrashPlanEngine -setup -email my@mail -password secret)

 

That might exist, I havn't looked, but I doubt it as the windows method is the 'official' way documented by the devs to configure a headless linux server.

 

I hand this over to brighter minds on the board to think about :)

 

I would say have a go - we're all here to help if you get stuck or need a nudge in the right direction.

Link to comment
  • 2 weeks later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.