danioj

unraid-autovmbackup: automate backup of virtual machines in unRAID - v0.4

128 posts in this topic Last Reply

Recommended Posts

WARNING - PLEASE CONSIDER THIS A WORK IN PROGRESS. I HAVE TESTED IT ON MY SERVER AND THERE WAS NO ISSUE BUT THAT DOESNT MEAN IN ANY WAY IT IS FREE FROM BUGS/ISSUS SO PLEASE USE AT YOUR OWN RISK UNTIL IT HAS BEEN TESTED FURTHER - WARNING

 

Well, this is what happens when I am in hospital with nothing to do but wait for tests. For those who don't like what I have done, then I am sorry!  :)

 

Anyway, I was discussing the need for backup of Virtual Machine vdisks. I certainly need a method that is not manual and I thought I would write a "Proof of Concept" script to do this for us to see if it might be something that would be needed by others too.

 

Like I said above, this is a proof of concept initially and it is missing most of the refinements that go into good code and a released product. For that reason I would suggest if you don't know what you're doing then don't use this.

 

 

If there are people who want to test this in their test environment and or give feedback please do the following:

 

- download script from github (link at bottom of the post).

- copy the script to your flash drive.

- read the notes in the file.

- fill in the mandatory inputs (e.g. virtual machine list and backup location).

- change optional switches if you wish to.

- from the cli execute it like below (for now).

 

sh  backup_unraid_vms.sh

 

- write a post and give feedback

 

:)8)

 

 

# this script has been created to automate the backup of unRAID vm's vdisks(s) to a specified location.

# this script does not yet run using variables passed from the cli as yet but there is intention to do so if there is interest.

# for now the variables below are what are needed to be ammended to have the script run.

 

 

# Change Log

 

# v0.1  20160330.

# initial release.

#

# notes: initial realease is more concept and to see if there is a need for it. it lacks basic error handling and recovery.

# which could be developed later.

 

# v0.2  20160330.

# bug fixes.

#

# changed how the bash script is recognising the failed shutdown flag as after testing even a failed shutdown would start a backup.

# changed default mins to wait for shutdown to 6.

 

# v0.3  20160415.

# bug fixes and enhancements.

#

# applied strickter naming conventions to variables.

# cleaned up the code, added comments, removed unecessary 'do' loops.

# added input validation and verification.

# added addition status messages. applied a bit more consistency.

# added ability to deal with vm's with names that have spaces in them.

# vms now seperated by "new line" not space.

# added rsync copy over standard nix cp command.

# added backup of vm xml configuration.

# added option to add timestamps to backup files.

# added script name check for version control.

# added option to enable / disable the script.

# added option to start vm after failure.

# added option to "dry-run" copy operation - note all other functionality is enabled.

# added ability to ignore vm's which are not installed on the system.

# set defaults for all options to 0.

# added guidence for options and inputs.

# added constraint to only backup .img vdisks so to skip installation media etc.

# changed method of obtaining vdisks list.

# fixed issue which had script fail if no vdisks were present.

 

# v0.4  20160416.

# administration.

#

# script name changed to facilitate code being added to github.

 

 

# bug tracker (order by severity)

 

# <reference> <description> <link to post> <severity> <date added> <by whome> <accepted/refected> <comment>

 

 

# to do list

 

# core

# error capturing and handling -- started (v0.3).

# apply stricter naming conventions -- done (v0.3).

# code clean up -- ongoing (v0.3)

# input validation and verification -- done (v0.3).

# logging.

# clean up status messages -- done (v0.3).

# deal with vm's named with spaces -- done (v0.3).

 

# possible improvements

#      use rsync to copy the vdisks instead of copy -- done (v0.3).

 

# possible future features

# backup vm xml file -- done (v0.3).

# have the script run from variables passed in from the cli.

# add timestamps to files -- done (v0.3).

# add iterations of backups and number of backups to maintain at any given time.

# plugin -- started (v0.3).

 

# plugin to-do's

# basic structure and requirements -- done (v0.3)

# git-hub account

# ui validation via menus

# .... much much more ....

 

 

script now available via github:

 

https://github.com/danioj/unraid-autovmbackup.git

Share this post


Link to post

Thanks, danioj. I'll test over the weekend.

 

Thanks. Let me know what you think.

 

I tested it over and over again when I had telnet access in the hospital and if the inputs are similar to as I have put in the example it seems to work just fine.

 

If there is enough interest in it I shall tighten it up.

Share this post


Link to post

I was not able to get the script to run successfully because of a couple of things specific to my setup, one being the name of my VM has spaces in it, and the other that I had my VM shutdown prior to running the script. Here's the play-by-play:

root@Tower:/boot/custom/bin# backup_unraid_vms.sh
starting backup_unraid_vms.sh to backup Win7 Virtual Desktop to /mnt/user/System/Win7-VM
starting backup of Win7
checking /mnt/user/System/Win7-VM/Win7 exists so i have somewhere to put the backup files
/mnt/user/System/Win7-VM/Win7 doesnt exist so i am going to create it
/mnt/user/System/Win7-VM/Win7 has been created
backup of Win7 failed
completed backup_unraid_vms.sh to backup Win7 Virtual Desktop to /mnt/user/System/Win7-VM
starting backup of Virtual
checking /mnt/user/System/Win7-VM/Virtual exists so i have somewhere to put the backup files
/mnt/user/System/Win7-VM/Virtual doesnt exist so i am going to create it
/mnt/user/System/Win7-VM/Virtual has been created
backup of Virtual failed
completed backup_unraid_vms.sh to backup Win7 Virtual Desktop to /mnt/user/System/Win7-VM
starting backup of Desktop
checking /mnt/user/System/Win7-VM/Desktop exists so i have somewhere to put the backup files
/mnt/user/System/Win7-VM/Desktop doesnt exist so i am going to create it
/mnt/user/System/Win7-VM/Desktop has been created
backup of Desktop failed
completed backup_unraid_vms.sh to backup Win7 Virtual Desktop to /mnt/user/System/Win7-VM

 

Things to fix:

1. It needs to handle VM names with embedded spaces.  My VM's name is "Win7 Virtual Desktop." When the script parses vms_to_backup="Win7 Virtual Desktop," it identifies VMs with names "Win7," "Virtual," and "Desktop" and creates subfolders for those backup vdisks.

2. It needs to trap invalid VM names (already on your list), such as what occurred in item 1 above (e.g., "Win7" is not a valid VM name).

3. I was not able to confirm, but I don't think it will back up vdisks for VMs that are already shutdown when the script starts. When you define the vm_state variable, use "virsh list --all" instead of "virsh list." Then add a state check for VM already shutdown. For this condition, you'll have to decide whether to restart the VM (my vote would be "no").

if [ "$vm_state" == "$vm shut off" ]; then 
  echo "$vm is already in shutdown state"
  can_backup_vm="y"
fi

 

I really appreciate the work you've done putting this script together. I expect it will get a lot of use as people discover how easy it is to use and how effectively it fills the need for backing up VMs.

Share this post


Link to post

Why do the VMs have to be shut down to copy the vdisk1.img? What happens if you try to copy it without shutting it down......will it affect the VM startup when you try to restore with copied vdisk1.img?

Share this post


Link to post
What happens if you try to copy it without shutting it down......will it affect the VM startup when you try to restore with copied vdisk1.img?

Short answer? It depends.

 

May work, may not work. In general, it's not worth the risk of a backup not working. Why do you want to back it up while it's running?

Share this post


Link to post

Why do the VMs have to be shut down to copy the vdisk1.img?

 

For a Windows VM, for example, I want to make sure Windows shuts down to force it to flush any volatile memory and disk buffers to the vdisk prior to making the backup.

Share this post


Link to post

Is it easy to turn this into a plugin so we don't have mess with the flash? Seems like any improvements that are made or features that added would be handled well through the update process with the plugins. Or, would setting up a plugin be too much effort?

Share this post


Link to post

Why do the VMs have to be shut down to copy the vdisk1.img? What happens if you try to copy it without shutting it down......will it affect the VM startup when you try to restore with copied vdisk1.img?

 

What happens if you try to copy it without shutting it down......will it affect the VM startup when you try to restore with copied vdisk1.img?

Short answer? It depends.

 

May work, may not work. In general, it's not worth the risk of a backup not working. Why do you want to back it up while it's running?

 

To add to what jonathanm said:

 

Copying the image files is ALWAYS ok if the guest virtual machines are stopped. But if they are running its possible that as the copy is taking place changes are made within the guest, leading to an inconsistent copy.

Share this post


Link to post

I was not able to get the script to run successfully because of a couple of things specific to my setup, one being the name of my VM has spaces in it, and the other that I had my VM shutdown prior to running the script. Here's the play-by-play:

 

Thanks for trying it out. It's good to captures as many individual setups as possible.

 

Things to fix:

1. It needs to handle VM names with embedded spaces.  My VM's name is "Win7 Virtual Desktop." When the script parses vms_to_backup="Win7 Virtual Desktop," it identifies VMs with names "Win7," "Virtual," and "Desktop" and creates subfolders for those backup vdisks.

 

Agreed.

 

2. It needs to trap invalid VM names (already on your list), such as what occurred in item 1 above (e.g., "Win7" is not a valid VM name).

 

Agreed.

 

3. I was not able to confirm, but I don't think it will back up vdisks for VMs that are already shutdown when the script starts. When you define the vm_state variable, use "virsh list --all" instead of "virsh list." Then add a state check for VM already shutdown. For this condition, you'll have to decide whether to restart the VM (my vote would be "no").

if [ "$vm_state" == "$vm shut off" ]; then 
  echo "$vm is already in shutdown state"
  can_backup_vm="y"
fi

 

I will have to check this again because for me, when the VM's were shutdown it worked perfectly.

 

Is it easy to turn this into a plugin so we don't have mess with the flash? Seems like any improvements that are made or features that added would be handled well through the update process with the plugins. Or, would setting up a plugin be too much effort?

 

I am already looking at a plugin for another script I wrote. I have been in hospital this week but am working on migrating this into a plugin already. Even if others don't want it, I want it for myself  :)

Share this post


Link to post

Dan,

 

Hope you're out of the hospital by now and things went well. Feeling back in action yet?

 

Cheers.

Share this post


Link to post

Thanks for the kind words.

 

I'm recovering. Made some in-roads into this while bed ridden though!!

Share this post


Link to post

Dan,

 

Hope you're out of the hospital by now and things went well. Feeling back in action yet?

 

Cheers.

 

Well, this is the first day I have actually had an hour without any pain. Anyway, as a result I took the opportunity to test the updates I made to the script while I was in hospital living off ice-cream!

 

This is a VERY significant update. Having all that time made me realise just how much was wrong with the damn thing before. I have made many changes. I had to remove the code from the forum now as there is too much for a post, but the file is of course attached.

 

Have a look at the change log. Any comments are welcome.

 

After testing myself I feel it is getting to a point where it might be getting good enough for a plugin. As I mentioned earlier, I am pursuing that now!

Share this post


Link to post

Dan, you using github?  Probably the best place for the code....

 

Sent from my LG-H815 using Tapatalk

 

 

Share this post


Link to post

Dan, you using github?  Probably the best place for the code....

 

Sent from my LG-H815 using Tapatalk

 

Good pointer. Done. Much better.  :)

Share this post


Link to post

How about a link?

 

It's on the bottom of the first post.

Share this post


Link to post

Any status on the plug-in? I'm interested in learning the process of making a plugin anyhow. Is there a noobs tutorial for this?

Share this post


Link to post

Any news on the Plug-In?

 

Thanks for your hard work!

Right now, you could easily schedule this to run whenever you want by using the user scripts plugin, or alternatively set this script to run in conjunction with appdata backup with CA.

Share this post


Link to post

run in conjunction with appdata backup with CA.

 

Is there any chance of adding VM backup natively into CA's backup function?

I think this would be the most streamlined way while we await (at some point) getting snapshot's added into the VM manager.

 

 

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


Copyright © 2005-2018 Lime Technology, Inc.
unRAID® is a registered trademark of Lime Technology, Inc.