Process to copy "appdata" docker folder form cache to array for backup

Just to throw my method in for this, I use rsnapshot to do nightly backups. I like this method because it'll keep version information in case I want to revert to an old version of some configuration.

This has been extremely helpful on my end, and for anyone who didn't look up all the options rsync has one that is great, especially if you backup plex is --exclude-from. This lets you setup an exclude file, so for example, mine looks like:

 

#!/bin/bash

#Stop Docker Containers
docker stop $(docker ps -a -q)

#Backup Docker App data via rsync
date >>/mnt/user/Backup/HADES/docker_app_backup.log
/usr/bin/rsync -avrtH --exclude-from '/boot/custom/exclude.txt' --delete /mnt/cache/apps/ /mnt/user/Backup/HADES/cache/apps >>/mnt/user/Backup/HADES/docker_app_backup.log

#Start Docker Containers
/etc/rc.d/rc.docker start

#Backup Flash Drive data via rsync
/usr/bin/rsync -avrtH --delete /boot/ /mnt/user/Backup/HADES/flash >>/mnt/user/Backup/HADES/docker_app_backup.log

 

in my exclude.txt I have this:

 

Plex/Library/Application Support/Plex Media Server/Cache/Transcode

 

Because I really don't care about backing up my synced content, and its also like 60GB of data, who cares if you have to re-sync, as long as your DB is backed up etc etc.

 

*note, you don't need a file, you can just use --exclude but I figured I might have some more exclusions down the road.

I notice your script also backs up the flash drive to the array. Of course you will have some trouble getting to that flash backup without a working flash drive.

 

Also, unRAID saves the array start/stop status in config/super.dat. If you restore a backup of flash that was taken with the array started, and then try to boot from that restored flash, unRAID will see that the array was not stopped and begin a correcting parity check due to an unclean shutdown.

 

If you stop the array, the flash drive is still accessible over the network if you have it shared. When I make a change to unRAID, I stop the array, copy the flash to my PC, then start the array. That way my flash backup is accessible without booting unRAID, and it has the correct start/stop status when restored.

I want to chime in and say that while most of this thread is still extremely helpful now I believe the method of modifying the go file should be depreciated and instead we should be suggesting people make .cron files to launch their scheduled script runs. I've modified both my daily cache trim, and nightly app share backup to launch from .cron files.

 

Also I think the addition of the function that reads .cron files into the crontab in 6.0 means that a plug-in for this might be a real possibility. I've thought about it some, but I'm a hobbyist, and don't know much PHP, so it's on the backburner for me.

 

I do want to know a bit more about the funtion that finds .cron files and puts them in the crontab... like what is doing it, and can it be invoked without a system reboot?

 

I notice your script also backs up the flash drive to the array. Of course you will have some trouble getting to that flash backup without a working flash drive.

 

Also, unRAID saves the array start/stop status in config/super.dat. If you restore a backup of flash that was taken with the array started, and then try to boot from that restored flash, unRAID will see that the array was not stopped and begin a correcting parity check due to an unclean shutdown.

 

If you stop the array, the flash drive is still accessible over the network if you have it shared. When I make a change to unRAID, I stop the array, copy the flash to my PC, then start the array. That way my flash backup is accessible without booting unRAID, and it has the correct start/stop status when restored.

 

Good point, and never thought about that, although to be honest the only reason I am backing it up is because I have random files throughout my flashdrive that I care about, such as custom scripts etc etc. So in reality I should be more granular with that approach and don't really care about backing up my full flash drive.

I want to chime in and say that while most of this thread is still extremely helpful now I believe the method of modifying the go file should be depreciated and instead we should be suggesting people make .cron files to launch their scheduled script runs. I've modified both my daily cache trim, and nightly app share backup to launch from .cron files.

 

Also I think the addition of the function that reads .cron files into the crontab in 6.0 means that a plug-in for this might be a real possibility. I've thought about it some, but I'm a hobbyist, and don't know much PHP, so it's on the backburner for me.

 

I do want to know a bit more about the funtion that finds .cron files and puts them in the crontab... like what is doing it, and can it be invoked without a system reboot?

 

Sorry if this sounds dumb, I have never really worked with .cron files... are you saying you should do something like this:

 

/boot/config/plugins/cachebackup/cachebackup.cron

 

# Daily automated backup at 4:30am:
30 4 * * * /boot/custom/backup.sh &> /dev/null

 

And unRAID just knows to look for .cron files in the plugins folder?

I want to chime in and say that while most of this thread is still extremely helpful now I believe the method of modifying the go file should be depreciated and instead we should be suggesting people make .cron files to launch their scheduled script runs. I've modified both my daily cache trim, and nightly app share backup to launch from .cron files.

 

Also I think the addition of the function that reads .cron files into the crontab in 6.0 means that a plug-in for this might be a real possibility. I've thought about it some, but I'm a hobbyist, and don't know much PHP, so it's on the backburner for me.

 

I do want to know a bit more about the funtion that finds .cron files and puts them in the crontab... like what is doing it, and can it be invoked without a system reboot?

 

Sorry if this sounds dumb, I have never really worked with .cron files... are you saying you should do something like this:

 

/boot/config/plugins/cachebackup/cachebackup.cron

 

# Daily automated backup at 4:30am:
30 4 * * * /boot/custom/backup.sh &> /dev/null

 

And unRAID just knows to look for .cron files in the plugins folder?

Yes. I have cache backup and md5 scripts scheduled this way.

I do want to know a bit more about the funtion that finds .cron files and puts them in the crontab... like what is doing it, and can it be invoked without a system reboot?

 

This is the script /usr/local/sbin/update_cron, it simply scans all folders under config/plugins for any .cron files and adds them to the cron entry list.

 

When changes are made in the cron files, or some addition/deletion of .cron files took place then you can simply rerun update_cron without system reboot.

...When changes are made in the cron files, or some addition/deletion of .cron files took place then you can simply rerun update_cron without system reboot.

Thanks for this. I was wondering how to do this myself.

I want to chime in and say that while most of this thread is still extremely helpful now I believe the method of modifying the go file should be depreciated and instead we should be suggesting people make .cron files to launch their scheduled script runs. I've modified both my daily cache trim, and nightly app share backup to launch from .cron files.

 

Also I think the addition of the function that reads .cron files into the crontab in 6.0 means that a plug-in for this might be a real possibility. I've thought about it some, but I'm a hobbyist, and don't know much PHP, so it's on the backburner for me.

 

I do want to know a bit more about the funtion that finds .cron files and puts them in the crontab... like what is doing it, and can it be invoked without a system reboot?

 

Sorry if this sounds dumb, I have never really worked with .cron files... are you saying you should do something like this:

 

/boot/config/plugins/cachebackup/cachebackup.cron

 

# Daily automated backup at 4:30am:
30 4 * * * /boot/custom/backup.sh &> /dev/null

 

And unRAID just knows to look for .cron files in the plugins folder?

 

Yes, as trurl said that works. I am also running all my custom tasks this way.

 

For simple tasks (AKA one line of code) you can put it directly into a .cron file instead of calling a script to execute.

 

I do want to know a bit more about the funtion that finds .cron files and puts them in the crontab... like what is doing it, and can it be invoked without a system reboot?

 

This is the script /usr/local/sbin/update_cron, it simply scans all folders under config/plugins for any .cron files and adds them to the cron entry list.

 

When changes are made in the cron files, or some addition/deletion of .cron files took place then you can simply rerun update_cron without system reboot.

 

Also thanks for this, is is really good information.

Super, super useful stuff guys. Top notch.

This should be a plugin. Just saying.

 

Or a Docker...

 

So theoretically, docker or plugin (i would see a plugin as easier to handle it I think), is a page that lets you:

 

1. Select run time

2. Select daily/weekly/monthly

3. Select backup folders

4. select exclusions

5. a button that updates the cron

 

This should be a plugin. Just saying.

 

Or a Docker...

 

So theoretically, docker or plugin (i would see a plugin as easier to handle it I think), is a page that lets you:

 

1. Select run time

2. Select daily/weekly/monthly

3. Select backup folders

4. select exclusions

5. a button that updates the cron

 

What you've outlined is a good basic start to a fully featured backup utility. The only thing I think you’re missing from the requirements of a basic system would be that you need to be able to select which dockers / plug-in you want to stop during the backup, and restart when the backup is done. 

 

First I think this is a case where a plug-in makes more sense than a docker for two reasons. First there is no need to install additional packages or binaries to make the plug-in work, so it avoids one of the major reasons that Docker is preferred. Second, I'm pretty sure that running this as a docker makes everything much harder, because the mapping would be weird, and you can't issue the stop all command to docker or else you stop the backup process as well...

 

Here's what in addition to what you mentioned I think should be included to make it a fully featured backup plug-in.

 

1) The ability to perform backups to remote locations

2) The ability to have multiple programmed backups with the ability to view the status of each and info about the last run

3) The option to customize your rsycn flags (perhaps with an English to flag GUI making it easier for people who aren't Linux / rsycn experts)

 

I think with those options it becomes much more fully featured. IE: it's really likely that I want to do a backup every day of some apps, but only weekly of others. Or that I want to make local backup of something and a full remote backup weekly or monthly.

 

With all this said, I think a lot of the code for this (especially the code to make a .cron file and update the crontab) already exists in the scheduler code base.

 

where is the scheduler data files stored, or more specifically the web gui pages, I know where standard plugins go but its not there...

 

this sounds like an awful idea for me to start learning how to create plugins hahaha

I would love to see this as a plugin or docker!

I would love to see this as a plugin or docker!

 

Feel free to head over freenode #linuxserver.io to tell us what you want in a bit more detail...

Just to throw my method in for this, I use rsnapshot to do nightly backups. I like this method because it'll keep version information in case I want to revert to an old version of some configuration.

 

I'm really interested in rsnapshot on unraid. Can you point me in the right direction to install this?

I have recreated the .sh file with notepad++ but I'm still getting the same error, anyone have any ideas?

 

Any chance you could re post what you had here? I'm trying to setup the dynamix system temp plugin and it's not installing the sensors-detect script by itself.

Lot of posts regarding developing this into a plugin. Is it available yet? Google isn't showing anything much besides this thread.

Any chance you could re post what you had here? I'm trying to setup the dynamix system temp plugin and it's not installing the sensors-detect script by itself.

As far as I know, the issue is not the script but the fact that perl has been removed from unRAID. See here. That thread is probably where you should be looking for answers about the plugin anyway.

Is perl included in the Nerd Pack? I think it might be..

Is perl included in the Nerd Pack? I think it might be..

No

Is perl included in the Nerd Pack? I think it might be..

No

I think it is now.

Is perl included in the Nerd Pack? I think it might be..

No

I think it is now.

What makes you think so? It is not in the list in the first post for the plugin, which was just updated today. It is also not on my server, which includes Nerd Pack.

 

Is perl included in the Nerd Pack? I think it might be..

Don't think so. I had to install perl separately.

Oops my bad.

 

Was wrong, not the first time, won't be the last.