[Plug-In] Community Applications

Is it possible to backup the appdata using the plugin to a remote share added to unassigned devices.

I have a usb flash drive attached to my router and would prefer to backup appdata there rather than my array.

Away from computer for a couple of days but I believe that I included support of ud mounted shares but since I don't use the plugin never tested.  If the share appears in the destination disk drop down then it should work.  Let me know

 

Sent from my SM-T560NU using Tapatalk

 

Unfortunately the share doesnt appear in the list. When mounting the remote smb share using the unassigned plugin it creates the remote share as a mount point

/mnt/disks/DSL-AC68U-83E8_ASUS

 

I can select this as a disk in the app backup module but then when asking for which share i cant get any further.

 

It would be great if you could make the module be able to backup appdata to shares off the server.

 

It works as is.  All you have to do is manually enter in the destination folder.  (It'll take a big ass rewrite of the entire module to get the drop down share picker to reflect that you're using a UD mounted device - maybe at some point in the future).  That being said, I make no guarantees about the ability to successfully complete a backup to a UD mounted device.  If the destination does not properly handle symlinks, then the backup is going to be incomplete, and you will run into trouble on a restore.  (You will know because errors will be returned on completion)  On my test to my secondary server, it will return errors due to symlinks.  Your mileage will vary.

 

 

Ok.  You'll have it by Saturday

You didn't say WHICH Saturday.  ;D

 

Are you angling for a position at limetech? 

https://cdn.meme.am/instances/500x/56525828.jpg[/img]

jonathan it's squid, on saturday it'll be....

 

-- Update to community applications 2016-06-25k

  Fixes all the balls up from 2016-06-25 to 2016-06-25j

 

 

 

Updated to 2016.06.25k with additional text on how to handle UD mounts 

 

 

 

jonathan it's squid, on saturday it'll be....

 

-- Update to community applications 2016-06-25k

  Fixes all the balls up from 2016-06-25 to 2016-06-25j

 

 

 

Updated to 2016.06.25k with additional text on how to handle UD mounts 

 

I'm saying nothing.......

 

But I am laughing.... lol

 

 

 

 

jonathan it's squid, on saturday it'll be....

 

-- Update to community applications 2016-06-25k

  Fixes all the balls up from 2016-06-25 to 2016-06-25j

 

 

 

Updated to 2016.06.25k with additional text on how to handle UD mounts 

 

I'm saying nothing.......

 

But I am laughing.... lol

Well, as a beta tester I would have to say that you don't find too many issues...  Maybe its time that I fire you... 

 

Is it possible to backup the appdata using the plugin to a remote share added to unassigned devices.

I have a usb flash drive attached to my router and would prefer to backup appdata there rather than my array.

Away from computer for a couple of days but I believe that I included support of ud mounted shares but since I don't use the plugin never tested.  If the share appears in the destination disk drop down then it should work.  Let me know

 

Sent from my SM-T560NU using Tapatalk

 

Unfortunately the share doesnt appear in the list. When mounting the remote smb share using the unassigned plugin it creates the remote share as a mount point

/mnt/disks/DSL-AC68U-83E8_ASUS

 

I can select this as a disk in the app backup module but then when asking for which share i cant get any further.

 

It would be great if you could make the module be able to backup appdata to shares off the server.

 

It works as is.  All you have to do is manually enter in the destination folder.  (It'll take a big ass rewrite of the entire module to get the drop down share picker to reflect that you're using a UD mounted device - maybe at some point in the future).  That being said, I make no guarantees about the ability to successfully complete a backup to a UD mounted device.  If the destination does not properly handle symlinks, then the backup is going to be incomplete, and you will run into trouble on a restore.  (You will know because errors will be returned on completion)  On my test to my secondary server, it will return errors due to symlinks.  Your mileage will vary.

 

 

Ok.  You'll have it by Saturday

You didn't say WHICH Saturday.  ;D

 

Are you angling for a position at limetech? 

https://cdn.meme.am/instances/500x/56525828.jpg[/img]

jonathan it's squid, on saturday it'll be....

 

-- Update to community applications 2016-06-25k

  Fixes all the balls up from 2016-06-25 to 2016-06-25j

 

 

 

Updated to 2016.06.25k with additional text on how to handle UD mounts 

 

Hi Squid thanks for the update.. Just sent you a beer

 

Ive backed up my appdata to my flash drive attached to my router.

I did get an error but i cant see what is missing. Anyhow i have done a backup to my array aswell!

 

log

un 25 23:02:08 Prime root: #######################################
Jun 25 23:02:08 Prime root: Community Applications appData Backup
Jun 25 23:02:08 Prime root: Applications will be unavailable during
Jun 25 23:02:08 Prime root: this process.  They will automatically
Jun 25 23:02:08 Prime root: be restarted upon completion.
Jun 25 23:02:08 Prime root: #######################################
Jun 25 23:02:09 Prime root: Stopping binhex-delugevpn
Jun 25 23:02:19 Prime kernel: vethd01bd93: renamed from eth0
Jun 25 23:02:19 Prime kernel: docker0: port 1(veth59673bd) entered disabled state
Jun 25 23:02:19 Prime kernel: docker0: port 1(veth59673bd) entered disabled state
Jun 25 23:02:19 Prime kernel: device veth59673bd left promiscuous mode
Jun 25 23:02:19 Prime kernel: docker0: port 1(veth59673bd) entered disabled state
Jun 25 23:02:19 Prime root: Stopping binhex-sabnzbd
Jun 25 23:02:29 Prime kernel: vethd9e4767: renamed from eth0
Jun 25 23:02:29 Prime kernel: docker0: port 2(veth6f7f723) entered disabled state
Jun 25 23:02:29 Prime kernel: docker0: port 2(veth6f7f723) entered disabled state
Jun 25 23:02:29 Prime kernel: device veth6f7f723 left promiscuous mode
Jun 25 23:02:29 Prime kernel: docker0: port 2(veth6f7f723) entered disabled state
Jun 25 23:02:29 Prime root: Stopping binhex-sickrage
Jun 25 23:02:39 Prime kernel: veth3443ba0: renamed from eth0
Jun 25 23:02:39 Prime kernel: docker0: port 3(veth46ba1fe) entered disabled state
Jun 25 23:02:39 Prime kernel: docker0: port 3(veth46ba1fe) entered disabled state
Jun 25 23:02:39 Prime kernel: device veth46ba1fe left promiscuous mode
Jun 25 23:02:39 Prime kernel: docker0: port 3(veth46ba1fe) entered disabled state
Jun 25 23:02:39 Prime root: Stopping EmbyServer
Jun 25 23:02:42 Prime ntpd[2525]: Deleting interface #3 docker0, 172.17.0.1#123, interface stats: received=0, sent=0, dropped=0, active_time=437 secs
Jun 25 23:02:43 Prime root: Backing up  appData from /mnt/cache/appdata/ to /mnt/disks/ROUTER_sda1/backupappdata/[email protected]
Jun 25 23:02:43 Prime root: Using command: /usr/bin/rsync -avXHq --delete --exclude "/mnt/cache/docker/docker.img"  --log-file="/var/lib/docker/unraid/community.applications.datastore/appdata_backup.log" "/mnt/cache/appdata/" "/mnt/disks/ROUTER_sda1/backupappdata/[email protected]" > /dev/null 2>&1
Jun 25 23:04:05 Prime root: Backup Complete
Jun 25 23:04:05 Prime root: Restarting binhex-delugevpn
Jun 25 23:04:06 Prime kernel: device veth8a614b8 entered promiscuous mode
Jun 25 23:04:06 Prime kernel: docker0: port 1(veth8a614b8) entered forwarding state
Jun 25 23:04:06 Prime kernel: docker0: port 1(veth8a614b8) entered forwarding state
Jun 25 23:04:06 Prime kernel: docker0: port 1(veth8a614b8) entered disabled state
Jun 25 23:04:06 Prime kernel: eth0: renamed from veth220fa21
Jun 25 23:04:06 Prime kernel: docker0: port 1(veth8a614b8) entered forwarding state
Jun 25 23:04:06 Prime kernel: docker0: port 1(veth8a614b8) entered forwarding state
Jun 25 23:04:06 Prime root: Restarting binhex-sabnzbd
Jun 25 23:04:06 Prime kernel: device veth5a50b65 entered promiscuous mode
Jun 25 23:04:06 Prime kernel: docker0: port 2(veth5a50b65) entered forwarding state
Jun 25 23:04:06 Prime kernel: docker0: port 2(veth5a50b65) entered forwarding state
Jun 25 23:04:06 Prime kernel: eth0: renamed from veth90ef4e6
Jun 25 23:04:06 Prime root: Restarting binhex-sickrage
Jun 25 23:04:06 Prime kernel: device veth7a8164e entered promiscuous mode
Jun 25 23:04:06 Prime kernel: docker0: port 3(veth7a8164e) entered forwarding state
Jun 25 23:04:06 Prime kernel: docker0: port 3(veth7a8164e) entered forwarding state
Jun 25 23:04:06 Prime kernel: eth0: renamed from vethb380fdb
Jun 25 23:04:06 Prime root: Restarting EmbyServer
Jun 25 23:04:07 Prime root: #######################
Jun 25 23:04:07 Prime root: appData Backup complete
Jun 25 23:04:07 Prime root: #######################
Jun 25 23:04:07 Prime root: Rsync Errors Occurred: Partial transfer due to error
Jun 25 23:04:07 Prime root: Last 10 lines of rsync log:
Jun 25 23:04:07 Prime root: 2016/06/25 23:04:05 [26460] >f+++++++++ kde/share/config/krusaderrc
Jun 25 23:04:07 Prime root: 2016/06/25 23:04:05 [26460] >f+++++++++ kde/share/config/ktimezonedrc
Jun 25 23:04:07 Prime root: 2016/06/25 23:04:05 [26460] >f+++++++++ kde/share/config/kuriikwsfilterrc
Jun 25 23:04:07 Prime root: 2016/06/25 23:04:05 [26460] >f+++++++++ kde/share/config/kwalletrc
Jun 25 23:04:07 Prime root: 2016/06/25 23:04:05 [26460] >f+++++++++ kde/share/config/phonondevicesrc
Jun 25 23:04:07 Prime root: 2016/06/25 23:04:05 [26460] cd+++++++++ kde/share/kde4/
Jun 25 23:04:07 Prime root: 2016/06/25 23:04:05 [26460] cd+++++++++ kde/share/kde4/services/
Jun 25 23:04:07 Prime root: 2016/06/25 23:04:05 [26460] sent 613,476,558 bytes  received 122,524 bytes  7,437,564.63 bytes/sec
Jun 25 23:04:07 Prime root: 2016/06/25 23:04:05 [26460] total size is 612,883,913  speedup is 1.00
Jun 25 23:04:07 Prime root: 2016/06/25 23:04:05 [26460] rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
Jun 25 23:04:07 Prime root: Rsync log to flash drive disabled
Jun 25 23:04:07 Prime sSMTP[29090]: Creating SSL connection to host
Jun 25 23:04:07 Prime sSMTP[29090]: SSL connection using ECDHE-RSA-AES128-GCM-SHA256
Jun 25 23:04:08 Prime ntpd[2525]: Listen normally on 4 docker0 172.17.0.1:123
Jun 25 23:04:08 Prime ntpd[2525]: new interface(s) found: waking up resolver
Jun 25 23:04:09 Prime sSMTP[29090]: Sent mail for ***********@gmail.com (221 2.0.0 closing connection bh7sm3630094wjb.22 - gsmtp) uid=0 username=root outbytes=724
Jun 25 23:04:09 Prime root: rsync returned errors.  Not deleting old backup sets of appdata

Jun 25 23:04:07 Prime root: 2016/06/25 23:04:05 [26460] rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
Jun 25 23:04:07 Prime root: Rsync log to flash drive disabled
Jun 25 23:04:07 Prime sSMTP[29090]: Creating SSL connection to host
Jun 25 23:04:07 Prime sSMTP[29090]: SSL connection using ECDHE-RSA-AES128-GCM-SHA256
Jun 25 23:04:08 Prime ntpd[2525]: Listen normally on 4 docker0 172.17.0.1:123
Jun 25 23:04:08 Prime ntpd[2525]: new interface(s) found: waking up resolver
Jun 25 23:04:09 Prime sSMTP[29090]: Sent mail for [email protected] (221 2.0.0 closing connection bh7sm3630094wjb.22 - gsmtp) uid=0 username=root outbytes=724
Jun 25 23:04:09 Prime root: rsync returned errors.  Not deleting old backup sets of appdata

Exact same error that I saw going to a user share (which doesn't support symlinks) on my secondary server.

 

There's no real way that I can determine where the error actually lies, since it's a non-fatal error.  If you set the module to save the logs to the flash drive, then you can go through and determine what the actual file(s) are.  But, odds are that its a symlink issue...  Whether or not its important to the app, only you can determine.

 

In my case, after a restore from the faulty backup, I couldn't tell if there was any issue caused by it.

 

I would think that since you've got another backup of the appdata onto the array, to test it I would trash your appdata folder, then restore from your UD mount and see what happens.

 

No guarantees... 

 

And thanks for the beer!

 

(there are a couple of display anomalies that I need to fix with regards to UD mounts (didn't notice them until after the release - hard to test absolutely everything  , but I'll look like a complete idiot and will never hear the end of it from CHBMB if I pump out 25l today... So I'll just wait for tomorrow)

 

 

(there are a couple of display anomalies that I need to fix with regards to UD mounts (didn't notice them until after the release - hard to test absolutely everything  , but I'll look like a complete idiot and will never hear the end of it from CHBMB if I pump out 25l today... So I'll just wait for tomorrow)

 

lol

I'm saying nothing.......

 

But I am laughing.... lol

Well, as a beta tester I would have to say that you don't find too many issues...  Maybe its time that I fire you... 

 

What do you expect from someone who has no vowels?

 

(there are a couple of display anomalies that I need to fix with regards to UD mounts (didn't notice them until after the release - hard to test absolutely everything  , but I'll look like a complete idiot and will never hear the end of it from CHBMB if I pump out 25l today... So I'll just wait for tomorrow)

 

lol

oh he's already laughing even louder.... 

Wh nds vwls anywy?

 

Nd 'm lghng alrght...

 

https://cdn.meme.am/instances/500x/42943177.jpg[/img]

 

- Fixed up some minor little things

- Added in automatic boot flash drive backup

 

Due to its minimal size, flash drive backup is automatic not optional.  A new folder will created (and updated every time) the backup process runs called Community_Applications_USB_Backup, without the config/super.dat file.

 

What this means is that in the event of a flash drive failure, you would recreate the flash drive (register it as a trial), set up all of your drives, and then copy the backed up files from either your appdata share, or your backup of appdata share.  After a reboot, unRaid will walk you through transferring your .key file to a new flash drive, and all of your plugins, docker templates, etc etc etc will be back the exact same as they were prior to the crash.

 

I'm specifically excluding the super.dat file due to the risks involved in restoring it should the drive configuration change in between backups.  If you want the system to keep track of super.dat, then back it up yourself, because CA is not going to put itself into a position that can potentially cause major data loss by backing up a super.dat (and having you restore it yourself).

 

Also on restores of appdata, the flash drive will NOT be restored onto the flashdrive (but it will get restored back into appdata)

 

 

Wh nds vwls anywy?

 

Nd 'm lghng alrght...

Last time I checked, a was a vowel

 

jonathan it's squid, on saturday it'll be....

 

-- Update to community applications 2016-06-25k

  Fixes all the balls up from 2016-06-25 to 2016-06-25j

 

Just for the hell of it I counted up all of the "official" releases I've made to CA (ie: the ones that actually made it into a change log) and it totals 93.  (TBH I'm a little surprised I haven't broken 100 yet) 

 

At least it keeps moving forward lol

 

 

 

jonathan it's squid, on saturday it'll be....

 

-- Update to community applications 2016-06-25k

  Fixes all the balls up from 2016-06-25 to 2016-06-25j

 

Just for the hell of it I counted up all of the "official" releases I've made to CA (ie: the ones that actually made it into a change log) and it totals 93.  (TBH I'm a little surprised I haven't broken 100 yet) 

 

At least it keeps moving forward lol

 

 

I hear that unraid 6.2 series won't enter RC phase until CA has at least 135 releases.

 

 

 

jonathan it's squid, on saturday it'll be....

 

-- Update to community applications 2016-06-25k

  Fixes all the balls up from 2016-06-25 to 2016-06-25j

 

Just for the hell of it I counted up all of the "official" releases I've made to CA (ie: the ones that actually made it into a change log) and it totals 93.  (TBH I'm a little surprised I haven't broken 100 yet) 

 

At least it keeps moving forward lol

 

 

I hear that unraid 6.2 series won't enter RC phase until CA has at least 135 releases.

Give me more ideas then

 

Sent from my LG-D852 using Tapatalk

 

 

 

https://cdn.meme.am/instances/500x/42943177.jpg[/img]

 

- Fixed up some minor little things

- Added in automatic boot flash drive backup

 

Due to its minimal size, flash drive backup is automatic not optional.  A new folder will created (and updated every time) the backup process runs called Community_Applications_USB_Backup, without the config/super.dat file.

 

What this means is that in the event of a flash drive failure, you would recreate the flash drive (register it as a trial), set up all of your drives, and then copy the backed up files from either your appdata share, or your backup of appdata share.  After a reboot, unRaid will walk you through transferring your .key file to a new flash drive, and all of your plugins, docker templates, etc etc etc will be back the exact same as they were prior to the crash.

 

I'm specifically excluding the super.dat file due to the risks involved in restoring it should the drive configuration change in between backups.  If you want the system to keep track of super.dat, then back it up yourself, because CA is not going to put itself into a position that can potentially cause major data loss by backing up a super.dat (and having you restore it yourself).

 

Also on restores of appdata, the flash drive will NOT be restored onto the flashdrive (but it will get restored back into appdata)

 

 

Wh nds vwls anywy?

 

Nd 'm lghng alrght...

Last time I checked, a was a vowel

Crp....

idea - maybe a .txt file to go with flash backup showing drive serial numbers and where they are assign in case they dont have it written down or printed somewhere?

 

 

Myk

Another idea - since the flash backup is so small - if they have it avail - maybe ftp to external site?

 

Myk

 

Maybe this has already been discussed and I just missed it, but I notice that in Backup Appdata my Save Log Files To Flash settings are not preserved when I click "Save". If I leave the settings page and return the setting is back to default (No).

Maybe this has already been discussed and I just missed it, but I notice that in Backup Appdata my Save Log Files To Flash settings are not preserved when I click "Save". If I leave the settings page and return the setting is back to default (No).

It actually was saving the value correctly.  It just wasn't displaying the saved value in the settings.  Small typo from a couple weeks ago

 

Fixed

I ran an Appdata backup prior to switching settings to use versioned backups. Will the original non-versioned backup be deleted when it's older than the age set in settings?

No. But additional folders (the dated ones) will be stored in there.

 

Sent from my LG-D852 using Tapatalk

 

 

Is it possible to delete un-needed backups? I don't know if it really matters or not. Didn't look to see how big it was.

When on dated backups you can select delete after x number of days and it will do it automatically.  I only keep 2 days worth. (I'm on the server every day and if I haven't noticed an issue with the cache drive within 2 days then I'm probably dead)

 

The existing non dated backup you will have to delete manually through mc or the command line or dolphin

 

My entire appdata size is only around 12gig.  But I don't have downloads (incomplete or otherwise) stored within it.  If you do then maybe you would want to exclude that folder

 

I guess that I can add a button to nuke the backup folder if need be because it is a pain to do without dropping to a command line or running a docker app due to the permissions

 

Sent from my LG-D852 using Tapatalk

 

- Added in automatic boot flash drive backup

 

Due to its minimal size, flash drive backup is automatic not optional.  A new folder will created (and updated every time) the backup process runs called Community_Applications_USB_Backup, without the config/super.dat file.

 

I'm specifically excluding the super.dat file due to the risks involved in restoring it should the drive configuration change in between backups.  If you want the system to keep track of super.dat, then back it up yourself, because CA is not going to put itself into a position that can potentially cause major data loss by backing up a super.dat (and having you restore it yourself).

 

Great addition!

 

I'd like to request though that you do add the super.dat file to the backup, but rename it.  Often, it's the one and only file that's needed from the backup.  If it's renamed, it has to be a conscious thought-out decision to use it, won't be automatically used.

 

Another idea, since the single biggest risk of the wrong super.dat is a parity drive change, why not rename the super.dat to include the model and serial of the parity drive (e.g. super.paritymodel_serial.parity2model_serial.dat).  At a glance, the user will probably know if it's OK to use.  While a slot change for a data drive or a drive addition is an important change, neither should cause data loss, and should be apparent on first array start, if it even can start.  A parity drive mistake however can be disastrous, and may not be immediately apparent.

 

I also like the idea above to create a text file with the current drive configuration, name it something like superdat.txt?

 

I assume that at some point, you'll add the ability to configure the flash backup destination path?

 

Great addition!

 

I'd like to request though that you do add the super.dat file to the backup, but rename it.  Often, it's the one and only file that's needed from the backup.  If it's renamed, it has to be a conscious thought-out decision to use it, won't be automatically used.

 

Another idea, since the single biggest risk of the wrong super.dat is a parity drive change, why not rename the super.dat to include the model and serial of the parity drive (e.g. super.paritymodel_serial.parity2model_serial.dat).  At a glance, the user will probably know if it's OK to use.  While a slot change for a data drive or a drive addition is an important change, neither should cause data loss, and should be apparent on first array start, if it even can start.  A parity drive mistake however can be disastrous, and may not be immediately apparent.

No problems.  (Although I believe that the single biggest PITA in redoing a flash from scratch is the plugin configs (especially dockerMan templates)

I also like the idea above to create a text file with the current drive configuration, name it something like superdat.txt?

Already thought about that after the previous post

I assume that at some point, you'll add the ability to configure the flash backup destination path?

Next week or so, although that pretty much requires that you would have a UD mounted volume / smb share somewhere, as any destination to the array is as good as the share for appdata backups already chosen.  Piggy backing onto the appdata backup was an easy solution to the main problems that I see on recreation -> docker templates, although restoring is a multistep process due to the need to do a trial key and initial drive setup prior to restoring. 

 

And, there's going to not be any sort of remote backup (ie: FTP).  CA Backup is and will remain a specialized backup solution for a specific problem, and its not going to evolve into a general backup system.  (Why its part of the CA banner - Although you could use it to keep 2 servers in sync by manually editing its configuration file. -> hint hint)

Is it possible to delete un-needed backups? I don't know if it really matters or not. Didn't look to see how big it was.

 

Just had the same scenario...

 

I ssh or telnet to the app data backup directory and use

 

sudo rm -rf foldername

 

FYI foldername is case sensitive and pay attention to what you're doing. It is irreversible and very powerful when accompanied with sudo.

 

 

Sent from my iPhone using Tapatalk