stignz

Sorry, I haven't visited in a while. Ayefly: Glad you worked though most of the problems. I'm not sure why you are getting the duplicate folders. If you want to see progress in the telnet session then just run the copy command from your rcloneCopy script, and add the option for verbose. Verbose which will give you much more detail and may give more clues to the folder duplication i.e. "nohup rclone copy -v /mnt/user/Photos AmazonCloudDrive:Photos &" If it a long runtime is expected, you can run it with NoHangUp so that you can close the session and the command will keep running, and you can fetch the nohup.out file later. nblain: The first line creates the logfile, so all other references to it on the following lines are invalid. If you used Notepad++ to edit the script, make sure that the format is 'UNIX' (right click, near bottom right corner) As Ayefly found out, if it is Macintosh or Windows then the line endings will be invalid in bash. Else, has the script has been made executable? e.g. "chmod u+x /boot/rclone/scripts/rcloneCopy.sh" If the above is all correct, can you let me know which version of unRAID you are running? I'm thinking we may need to specify the shell in the first line of the script. Edit: It looks like rclone now has logging built into it. I'll have a play and update.

This is a guide of how I use 'rclone' to backup/sync folders from my unRAID 5 Tower to my cloud storage. For various reasons CrashPlan was no longer suiting my needs, so I went looking for an alternative solution. I use it with AmazonCloudServices but it will work for others: At the time of writing: DropBox; OneDrive; GoogleDrive; Amazon S3; Backblaze B2; and many more. Refer to http://rclone.org for further details. I'm not intimate with Linux and Slackware, but I have done everything below from the unRAID command line for clarity. I have broken it down into Installation, Configure, Usage, and making sure it re-installs itself on system reboot. Installation: # make supporting directory structure on flash drive mkdir -p /boot/rclone mkdir -p /boot/rclone/install mkdir -p /boot/rclone/scripts mkdir -p /boot/rclone/logs #download dependencies to /boot/rclone/install wget http://mirrors.slackware.com/slackware/slackware-13.1/slackware/ap/man-1.6f-i486-2.txz -O //boot/rclone/install/man-1.6f-i486-2.txz wget http://mirrors.slackware.com/slackware/slackware-13.1/slackware/a/infozip-6.0-i486-1.txz -O //boot/rclone/install/infozip-6.0-i486-1.txz curl -o /boot/rclone/install/ca-certificates.crt https://raw.githubusercontent.com/bagder/ca-bundle/master/ca-bundle.crt #Install dependencies (skip infozip if you already have it installed) installpkg /boot/rclone/install/man-1.6f-i486-2.txz installpkg /boot/rclone/install/infozip-6.0-i486-1.txz #Download rclone and unzip wget http://downloads.rclone.org/rclone-current-linux-386.zip -O //boot/rclone/install/rclone-current-linux-386.zip unzip /boot/rclone/install/rclone-current-linux-386.zip #copy binary file cp /boot/rclone/install/rclone-v*/rclone /usr/sbin/ chown root:root /usr/sbin/rclone chmod 755 /usr/sbin/rclone #install manpage (makewhatis is also man-db on some Linux distributions) mkdir -p /usr/local/share/man/man1 cp /boot/rclone/install/rclone-v*/rclone.1 /usr/local/share/man/man1/ makewhatis #copy ca-certificates mkdir -p /etc/ssl/certs/ cp /boot/rclone/install/ca-certificates.crt /etc/ssl/certs/ Okay, rclone is now installed. Configuring: Configuring rclone to connect to your cloud provider is covered in the documentation at http://rclone.org/docs You are going to create a "Remote" which contains all the connection details and credentials to connect to your cloud service. You can now run "rclone -config" to set it up from the command line. This will generate a ".rclone.conf" file in the /root/ directory. you will want to make a copy of this to the flash so you can re-install on reboot. #cp /root/.rclone.conf /boot/rclone/install/ You could also install rclone on another Windows/Linux/OSx machine which has a gui, generate this file, and then copy it to the "/boot/rclone/install/" directory. Then run 'cp /boot/rclone/install/.rclone.conf /root/' to copy it to the rclone config default directory. Note: Some cloud providers require authentication via a web browser, so this is the only way to generate the config file. Useage: To test that the config is loaded properly Run "rclone -config" again and you should see your remote. Run "rclone size <RemoteName>:<FolderPath>" and it should spit out the number of files and size. rclone has two modes, copy, and sync. (both uni directional) 'Copy' will just copy anything from the source directory to the remote directory, if it doesn't exist in the remote directory. 'Sync' will sync the two directories, i.e. if it finds something in the remote directory that is not in the source directory, it will delete it. #to start a copy from a folder in an unraid share: rclone copy /mnt/user/<ShareName>/<FolderName> <RemoteName>:<RemoteFolderName> #likewise for a sync: rclone sync /mnt/user/<ShareName>/<FolderName> <RemoteName>:<RemoteFolderName> (If you are doing a long running test from putty then use "nohup <command> &" to prevent a hangup on the running command when you exit the client.) Running it manually may be of little value to you, so now we make a script and add it to crontab. I made a script called "rcloneCopy.sh", in the /boot/rclone/scripts folder, with the following inside: (The logfiles are optional but would be useful for trouble shooting and checking up on it etc.) LOGFILE=/boot/rclone/logs/rclone-$(date "+%Y%m%d").log echo rclone log $(date) $'\r'$'\r' >> $LOGFILE 2>&1 echo "Starting rclone copy" $'\r'>> $LOGFILE 2>&1 rclone copy /mnt/user/<ShareName>/<FolderName> <RemoteName>:<RemoteFolderName> >> $LOGFILE 2>&1 #To make it executable: chmod u+x /boot/rclone/scripts/rcloneCopy.sh #To add it to crontab to be run at midnight, everynight: crontab -l | { cat; echo "0 0 * * * /boot/rclone/scripts/rcloneCopy.sh > /dev/null 2>&1"; } | crontab - If you are doing an initial bulk upload and don't want it using bandwidth during the day then the following script inside "rcloneStop.sh" would stop it: LOGFILE=/boot/rclone/logs/rclone-$(date "+%Y%m%d").log PROCESSID=$(pgrep rclone) if [ -n $PROCESSID ]; then echo "Attempting to terminate rclone" $'\r'>> $LOGFILE 2>&1 echo "processid of rclone: $PROCESSID" $'\r' >> $LOGFILE 2>&1 kill -s 15 $(pgrep rclone) echo "rclone job termination requested by cron at "$(date "+%T") $'\r'>> $LOGFILE 2>&1 fi #again make it executable, and add to crontab to be run at say 8am: chmod u+x /boot/rclone/scripts/rcloneStop.sh crontab -l | { cat; echo "0 8 * * * /boot/rclone/scripts/rcloneStop.sh > /dev/null 2>&1"; } | crontab - Auto Re-install on system restart Lastly, we don't want to do this again when unRAID is shutdown and restarted. The following code inside the script "rcloneStartup.sh" would auto install and set everything running again. sleep 5m && #Install dependancies installpkg /boot/rclone/install/man-1.6f-i486-2.txz installpkg /boot/rclone/install/infozip-6.0-i486-1.txz #Download fresh version of rclone wget http://downloads.rclone.org/rclone-current-linux-386.zip -O //boot/rclone/rclone-current-linux-386.zip #check if file downloaded if [ -f "/boot/rclone/rclone-current-linux-386.zip" ] then #remove existing rclone zip rm /boot/rclone/install/rclone-current-linux-386.zip rm -rf /boot/rclone/install/rclone-v* #move new rclone zip to install directory mv /boot/rclone/rclone-current-linux-386.zip /boot/rclone/install #unzip the rclone download into the install directory unzip /boot/rclone/install/rclone-current-linux-386.zip -d /boot/rclone/install/ fi #Install rclone #copy binary file cp /boot/rclone/install/rclone-v*/rclone /usr/sbin/ chown root:root /usr/sbin/rclone chmod 755 /usr/sbin/rclone #install manpage mkdir -p /usr/local/share/man/man1 cp /boot/rclone/install/rclone-v*/rclone.1 /usr/local/share/man/man1/ makewhatis # copy config file to default location cp /boot/rclone/install/.rclone.conf /root/ # set ssl certs mkdir -p /etc/ssl/certs/ cp /boot/rclone/install/ca-certificates.crt /etc/ssl/certs/t #make any scripts executable chmod u+x /boot/rclone/scripts/rcloneCopy.sh # set cron job, this will run at 1 min past midnight every night crontab -l | { cat; echo "1 0 * * * /boot/rclone/scripts/rcloneCopy.sh > /dev/null 2>&1"; } | crontab - Then add the following line to the end of your "go" file: chmod u+x /boot/rclone/scripts/rcloneStartup.sh | /boot/rclone/scripts/rcloneStartup.sh I have attached the scripts. Hopefully my version of how to execute this makes enough sense. Any suggestions to improve on the above would be welcome as well. I'd like to add functionality to email monthly stats at a later date. Lastly, a very big thank you to the writer of rclone. It has solved a massive problem for me. rcloneCopy.sh.txt rcloneStartup.sh.txt