wsume99 Posted September 2, 2011 Share Posted September 2, 2011 I've followed all the instruction in the infamous Tibbar & WeeboTech rsync thread. I am running deltacopy client on my windows 7 desktop and I'm trying to backup photos from that machine onto my unraid server. Here is my /boot/custom/etc/rsyncd.conf file uid = root gid = root use chroot = no max connections = 4 pid file = /var/run/rsyncd.pid timeout = 600 log file = /var/log/rsyncd.log [photobackups] path = /mnt/user/photobackups comment = photobackups read only = FALSE list = yes And the contents of /boot/custom/etc/rc.d/S20-init.rsyncd #!/bin/bash if ! grep ^rsync /etc/inetd.conf > /dev/null ; then cat <<-EOF >> /etc/inetd.conf rsync stream tcp nowait root /usr/sbin/tcpd /usr/bin/rsync --daemon EOF read PID < /var/run/inetd.pid kill -1 ${PID} fi I setup a new profile named temp to test out the system and here is the response I get Executing: rsync.exe -v -rlt -z --chmod=a=rw,Da+x --delete "/cygdrive/C/Users/John/Documents/motherboard_manual_ga-ma790gp-ud4h_e.pdf" "192.168.1.150::photobackups/Documents/motherboard_manual_ga-ma790gp-ud4h_e.pdf" Profile 'temp' executed in 2 milliseconds. It ran successfully. The only problem is that there are no files in my photobackups share. I'm guessing that I don't have the path to my share setup correctly but it looks right to me. I've also attached a screenshot of the deltacopy profile I created. Any ideas? Thanks in advance. Quote Link to comment
wsume99 Posted September 2, 2011 Author Share Posted September 2, 2011 Well I've searched all over the unraid forums and the web and I can't seem to find any answer. Hopefully someone will help me out? Quote Link to comment
cyrnel Posted September 3, 2011 Share Posted September 3, 2011 This line: /cygdrive/C/Users/John/Documents/motherboard_manual_ga-ma790gp-ud4h_e.pdf" "192.168.1.150::photobackups/Documents/motherboard_manual_ga-ma790gp-ud4h_e.pdf" You only want the one file? If there's a /mnt/user/photobackups directory on unRAID, does it have a documents directory inside containing the PDF? If the file hasn't changed, any rsyncs after the first would finish almost instantly, so the 2 milliseconds message wouldn't be surprising. What do you see in /var/log/rsyncd.log? Quote Link to comment
wsume99 Posted September 3, 2011 Author Share Posted September 3, 2011 I'm only copying a single file right now to make sure everything works properly. I did try copying over the directory I actually plan on backing up and I got the same result. /mnt/user/photobackups is completely empty. I manually created a Documents folder in /mnt/user/photobackups just to see if that made a difference and it did not. Here are the contents of /var/log/rsync.log (192.168.1.151 is the IP address of the client PC I'm copying from): 2011/09/01 22:34:21 [1578] name lookup failed for 192.168.1.151: Name or service not known 2011/09/01 22:34:21 [1578] connect from UNKNOWN (192.168.1.151) 2011/09/01 22:34:21 [1578] module-list request from UNKNOWN (192.168.1.151) 2011/09/01 22:35:11 [1580] name lookup failed for 192.168.1.151: Name or service not known 2011/09/01 22:35:11 [1580] connect from UNKNOWN (192.168.1.151) 2011/09/01 22:35:11 [1580] module-list request from UNKNOWN (192.168.1.151) 2011/09/01 22:35:13 [1581] name lookup failed for 192.168.1.151: Name or service not known 2011/09/01 22:35:13 [1581] connect from UNKNOWN (192.168.1.151) 2011/09/01 22:35:13 [1581] module-list request from UNKNOWN (192.168.1.151) 2011/09/01 22:47:56 [1883] name lookup failed for 192.168.1.151: Name or service not known 2011/09/01 22:47:56 [1883] connect from UNKNOWN (192.168.1.151) 2011/09/01 22:47:56 [1883] module-list request from UNKNOWN (192.168.1.151) 2011/09/02 16:58:47 [20828] name lookup failed for 192.168.1.151: Name or service not known 2011/09/02 16:58:47 [20828] connect from UNKNOWN (192.168.1.151) 2011/09/02 16:58:47 [20828] module-list request from UNKNOWN (192.168.1.151) 2011/09/02 16:59:08 [20893] name lookup failed for 192.168.1.151: Name or service not known 2011/09/02 16:59:08 [20893] connect from UNKNOWN (192.168.1.151) 2011/09/02 16:59:08 [20893] module-list request from UNKNOWN (192.168.1.151) 2011/09/02 17:39:55 [25142] name lookup failed for 192.168.1.151: Name or service not known 2011/09/02 17:39:55 [25142] connect from UNKNOWN (192.168.1.151) 2011/09/02 17:39:55 [25142] module-list request from UNKNOWN (192.168.1.151) 2011/09/02 17:39:57 [25143] name lookup failed for 192.168.1.151: Name or service not known 2011/09/02 17:39:57 [25143] connect from UNKNOWN (192.168.1.151) 2011/09/02 17:39:57 [25143] module-list request from UNKNOWN (192.168.1.151) Based on the timestamps I believe that all these messages are generated when I'm creating the profile in Deltacopy. When I actually run the profile nothing is documented in the log file. Instead I get the message in the Deltacopy window that I attached in my previous post that says it completed successfully. Quote Link to comment
cyrnel Posted September 3, 2011 Share Posted September 3, 2011 The results you showed earlier: Executing: rsync.exe -v -rlt -z --chmod=a=rw,Da+x --delete "/cygdrive/C/Users/John/Documents/motherboard_manual_ga-ma790gp-ud4h_e.pdf" "192.168.1.150::photobackups/Documents/motherboard_manual_ga-ma790gp-ud4h_e.pdf" Profile 'temp' executed in 2 milliseconds. It ran successfully. Was that everything that was in the deltacopy status window? That's tells me it sees the file at the destination already. Try switching things up. Pick a different source file or even use a smallish folder. Change the destination to something different in your rsyncd module. Change options in the deltacopy profile so that only verbose logging and recursive/preserve time are set. Now what do you get in the status window? Does anything show up at /mnt/user/whatever? Quote Link to comment
cyrnel Posted September 3, 2011 Share Posted September 3, 2011 I just installed deltacopy again, after many a moon. Now I remember why I use command-line rsync. Anyway, how are you running the profile? It works fine for me when I use "run now. I tried "display run command" instead and got: Executing: rsync.exe -v -rlt -v "/cygdrive/C/Users/Me/Desktop/filetest/" "tower::backupst/filetest/" Profile 'testb' executed in 1 milliseconds. It ran successfully. Looks familiar. Quote Link to comment
wsume99 Posted September 3, 2011 Author Share Posted September 3, 2011 I just installed deltacopy again, after many a moon. Now I remember why I use command-line rsync. If you know of a better way to backup files from a windows machine to the server then I'm all ears. I just want to automate the process of backing up my photos to the server. Anyway, how are you running the profile?I was using only the "Display Run Command" option. I tried the "Run Now" method and it worked with my test file. I knew I was doing something really dumb. Now let's see if it'll work as a scheduled task. Thanks a lot for the help! Quote Link to comment
cyrnel Posted September 4, 2011 Share Posted September 4, 2011 If you know of a better way to backup files from a windows machine to the server then I'm all ears. Command-line rsync. http://www.itefix.no/i2/node/10650 Really, once you see how a few of those options work it isn't any more difficult, and behavior is definitely simpler to control via scripting, for backup rotation, versioning, determining if you're on the right network, etc. My production rsync setups are mostly done through universal scripts located on the server that assume minimum backup sources, then add include/exclude pattern files customized for and located on each client. I tried the "Run Now" method and it worked with my test file. I knew I was doing something really dumb. Now let's see if it'll work as a scheduled task. Thanks a lot for the help! Always nice when it's something simple. Quote Link to comment
wsume99 Posted September 4, 2011 Author Share Posted September 4, 2011 Command-line rsync. http://www.itefix.no/i2/node/10650 Really, once you see how a few of those options work it isn't any more difficult, and behavior is definitely simpler to control via scripting, for backup rotation, versioning, determining if you're on the right network, etc. My production rsync setups are mostly done through universal scripts located on the server that assume minimum backup sources, then add include/exclude pattern files customized for and located on each client. Ok. I'll check that out. I assume there is a way to make the script wake up the client PC if it is in sleep mode (S3). Sounds like it's time to do some more reading. Quote Link to comment
cyrnel Posted September 4, 2011 Share Posted September 4, 2011 There's no auto-wake with cwrsync. It's a raw tool with a few batch samples. However, you will recognize everything you've done with deltacopy with many more options available. To wake a system you can use wake-on-lan, triggered from your server, or set systems to auto-boot from the BIOS shortly before the scheduled task. There are countless tools to send WOL magic packets. Here's a start: http://www.wlug.org.nz/WakeOnLanNotes I use etherwake on nix but it's usually wolcmd on Windows since that's what I'm working with at client sites. http://www.depicus.com/wake-on-lan/wake-on-lan-cmd.aspx Because of the hardware requirements and variety it's rare to get every system working with WOL. That means using the BIOS auto-boot facility, which means changing the schedule is a pain. Do everything possible with WOL and then BIOS auto-boot where you have to, spreading systems out across the available backup time. Quote Link to comment
wsume99 Posted September 4, 2011 Author Share Posted September 4, 2011 Well now when I setup a scheduled task in Deltacopy it does not execute so I've decided to abandon that method and use cwrsync. My plan is to create a script that will wake my windows pc and then run a rsync job on the server. I'll load the script as a cron job so it'll run regularly. Does this make sense? I found this post where WeeboTech posted a compiled version of ether-wake. I copied the file onto my flash drive and I can now wake my windows machine over the LAN. I also installed cwrsync server onto my windows machine. I then modified the rsync.conf file on the windows machine and added a "test" module. Here are the contents of the rsync.conf file: use chroot = false strict modes = false hosts allow = * log file = rsyncd.log # Module definitions # Remember cygwin naming conventions : c:\work becomes /cygwin/c/work # [test] path = /cygdrive/c/Users/John/Downloads read only = true transfer logging = yes As a test I tried to run rsync from the server and copy the contents of c:/Users/John/Downloads to the photobackups share on the server. Here is the command I used: rsync -v -rlt -z --chmod=a=rw,Da+x --delete 192.168.1.152::test /mnt/user/photobackups/ I received the following response: root@Tower:~# rsync -v -rlt -z --chmod=a=rw,Da+x --delete 192.168.1.152::test /m nt/user/photobackups/ @ERROR: invalid uid nobody rsync error: error starting client-server protocol (code 5) at main.c(1495) [rec eiver=3.0.2] I also tried this command thinking I needed to have a user account identified on the windows pc: rsync -v -rlt -z --chmod=a=rw,Da+x --delete [email protected]::test /mnt/user/photobackups/ But I got the same error as before. What am I doing wrong here? Quote Link to comment
cyrnel Posted September 4, 2011 Share Posted September 4, 2011 I prefer using cwrsync as a client on backup clients but it'll work either way. I also installed cwrsync server onto my windows machine. I then modified the rsync.conf file on the windows machine and added a "test" module. Here are the contents of the rsync.conf file: rsync.conf or rsyncd.conf? Sticking with convention of rsyncd.conf for the server, it needs uid & gid = 0 to play well. uid = 0 gid = 0 use chroot = false strict modes = false hosts allow = * log file = rsyncd.log ... I'd start over with the rsync options. Try -a (useful combo), -P (progress and partial), and as many v's as are needed/interesting. -z is compression and doesn't usually help over a LAN though it does work your cpus. rsync -avP maybe with --delete Look into -b, --backup-dir, --include-from and --exclude-from to help manage things. Quote Link to comment
wsume99 Posted September 4, 2011 Author Share Posted September 4, 2011 I prefer using cwrsync as a client on backup clients but it'll work either way.Is that just a personal preference or is there reason using cwrsync as a client on the windows pc is superior? rsync.conf or rsyncd.conf? Sticking with convention of rsyncd.conf for the server, it needs uid & gid = 0 to play well.It was rsyncd.conf. I added the uid & gid = 0 to the file and it worked. I used rsync -avP --delete and it worked fine, except it did not copy any of the files that were in the source folder that had a .exe extension. I was thinking that it would be an exact duplicate of the source but it wasn't. I guess I'll need to start experimenting with the options. Thank you for all your assistance. Quote Link to comment
cyrnel Posted September 5, 2011 Share Posted September 5, 2011 Might be blood sugar and sunshine but that's odd. -C would do it. What's the entire command line? About using client vs. server, it's mostly how you think about things. I've never liked publishing entire drives for external access or leaving extra code running. But those aren't exactly show stoppers. Either way will work fine, each with its own flavor. Quote Link to comment
wsume99 Posted September 5, 2011 Author Share Posted September 5, 2011 The command I used was rsync -avP --delete 192.168.1.152::test /mnt/user/photobackup I've been running more tests and I've determined that the files are there but for whatever reason I cannot see them in windows explorer. I can see all the files if I use the file browser in unmenu. Quote Link to comment
cyrnel Posted September 5, 2011 Share Posted September 5, 2011 Ah, the joys of Windows vs. nix permissions and Samba... Try adding the option "--chmod=Fo-X" to your rsync command. That keeps the other execute bit (001) clear for files. Samba maps it to the hidden attribute. rsync -avP --delete --chmod=Fo-X 192.168.1.152::test /mnt/user/photobackup Quote Link to comment
wsume99 Posted September 5, 2011 Author Share Posted September 5, 2011 I added the --chmod=Fo-X option to the command and now the .exe files are visible in windows explorer. Thanks so much. I'm actually beginning to like the CLI. I have another question. I assume that only one rsync instance can be running at a time between the client and the server. Is this correct? For example, if I wanted to backup c:/users/Ziggy/Downloads/ and c:/users/Ziggy/My Pictures/ I would have to write a single script like this: #!/bin/bash rsync -avP --delete --chmod=Fo-X 192.168.1.152::test /mnt/user/photobackups/Downloads/ rsync -avP --delete --chmod=Fo-X 192.168.1.152::test1 /mnt/user/photobackups/My Pictures/ Where rsyncd.conf defines test as /cygdrive/c/Users/Ziggy/Downloads/ and test1 as /cygdrive/c/Users/Ziggy/My Pictures/. I could then place the script into /etc/cron.daily/ and it should execute a backup once a day of the two folders, first test and then test1. Is that correct? Quote Link to comment
cyrnel Posted September 5, 2011 Share Posted September 5, 2011 Multiple transfers can run simultaneously, up to the limits of max connections (in rsyncd.conf) and your system. Really though, it makes little sense to fill memory and thrash your disks between just two boxes. Better to do them serially as you are now. I spoke earlier about not publishing an entire drive, but I'll go against that for now. You can make your setup simpler by backing out the module one or more levels. e.g. Change the module path for [test] to /cygdrive/c/Users/John. Then, in your rsync command, use: rsync -avP --delete --chmod=Fo-X 192.168.1.152::test/Downloads/ /mnt/user/photobackups/Downloads/ If you plan on more than a few paths, better yet would be to put it in a loop using include-from to pull from a file containing your list of backup sources. Then add exclude-from to filter out the unwanted. Echoing myself, you might also look into the -b and --backup-dir options which make versioning simple. Create a timestamp, use it in a directory name for --backup-dir, and anything that's changed or been removed will be moved there instead of being lost with the --delete option. Quote Link to comment
wsume99 Posted September 5, 2011 Author Share Posted September 5, 2011 Now I'm having trouble with the spaces in the path /mnt/user/photobackups/My Pictures/. I had this issue before but I just eliminated the spaces from my test directories. That is not really a solution because I don't want to have to rename all my existing directories. I've tried all of the variations below and it still won't work? root@Tower:/etc/cron.hourly# rsync -avP --delete --chmod=Fo-X 192.168.1.152::tes t1 /mnt/user/photobackups/My Pictures/ Unexpected local arg: /mnt/user/photobackups/My If arg is a remote file/dir, prefix it with a colon (. rsync error: syntax or usage error (code 1) at main.c(1207) [receiver=3.0.2] root@Tower:/etc/cron.hourly# rsync -avP --delete --chmod=Fo-X 192.168.1.152::tes t1 "/mnt/user/photobackups/My Pictures/" @ERROR: chdir failed rsync error: error starting client-server protocol (code 5) at main.c(1495) [rec eiver=3.0.2] root@Tower:/etc/cron.hourly# rsync -avP --delete --chmod=Fo-X 192.168.1.152::tes t1 '/mnt/user/photobackups/My Pictures/' @ERROR: chdir failed rsync error: error starting client-server protocol (code 5) at main.c(1495) [rec eiver=3.0.2] root@Tower:/etc/cron.hourly# rsync -avP --delete --chmod=Fo-X 192.168.1.152::tes t1 /mnt/user/photobackups/My\ Pictures/ @ERROR: chdir failed rsync error: error starting client-server protocol (code 5) at main.c(1495) [rec eiver=3.0.2] root@Tower:/etc/cron.hourly# rsync -avP --delete --chmod=Fo-X 192.168.1.152::tes t1 /mnt/user/photobackups/My"\" Pictures/ I think I will look into the -b and --backup-dir options. I like the thought of having older versions of files for maybe a month or so. But first thing for me is to get the automation working the way I need it to, then I can play with the options. I never imagined that I would need to do so much experimenting with test files - silly me. EDIT: I'm such a noob. All the options I tried did work - well at least the space in the path wasn't the problem. The first error was a result of the space in the path but the rest of them were a problem with the test1 module I've created. I'm not sure what's wrong with it yet but I'll have to look into it. Also, I found that /mnt/user/photobackups/"My Pictures"/ syntax works as well. Quote Link to comment
cyrnel Posted September 5, 2011 Share Posted September 5, 2011 I think libraries are getting in your way. There isn't a /users/name/My Pictures from cygwin's perspective. Try navigating from the windows command line to see what cwrsync sees. Just "Pictures" should do. rsync -avP --delete --chmod=Fo-X 192.168.1.152::test1 /mnt/user/photobackups/Pictures/ Quote Link to comment
wsume99 Posted September 5, 2011 Author Share Posted September 5, 2011 The problem appears to be the spaces in the path for the module I defined in the rsyncd.conf file on the windows machine. I have the following: uid = 0 gid = 0 use chroot = false strict modes = false hosts allow = 192.168.1.150 log file = rsyncd.log # Module definitions # Remember cygwin naming conventions : c:\work becomes /cygwin/c/work # [test] path = /cygdrive/c/Users/John/"My Pictures"/ read only = true transfer logging = yes Again the problem is the fact that there are spaces in the path. The folder c:\Users\John\My Pictures\ does exist on the machine. I've read quite a few posts with similar issues but haven't seen a solid solution yet. Quote Link to comment
cyrnel Posted September 5, 2011 Share Posted September 5, 2011 Can you paste the command prompt output from: dir c:\Users\John Quote Link to comment
wsume99 Posted September 5, 2011 Author Share Posted September 5, 2011 Ah Ha - I guess I should have listened better when you warned me about libraries but I didn't think that was the problem because I could see the My Pictures folder in windows explorer. I guess the lesson I should learn here is not to trust what windows explorer says. C:\Users\John>dir Directory of C:\Users\John 07/14/2011 03:18 AM <DIR> . 07/14/2011 03:18 AM <DIR> .. 10/10/2010 01:27 PM <DIR> .idlerc 04/12/2011 10:04 PM <DIR> Contacts 09/04/2011 10:55 PM <DIR> Desktop 07/16/2011 02:32 PM <DIR> Documents 08/19/2011 09:05 PM <DIR> Downloads 04/12/2011 10:04 PM <DIR> Favorites 04/12/2011 10:04 PM <DIR> Links 02/23/2011 07:43 PM <DIR> Logitech 04/12/2011 10:04 PM <DIR> Music 09/01/2011 11:00 PM <DIR> Pictures 04/12/2011 10:04 PM <DIR> Saved Games 04/12/2011 10:04 PM <DIR> Searches 03/14/2010 05:58 PM 0 Sti_Trace.log 04/12/2011 10:04 PM <DIR> Videos 09/12/2010 04:51 PM 99 wakeup.bat 2 File(s) 99 bytes 15 Dir(s) 13,327,364,096 bytes free So according to the dir listing there is no My Pictures folder. The proper path should be c:/Users/John/Pictures/. So the rsyncd.conf file should have a module named test1 with this path: path = /cygdrive/c/Users/John/Pictures/ and the rsync command would be: rsync -avP --delete --chmod=Fo-X 192.168.1.152::test1 /mnt/user/photobackups/"whatever I want to call it but I need quotes if there are spaces"/ Quote Link to comment
cyrnel Posted September 6, 2011 Share Posted September 6, 2011 I well remember my own shock at seeing that the first time. I'm sure it's all about compatibility. With software, not us. Again, I'd publish the module as /cygdrive/c/Users/John, or even higher up the tree so can get whatever you like without adding more modules. e.g.: If module uses rsync uses /cygdrive/c/Users/John/Pictures address::module /cygdrive/c/Users/John address::module/Pictures /cygdrive/Users address::module/John/Pictures ... If you get to the point of looping through your list of sources vs. hard-coding individual rsync lines, it's less confusing to have one point of origin. rsync -avP --delete --chmod=Fo-X 192.168.1.152::test1 /mnt/user/photobackups/"whatever I want to call it but I need quotes if there are spaces"/ Yep, quotes or escape them with \. Quote Link to comment
dalben Posted September 6, 2011 Share Posted September 6, 2011 I've been watching this thread with interest. I've trolled Google and can't find anywhere that explains the syntax and usage of the --chmod switch. If you could point me towards a page or reference I'd be most grateful. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.