Copy data from unraid 5 to unraid b6

Hi,

I have been told in the forum to copy the data either over lan or moving HDD by HDD to the new array and then using snap or mounting the HDD manually.

Also, that I should check the copied files for integrity after they have been copied.

I also have read about mounting the share to be copied in the new unraid (where I previously have created the same share) and then copy it using rsync or similar command.....

As you can read there are lots of options. Which is the faster and most reliable way to do it?

My lan is gigabit.

Rgds.

Hi -

 

Are the old hard drives going to be part of the new array, or will the new array have a new set of HDs?

I had to go looking for your other posts about this on another thread. Give us a link next time.

 

Still not clear what you want to do though.

 

Do you intend to have 2 servers, keeping the disks in the old server and just getting the data copied from it to new disks in the new server?

 

Or do you intend to have only the new server when you're finished, and re-use the disks from the old server in the new server. In which case it is not necessary to copy anything at all since you can just add the existing drives with their data to the new server.

 

 

Hi, sorry for not posting the link. I want to migrate all HDD to the new array.

Adding the disks is not possible since they need to be converted from btrfs to xfs.

Adding the disks is not possible since they need to be converted from btrfs to xfs.

I think you can still add btrfs drives to the array.

 

Then you could add a new XFS drive to the array, and copy the files from one of the btrfs drives to the new XFS drive.

 

Then when you are satisfied that the copy was successful, you could format that btrfs drive to XFS, thus creating an empty XFS drive to copy the next btrfs drive to.

 

And repeat as needed.

 

This would even work with a parity drive in the array.

 

 

Sounds good trurl! I will go this route. Garycase wrote in the other post:  >Just copy the data across the network from the old server to the new one. After the copy's done, you can either keep the old server as your backup; or just reformat the disks in whatever file system you want. Use teracopy in verify mode or create md5sums of the whole disk and verify it before reformatting the original disks. Frankly I prefer rsync, yet in any case, do something that verifies the source and destination before erasing the source..........Pls, what are the CLI commands for that?

 

Hi, let me try to answer myself....

>Source server: share to be copied is DATA

>Target server: I create a share with the same name as the one in the source server

> rsync -aruv /mnt/user/data 192.168.1.103:/mnt/user/data

 

Pls, can you confirm the above command is correct. I am new to Linux and this is the first time I read about rsync. I just checked in internet and  it has lots of options...and the command i posted was extracted from a post by bobkart from 18 dec.

http://lime-technology.com/forum/index.php?topic=35646.0

 

Anything else I should consider? NFS: yes and public.

 

And finally how would you check files were copied correctly?

Hi, pls I need help with some doubts I still have.

 

If I mount the old drives (old array) one-at-a-time in the new array and copy the data to the new server (using MC):

 

> what write speed would I achieve compared to transfering files over ethernet gigabit from the old to the new server? hdd specs are: 7200rpm, 64MB buffer, 2TB, sata 6gb

 

> files/directories are going to be copied with the same permissions as the original ones in the old array?

 

Also, when I stop the old array and bring the disk out (to the new array) what happens to the array when I start it again without this disk (which is going to stay in the new array)?

 

Rgds.

 

 

 

 

...Also, when I stop the old array and bring the disk out (to the new array) what happens to the array when I start it again without this disk (which is going to stay in the new array)?

Why do you want to start the old array? Do you plan to keep it with some disks still in it? I thought you were moving all the disks to the new server.

 

If you do need to run the old server after removing a disk, then you will have to do a new config on it and let it do a parity sync. You could also run it without parity in which case no parity sync would be needed, but I would suggest not writing to it in that case.

 

What do you need to do with the old server other than get the disks out? If you don't need to run it then don't.

Why do you want to start the old array? Do you plan to keep it with some disks still in it? I thought you were moving all the disks to the new server.

I will move all data from it to the new array, but maybe in a future use it as backup so maybe I leave only 2 disks in it (1 parity and 1 disk with my data (not movies)).

If you do need to run the old server after removing a disk, then you will have to do a new config on it and let it do a parity sync.

What do you mean with doing a new config? If I run a parity sync -after removing 1 drive and not installing 1 new drive- will this parity check run smooth or will this cause any trouble?

 

Regarding moving all the disks to the new array, I guess I should run a reset config in Unraid-Tools before (in the new server), then switch off and connect all the drives from the old array, start it and then what should I do first? Take into account I have already connected 2 new drives (1 3TB parity and 1 2TB disk) in the new array. The next step will be assigning the old and new disks, and then running a parity check? Does the order of how I connect the drives affect this first-time parity check?

 

I do not want to loose any data, sorry for asking so much but when I started running unraid several years ago, I lost a couple of times data due to not asking enough and assuming I knew things

 

If you do need to run the old server after removing a disk, then you will have to do a new config on it and let it do a parity sync.

What do you mean with doing a new config? If I run a parity sync -after removing 1 drive and not installing 1 new drive- will this parity check run smooth or will this cause any trouble?

New Config in the Tools page. When you set a new config it will rebuild parity unless you check the trust parity box. In this case don't check it because parity needs to rebuild.

Regarding moving all the disks to the new array, I guess I should run a reset config in Unraid-Tools before (in the new server), then switch off and connect all the drives from the old array, start it and then what should I do first? Take into account I have already connected 2 new drives (1 3TB parity and 1 2TB disk) in the new array. The next step will be assigning the old and new disks, and then running a parity check? Does the order of how I connect the drives affect this first-time parity check?

If you add drives that contain data you will have to do a New Config and let parity rebuild. If you just add them without New Config unRAID will clear them.

 

So, stop the array, and instead of selecting the drives into new slots, go to the Tools page and do New Config. You will have to assign all drives because it will not remember. Make sure you assign them correctly, especially the parity drive. If you accidentally assign a data drive to the parity slot and start the array, unRAID will write parity to it and you will be sorry.

Thx trurl, exactly the info i needed. I will go this route. I hope everythings works fine.

Hi,

 

I finally started migrating my data from unraid 5 to 6. I used Teracopy for copying disk_2. I highly recommend it since you just check the "verify (after copy)" and it automatically checks for any files/folders that were not copied. It is a simple copy and paste functionality within windows with that added security (and it is free).

 

Now I was wondering about how should I proceed to take outside of my Unraid 5 array the copied disk_2. I will install it in an external usb case (so I do not plan to use it further in the array).

 

I plan to copy disk by disk to my new Unraid 6 array. And after each disk is copied I will take it outside the Unraid 5 array and install it in the new Unraid 6 array. But I do not want to loose parity on my old Unraid 5.

 

How should I proceed?

Hi,

 

I finally started migrating my data from unraid 5 to 6. I used Teracopy for copying disk_2. I highly recommend it since you just check the "verify (after copy)" and it automatically checks for any files/folders that were not copied. It is a simple copy and paste functionality within windows with that added security (and it is free).

 

Now I was wondering about how should I proceed to take outside of my Unraid 5 array the copied disk_2. I will install it in an external usb case (so I do not plan to use it further in the array).

 

I plan to copy disk by disk to my new Unraid 6 array. And after each disk is copied I will take it outside the Unraid 5 array and install it in the new Unraid 6 array. But I do not want to loose parity on my old Unraid 5.

 

How should I proceed?

There is a (somewhat theoretical) command line technique for zeroing a drive in a parity array and so maintain parity when you remove it, but I have not done it and it is not for the faint of heart. Not sure how successful people have been with it.

 

Why do you want to maintain parity on the old server? Does it still need to be used during and after all of this? Do you have backups of this data, at least the important stuff?

 

Are you sure you want to continue to do this over the network instead of just adding the old drives with their data intact to the new server? You can't remove a disk without doing a new config on the old server and rebuilding parity (except as I mentioned above). You will also have to clear each old disk to add it to an array with parity on the new server.

 

If you plan to move the rest of the drives to the new server, you could just take them out of the old server, put them all in the new server and do a new config. That way you just do one parity rebuild, and then you can move the data around however you want on the new server and you will never lose parity. Seems a lot simpler, quicker, and less disk activity. That is the way I would do it.

 

The only caution I would mention if you take this approach is you must make sure that you do a new config instead of just adding the drives to the array, or unRAID will clear them.

 

Whatever you decide, ask if you are unsure what to do at any point.

If you plan to move the rest of the drives to the new server, you could just take them out of the old server, put them all in the new server and do a new config.

 

Ok, you are right ... it seems the best way to do it. Pls, let me know the exact steps to follow. I guess...

 

1) when turn on the array, I guess it should not be mounted to perform the new config command?

2) after new_config I just assign the disks, right?

3) can I assign a different (bigger) disk to my parity slot (which will replace the actual parity drive)? is yes, I just need to strat the array and

4) let unraid rebuild the parity, right?

 

Thx for support.

unRAID will see that the disks are different and it won't start the array.

 

Yes, this would be a good time to put a larger parity drive in.

 

As usual, the main thing when doing new config is make sure you assign the disks correctly. If you assign a data drive to the parity slot unRAID will overwrite it with parity.

 

After you have assigned the disks (and don't check Trust Parity) you can start the array and unRAID will build parity.

 

If you want to change the format of the disks (to XFS), then I'd just keep doing it over the network, copying one disk at a time from the old array to the new one; then removing the copied disk, adding it to the new array, etc.

 

The reason is twofold:  (1)  It may take a bit longer; but it's less prone to any errors; and (2) if you put all the disks in the new array, then start moving data around to reformat the disks in XFS, you have to be VERY careful that you don't copy to a user share, or you can lose all of the data involved (search for the "User Share Bug" if you want the details on this).

 

As I noted, I'd just continue the process of copying one disk at a time across your network -- the speed of the copies via a Gb network is effectively the same as if they were local to the system, since the bottleneck is the write speed to the protected array.

 

I'd do this:

 

(1)  Copy a disk [using TeraCopy with the validate option]

(2)  Do a New Config (on the old array) without the disk you just copied

(3)  Do a pre-clear with the-n option on the disk you just copied (so it can be quickly added to the new array).  The -n option will take about 1/4th the time of a full pre-clear cycle

 

Note:  While the pre-clear is running, you could copy the next disk if there's adequate room on the new server.

 

(4)  Add the disk you just pre-cleared to the new server.

 

Repeat until done 

Each "repeat until done" is a parity rebuild on the old array (invalid till done) and a preclear -n on the next disk. Perhaps less likely to make a mistake though. How many repeats are there?

I wouldn't bother doing a parity rebuild on the old system -- just do a New Config without parity.

 

... and the pre-clear's aren't likely to take any longer than the next drive copy -- so as long as there's one disk's worth of spare space on the new array they don't actually add any time.

 

If you do it like garycase suggests, not only is there less chance of hitting the "share copy bug", but maybe also less chance of making the "change format on a disk with data" mistake. Certainly seen some of both on the forum. Also Teracopy will let you do the verify without resorting to rsync commands. So if you're not in a hurry...

 

...  So if you're not in a hurry...

 

It also won't take appreciably longer, as long as you don't do a parity sync after every disk is removed from the old system.    As I noted, the pre-clears will overlap the "next" copy ... so they don't add much time (if any) ... and since the plan is to change the disk format anyway (to XFS) even if all of the drives were added to the new server, all of the data copies would still be necessary -- and there are a LOT more opportunities to make mistakes !!

 

... I tend to do things in the SAFEST way possible 

 

I'm lost on what you're trying to accomplish. If you want to move V5 disks to a B6 array then just move them into the new server, set a new config and build parity. No need to copy data off one disk and then physically move the disk. You'd only need to do that coming from some other storage means where the filesystem isn't supported.

 

There really is little advantage to migrating to XFS. The only time XFS can be an advantage is when you're filling the disk right full. Otherwise, any disk supported in V5 will work in V6.

 

 

There is a (somewhat theoretical) command line technique for zeroing a drive in a parity array and so maintain parity when you remove it, but I have not done it and it is not for the faint of heart. Not sure how successful people have been with it.

 

Not theoretical or hard to do. Just fill the disk with zeroes. I created a "how to" a long time ago. Now with the 'trust parity" option it is even easier.

 

If you don't like that one, here's another way. Run dd a from one unRaid device to an empty unRaid disk (e.g., md4 to md7), and then you can remove both disks from the array and trust parity and preserve parity throughout. Switch to maintenance mode first of course.

 

Why? the parity of the two disks will perfectly cancel each other out, leaving the array parity protected.