Remove Drive Without Losing Parity

[bjp999]

unRAID does not (currently) provide a feature to remove a drive from the array without losing parity integrity.  So if you do remove a disk, and while you are rebuilding parity with the remaining you hit a read error on one of the disks already in the array (or worse yet lose a drive), you have no way of recovering.  If you had run a parity check just before the removal, this would be very unlikely, but if you didn't, the chances of encountering at least one read error on a large array are certainly significant.

Not too long ago, Tom revealed a command that I thought might allow a way to accomplish a drive removal while maintaining parity protection.  I experimented with this yesterday, and proved it can be done.  Below is the documentation of what I did.

To do this test I setup a test array with some spare drives.  My parity is 300G, my data1 is 250G, and my data2 is an older 20G drive.  My goal is to remove the 20G drive without losing parity protection.  I first copied a bunch of data to the 20G drive to make sure it was full of non-zero data.

UPDATED 8/16/10 for new unRAID versions and to reference larger buffer to increase speed.

UPDATED 1/16/12.  These instructions rely on the "mdcmd set invalidslot ..." command introduced into unRAID in release 4.3.2 and worked through version 4.7.  However, with the 5.0 betas, this functionality is broken in some betas and works differently than documented in this post in other betas.  If you are beyond version 4.7, research the "set invalidslot" functionality in your version of unRAID, and be prepared to adapt the instructions here.  Once the 5.0 release comes out, I will update these instructions for 5.0.

**** STANDARD DISCLAIMER:  USE THIS PROCESS AT YOUR OWN RISK. ****

1:  CLEAN BEGINNING - Parity check in progress on the array.  Notice that the parity check is past the 20G drive's size with no sync errors.



2:  BEGIN FILL - TELNET - Just ran umount and dd commands to fill the 20G drive with zeros

UPDATE:  By increasing the blocksize, this step can be sped up considerably.  (Thanks Joe L.).  See later post for more info, but the syntax for this example would be:

dd if=/dev/zero bs=2048k of=/dev/md2



3:  BEGIN FILL - WEB GUI - Web GUI shortly after dd begins.  Notice drive instantly becomes unformatted when you fill it with zeros.  But the parity is still being updated as the drive is being zeroed.



4:  END FILL - TELNET - dd command is done.  This took a long time (see "FINAL THOUGHTS" below)



5:  END FILL - WEB GUI - Web GUI after dd ends



6:  STOPPED THE ARRAY - Just pressed the Stop button



7:  GO TO DEVICES TAB - About to unassign Disk2 (20G drive)



8:  UNASSIGN DEVICE - Drive unassigned



9:  BACK TO MAIN PAGE - Drive is missing



10:  RESTORE READY - About to press the dreaded restore button - must do in order to remove a drive from the array.

UPDATE: The Restore button was replaced with an "initconfig" command in later versions of unRAID. If the restore button does not appear, simply run this command from a telnet prompt.



11:  RESTORE PRESSED - Technically parity protection is gone now, but array is offline. Hard to think of a way you coud lose data now.  Parity will be restored in 15 seconds.



12:  SET INVALIDSLOT - Special command that tells unRAID that the array is already protected.  Starting the array after this command will trigger a PARITY CHECK and not a PARITY REBUILD.  (Remember that the array is protected during a parity check, but not during a parity rebuild).



13:  ARRAY JUST STARTED - Back to Web GUI and pressed Start button.  Notice that the parity CHECK automatically starts and is in progress. (Remember the array IS protected during a parity check.)  If we hadn't run the set invalidslot, parity would be rebuilding, and the array would be totally unprotected until the rebuild completed.  Changing the parity rebuild to a parity check is the goals of this process.   Although not recommended, you could stop the parity check and the array would be protected.



13:  PARITY CHECK - FINAL - Parity check is past the size of the 20G drive.  ZERO sync errors.  Mission accomplished.



FINAL THOUGHTS:

The only problem with this procedure is the amount of time it took to fill the drive with zeros with the "dd" command.  The 20G drive is old and pretty slow (likely a 5400 RPM drive), but filling it with zeros took 10+ hours (remember this is only a 20G drive).  There has to be a faster way to accomplish this.  If we can get past that performance problem, I think this is a very plausible way to remove a drive while maintaining parity protection.

[Joe L.]

Perhaps the 20 Gig drive was operating in PIO mode?   Might want to examine the syslog for clues.

Other than that, you might be able to specify the block size in the "dd"  command to gain some efficiency. I think the default is to only write 512 bytes at a time.

Joe L.

[WeeboTech]

Excellent test.. Very informative.

There has to be a faster way to accomplish this.  If we can get past that performance problem, I think this is a very plausible way to remove a drive while maintaining parity protection.

Try issuing dd with a bsize parameter.
I.E. bsize=1024 or bsize=10240 or bsize=102400

If we could zero out parity "VIRTUALLY" on the drive to be removed this would be perfect.
It would allow us to have roaming data drives yet leaving the parity protected during the removal.

I really wish there was a way to handle roaming elegantly.
I.E. adding a drive into the array just updating parity for the new drive and removing parity virtually from the drive to be removed.
One can dream 

[bjp999]

Joe L. - Here is the syslog.  I don't think the drive is in PIO mode, but do see some strangeness in there that maybe RobJ or you could make some sense of.  The time seems to be off in the log.

I did copy a bunch of data to this disk - but that was before enabling parity.  It did not seem so slow (less than an hour to copy 20G, but I wasn't really keeping track).  Parity check was at about 22,000KB/sec while that disk is involved and about 36,000KB/sec using just the 2 larger drives.  (These are all IDE drives).  The 2 data drives are on the same channel, the parity is on its own channel.

WeeboTech - Thanks! And also thanks for supplying help on the commands to clear the drive.  They seemed to work well.  I will have to retry my test and add the bsize to see if it improves performance.

It did appear by observice the Web GUI that both the 20G and parity drives were each read and wrote each sector.  (That's 80G of mixed I/O).

*  *  *  *  * 

I am wondering if anything "resets" the invalidslot back to 0?  Tom, if you are reading, is this something that has to be done manually, or is it automatically reset after the array starts?  If invalidslot is set to something besides 0, will that affect how unRAID reacts in case of a parity error during a parity check?

[Joe L.]

Joe L. - Here is the syslog.  I don't think the drive is in PIO mode, but do see some strangeness in there that maybe RobJ or you could make some sense of.  The time seems to be off in the log.

I did copy a bunch of data to this disk - but that was before enabling parity.  It did not seem so slow (less than an hour to copy 20G, but I wasn't really keeping track).  Parity check was at about 22,000KB/sec while that disk is involved and about 36,000KB/sec using just the 2 larger drives.  (These are all IDE drives).  The 2 data drives are on the same channel, the parity is on its own channel.

WeeboTech - Thanks! And also thanks for supplying help on the commands to clear the drive.  They seemed to work well.  I will have to retry my test and add the bsize to see if it improves performance.

It did appear by observice the Web GUI that both the 20G and parity drives were each read and wrote each sector.  (That's 80G of mixed I/O).

*  *  *  *  * 

I am wondering if anything "resets" the invalidslot back to 0?  Tom, if you are reading, is this something that has to be done manually, or is it automatically reset after the array starts?  If invalidslot is set to something besides 0, will that affect how unRAID reacts in case of a parity error during a parity check?

If it took an hour or so to copy 20 Gig to the drive, then to read and write 80 gig could easily take 4 hours or more.   10 Hours seems high.

[RobJ]

Good idea, Brian!

For the period of time reported in the syslog, the 20GB was at a speed of UDMA/100, very good for IDE drives, but this syslog does not cover the hour when the copying was done, so can't speculate on any errors or speed degradation then.

I'm not sure that the hour of straight copying of 20GB will scale linearly to 4 hours of mixed I/O, with an old 20GB drive with slower RPM's and perhaps very little cache.  With parity in use, you have to read a data block first before you can write the zeroed block back, so I could easily see this (without enough cache for track read-ahead) having to wait a rev before it can write the sector, then wait *another* rev to read the next sector.  That would lead to *much* longer times, especially with a slower RPM drive.

With a modern drive, I would think a larger bsize should help a little, although not as much as you might think, because the drive's own cache should be helping already.  With this old drive, I am curious how much it would help.  It's possible it could help a lot, but it is also possible that it may make it even worse, trying to satisfy larger read and write requests without enough caching, especially for blocks of size bsize that span tracks.  It would be too time consuming (and pointless) to try various bsize values, for a one-time operation, that won't really be helpful to anyone else with more modern drives with decent caches.

[Joe L.]

Good idea, Brian!

For the period of time reported in the syslog, the 20GB was at a speed of UDMA/100, very good for IDE drives, but this syslog does not cover the hour when the copying was done, so can't speculate on any errors or speed degradation then.

I'm not sure that the hour of straight copying of 20GB will scale linearly to 4 hours of mixed I/O, with an old 20GB drive with slower RPM's and perhaps very little cache.  With parity in use, you have to read a data block first before you can write the zeroed block back, so I could easily see this (without enough cache for track read-ahead) having to wait a rev before it can write the sector, then wait *another* rev to read the next sector.  That would lead to *much* longer times, especially with a slower RPM drive.

With a modern drive, I would think a larger bsize should help a little, although not as much as you might think, because the drive's own cache should be helping already.  With this old drive, I am curious how much it would help.  It's possible it could help a lot, but it is also possible that it may make it even worse, trying to satisfy larger read and write requests without enough caching, especially for blocks of size bsize that span tracks.  It would be too time consuming (and pointless) to try various bsize values, for a one-time operation, that won't really be helpful to anyone else with more modern drives with decent caches.

Just yesterday I was working with different values for bs (block-size) on the "dd" command.   I did a bit of research on google and generally it was suggested to use 2048k as the block size.

The correct command would then be
dd if=/dev/zero bs=2048k of=/dev/md?

where md? = md1 through md15, depending on the drive being zeroed.

When doing many tests with my older 8Gig disk, the speed increased from roughly 10Mb/s to 24Mb/s using the 2048k block size vs. the default 512 byte block size.

Joe L.

[Joe L.]

I just removed a 750 Gig SATA drive from my array using this technique. 
(It had no files on it, and I had added it as a test of my preclear_disk.sh script, so I went through the steps outlined in this thread using a
"dd if=/dev/zero bs=2048k of=/dev/md8"

The SATA drive was hooked up to a 4 port 150MHz SATA 1 controller plugged into the PCI bus on my older Intel motherboard.

It took a little over 11 hours to zero the drive.   It started out writing at about 19.5 MB/s and finished at about 18.5 MB/s.

The SATA drive was then removed as described above from the array, then I powered down and physically removed it.  (It will be used in the new server I'm building.)

Minimal down-time, and continued parity protection... life is good.

Joe L.

[bjp999]

I was able to do this a while ago (see first post in this thread).  Pretty slow to clear the drive (somehow Tom does it much quicker when he adds a drive to the array). 

Joe L.'s  use of the 2K buffer certainly made things move alot faster than my experience.

[Joe L.]

somehow Tom does it much quicker when he adds a drive to the array. 

When Tom clears a drive prior to adding it to an array he is only writing to the drive, not reading both it and parity and writing it and parity. 
This trick to maintain parity while zeroing a drive will always be slower than the initial clearing of a drive prior to adding it to the array. 4 I/O operations per block vs. 1.

Joe L.'s  use of the 2K buffer certainly made things move alot faster than my experience.

I used 2048k as the buffer size, not 2k.  Much  bigger (2 Meg buffer vs. default size of 512 bytes), but it still took 11+ hours.

Joe L.

[jbuszkie]

Just for more data.... 

On my 300GB drive...  Using DD with no block size I get

27769+0 records out
14217728 bytes (14 MB) copied, 31.8698 s, 446 kB/s

66361+0 records out
33976832 bytes (34 MB) copied, 81.6211 s, 416 kB/s

If I do
dd if=/dev/zero bs=2048k of=/dev/md2 &

I get:

78+0 records out
163577856 bytes (164 MB) copied, 15.3426 s, 10.7 MB/s

1886+0 records out
3955228672 bytes (4.0 GB) copied, 363.529 s, 10.9 MB/s

2678+0 records out
5616173056 bytes (5.6 GB) copied, 516.283 s, 10.9 MB/s

So for me using the block size really helps!

FYI..

[timz]

Thanks for the original post. I used this procedure to remove two drives simultaneously without losing parity. Here is the procedure that I used.

1. umount /dev/md1
    umount /dev/md2
2. I opened another telnet session.
3. Telnet 1:  dd if=/dev/zero bs=2048k  of=/dev/md1
4. Telnet 2:  dd if=/dev/zero bs=2048k  of=/dev/md2
5. Wait ~5 1/2 hours (Both drives were 300GB Drives)
   
Telnet 1 output:

Tower:~# dd if=/dev/zero bs=2048k of=/dev/md1
dd: writing `/dev/md1': No space left on device
143095+0 records in
143094+0 records out
300090695680 bytes (300 GB) copied, 26049.5 s, 11.5 MB/s


Telnet 2 output:
Tower:~# dd if=/dev/zero bs=2048k  of=/dev/md2
dd: writing `/dev/md2': No space left on device
143095+0 records in
143094+0 records out
300090695680 bytes (300 GB) copied, 25847.2 s, 11.6 MB/s


6. Stopped the array. (Make sure commands from step 5 are complete)
7. Removed disk1,disk2 from array.
8. Restored Array. (Pressed Restore Button)
9. From Telnet 1: >mdcmd set invalidslot 99
10. Started Array (Start Button)
11. Checked Parity (Successful)

Whole process was smooth and no issues.

[jbuszkie]

FYI...  If you are using UnRaid 4.5.4,  they removed the restore button.  You now have to
type initconfig from a telnet windown or the system console.

FYI....

Jim

[Joe L.]

Hi, my 500gb samsung usb h.d stopped working when my O.Sw7 did. I;m not sure why- possibly a virus. I have since reinstalled W7 and i cant open the hard drive- in disc management it tells me its a healthy, active primary partition RAW but keeps asking me to format it. I downloaded a data recovery program which shows me all my files are still on there but i need to reformat it without losing those files. i have been told i can reformat it to a NTFS format from the command prompt without losing data but im a little nervous about it, any ideas?

I do not believe you can reformat ANY drive without losing the files on it.  Sorry but I think you were mis-informed.

You probably can set the partition type without losing the data in the partition, but that does not format the drive.

[fitbrit]

Would this be possible to do safely on two drives at the same time? I have two 1 TB drives that I wish to remove. I've already copied their data onto a single 2 TB drive in the array. I want to replace the slots of those drives with another 2 TB drive to expand the array, and keep the next slot for preclearing other new 2(+) TB  drives. I have several 1 TB drives to replace with 2(+) TB ones, so repeating this process over time, will always ensure I have slots to preclear further drives.