Remove Drive Without Losing Parity - currently you cannot


Recommended Posts

unRAID does not (currently) provide a feature to remove a drive from the array without losing parity integrity.  So if you do remove a disk, and while you are rebuilding parity with the remaining you hit a read error on one of the disks already in the array (or worse yet lose a drive), you have no way of recovering.  If you had run a parity check just before the removal, this would be very unlikely, but if you didn't, the chances of encountering at least one read error on a large array are certainly significant.

 

Not too long ago, Tom revealed a command that I thought might allow a way to accomplish a drive removal while maintaining parity protection.  I experimented with this yesterday, and proved it can be done.  Below is the documentation of what I did.

 

To do this test I setup a test array with some spare drives.  My parity is 300G, my data1 is 250G, and my data2 is an older 20G drive.  My goal is to remove the 20G drive without losing parity protection.  I first copied a bunch of data to the 20G drive to make sure it was full of non-zero data.

 

UPDATED 8/16/10 for new unRAID versions and to reference larger buffer to increase speed.

 

UPDATED 1/16/12.  These instructions rely on the "mdcmd set invalidslot ..." command introduced into unRAID in release 4.3.2 and worked through version 4.7.  However, with the 5.0 betas, this functionality is broken in some betas and works differently than documented in this post in other betas.  If you are beyond version 4.7, research the "set invalidslot" functionality in your version of unRAID, and be prepared to adapt the instructions here.  Once the 5.0 release comes out, I will update these instructions for 5.0.

 

**** STANDARD DISCLAIMER:  USE THIS PROCESS AT YOUR OWN RISK. ****

 

1:  CLEAN BEGINNING - Parity check in progress on the array.  Notice that the parity check is past the 20G drive's size with no sync errors.

 

rm6.jpg

 

2:  BEGIN FILL - TELNET - Just ran umount and dd commands to fill the 20G drive with zeros

 

UPDATE:  By increasing the blocksize, this step can be sped up considerably.  (Thanks Joe L.).  See later post for more info, but the syntax for this example would be:

 

dd if=/dev/zero bs=2048k of=/dev/md2

 

rm2.jpg

 

3:  BEGIN FILL - WEB GUI - Web GUI shortly after dd begins.  Notice drive instantly becomes unformatted when you fill it with zeros.  But the parity is still being updated as the drive is being zeroed.

 

rm3.jpg

 

4:  END FILL - TELNET - dd command is done.  This took a long time (see "FINAL THOUGHTS" below)

 

rm4.jpg

 

5:  END FILL - WEB GUI - Web GUI after dd ends

 

rm5.jpg

 

6:  STOPPED THE ARRAY - Just pressed the Stop button

 

rm6qh7.jpg

 

7:  GO TO DEVICES TAB - About to unassign Disk2 (20G drive)

 

rm7yy0.jpg

 

8:  UNASSIGN DEVICE - Drive unassigned

 

rm8nw7.jpg

 

9:  BACK TO MAIN PAGE - Drive is missing

 

rm9.jpg

 

10:  RESTORE READY - About to press the dreaded restore button - must do in order to remove a drive from the array.

 

UPDATE: The Restore button was replaced with an "initconfig" command in later versions of unRAID. If the restore button does not appear, simply run this command from a telnet prompt.

 

rm10.jpg

 

11:  RESTORE PRESSED - Technically parity protection is gone now, but array is offline. Hard to think of a way you coud lose data now.  Parity will be restored in 15 seconds.

 

rm11.jpg

 

12:  SET INVALIDSLOT - Special command that tells unRAID that the array is already protected.  Starting the array after this command will trigger a PARITY CHECK and not a PARITY REBUILD.  (Remember that the array is protected during a parity check, but not during a parity rebuild).

 

rm13.jpg

 

13:  ARRAY JUST STARTED - Back to Web GUI and pressed Start button.  Notice that the parity CHECK automatically starts and is in progress. (Remember the array IS protected during a parity check.)  If we hadn't run the set invalidslot, parity would be rebuilding, and the array would be totally unprotected until the rebuild completed.  Changing the parity rebuild to a parity check is the goals of this process.  Although not recommended, you could stop the parity check and the array would be protected.

 

rm14.jpg

 

13:  PARITY CHECK - FINAL - Parity check is past the size of the 20G drive.  ZERO sync errors.  Mission accomplished.

 

rm15.jpg

 

FINAL THOUGHTS:

 

The only problem with this procedure is the amount of time it took to fill the drive with zeros with the "dd" command.  The 20G drive is old and pretty slow (likely a 5400 RPM drive), but filling it with zeros took 10+ hours (remember this is only a 20G drive).  There has to be a faster way to accomplish this.  If we can get past that performance problem, I think this is a very plausible way to remove a drive while maintaining parity protection.

Link to comment

Perhaps the 20 Gig drive was operating in PIO mode?  Might want to examine the syslog for clues.

 

Other than that, you might be able to specify the block size in the "dd"  command to gain some efficiency. I think the default is to only write 512 bytes at a time.

 

Joe L.

Link to comment

Excellent test.. Very informative.

 

There has to be a faster way to accomplish this.  If we can get past that performance problem, I think this is a very plausible way to remove a drive while maintaining parity protection.

 

Try issuing dd with a bsize parameter.

I.E. bsize=1024 or bsize=10240 or bsize=102400

 

If we could zero out parity "VIRTUALLY" on the drive to be removed this would be perfect.

It would allow us to have roaming data drives yet leaving the parity protected during the removal.

 

I really wish there was a way to handle roaming elegantly.

I.E. adding a drive into the array just updating parity for the new drive and removing parity virtually from the drive to be removed.

One can dream  ;D

Link to comment

Joe L. - Here is the syslog.  I don't think the drive is in PIO mode, but do see some strangeness in there that maybe RobJ or you could make some sense of.  The time seems to be off in the log.

 

I did copy a bunch of data to this disk - but that was before enabling parity.  It did not seem so slow (less than an hour to copy 20G, but I wasn't really keeping track).  Parity check was at about 22,000KB/sec while that disk is involved and about 36,000KB/sec using just the 2 larger drives.  (These are all IDE drives).  The 2 data drives are on the same channel, the parity is on its own channel.

 

WeeboTech - Thanks! And also thanks for supplying help on the commands to clear the drive.  They seemed to work well.  I will have to retry my test and add the bsize to see if it improves performance.

 

It did appear by observice the Web GUI that both the 20G and parity drives were each read and wrote each sector.  (That's 80G of mixed I/O).

 

*  *  *  *  * 

 

I am wondering if anything "resets" the invalidslot back to 0?  Tom, if you are reading, is this something that has to be done manually, or is it automatically reset after the array starts?  If invalidslot is set to something besides 0, will that affect how unRAID reacts in case of a parity error during a parity check?

Link to comment

Joe L. - Here is the syslog.  I don't think the drive is in PIO mode, but do see some strangeness in there that maybe RobJ or you could make some sense of.  The time seems to be off in the log.

 

I did copy a bunch of data to this disk - but that was before enabling parity.  It did not seem so slow (less than an hour to copy 20G, but I wasn't really keeping track).  Parity check was at about 22,000KB/sec while that disk is involved and about 36,000KB/sec using just the 2 larger drives.  (These are all IDE drives).  The 2 data drives are on the same channel, the parity is on its own channel.

 

WeeboTech - Thanks! And also thanks for supplying help on the commands to clear the drive.  They seemed to work well.  I will have to retry my test and add the bsize to see if it improves performance.

 

It did appear by observice the Web GUI that both the 20G and parity drives were each read and wrote each sector.  (That's 80G of mixed I/O).

 

*  *  *  *  * 

 

I am wondering if anything "resets" the invalidslot back to 0?  Tom, if you are reading, is this something that has to be done manually, or is it automatically reset after the array starts?  If invalidslot is set to something besides 0, will that affect how unRAID reacts in case of a parity error during a parity check?

If it took an hour or so to copy 20 Gig to the drive, then to read and write 80 gig could easily take 4 hours or more.  10 Hours seems high.
Link to comment

Good idea, Brian!

 

For the period of time reported in the syslog, the 20GB was at a speed of UDMA/100, very good for IDE drives, but this syslog does not cover the hour when the copying was done, so can't speculate on any errors or speed degradation then.

 

I'm not sure that the hour of straight copying of 20GB will scale linearly to 4 hours of mixed I/O, with an old 20GB drive with slower RPM's and perhaps very little cache.  With parity in use, you have to read a data block first before you can write the zeroed block back, so I could easily see this (without enough cache for track read-ahead) having to wait a rev before it can write the sector, then wait *another* rev to read the next sector.  That would lead to *much* longer times, especially with a slower RPM drive.

 

With a modern drive, I would think a larger bsize should help a little, although not as much as you might think, because the drive's own cache should be helping already.  With this old drive, I am curious how much it would help.  It's possible it could help a lot, but it is also possible that it may make it even worse, trying to satisfy larger read and write requests without enough caching, especially for blocks of size bsize that span tracks.  It would be too time consuming (and pointless) to try various bsize values, for a one-time operation, that won't really be helpful to anyone else with more modern drives with decent caches.

Link to comment
  • 5 weeks later...

Good idea, Brian!

 

For the period of time reported in the syslog, the 20GB was at a speed of UDMA/100, very good for IDE drives, but this syslog does not cover the hour when the copying was done, so can't speculate on any errors or speed degradation then.

 

I'm not sure that the hour of straight copying of 20GB will scale linearly to 4 hours of mixed I/O, with an old 20GB drive with slower RPM's and perhaps very little cache.  With parity in use, you have to read a data block first before you can write the zeroed block back, so I could easily see this (without enough cache for track read-ahead) having to wait a rev before it can write the sector, then wait *another* rev to read the next sector.  That would lead to *much* longer times, especially with a slower RPM drive.

 

With a modern drive, I would think a larger bsize should help a little, although not as much as you might think, because the drive's own cache should be helping already.  With this old drive, I am curious how much it would help.  It's possible it could help a lot, but it is also possible that it may make it even worse, trying to satisfy larger read and write requests without enough caching, especially for blocks of size bsize that span tracks.  It would be too time consuming (and pointless) to try various bsize values, for a one-time operation, that won't really be helpful to anyone else with more modern drives with decent caches.

Just yesterday I was working with different values for bs (block-size) on the "dd" command.  I did a bit of research on google and generally it was suggested to use 2048k as the block size.

 

The correct command would then be

dd if=/dev/zero bs=2048k of=/dev/md?

 

where md? = md1 through md15, depending on the drive being zeroed.

 

When doing many tests with my older 8Gig disk, the speed increased from roughly 10Mb/s to 24Mb/s using the 2048k block size vs. the default 512 byte block size.

 

Joe L.

Link to comment
  • 1 month later...

I just removed a 750 Gig SATA drive from my array using this technique. 

(It had no files on it, and I had added it as a test of my preclear_disk.sh script, so I went through the steps outlined in this thread using a

"dd if=/dev/zero bs=2048k of=/dev/md8"

 

The SATA drive was hooked up to a 4 port 150MHz SATA 1 controller plugged into the PCI bus on my older Intel motherboard.

 

It took a little over 11 hours to zero the drive.  It started out writing at about 19.5 MB/s and finished at about 18.5 MB/s.

 

The SATA drive was then removed as described above from the array, then I powered down and physically removed it.  (It will be used in the new server I'm building.)

 

Minimal down-time, and continued parity protection... life is good.

 

Joe L.

Link to comment

I was able to do this a while ago (see first post in this thread).  Pretty slow to clear the drive (somehow Tom does it much quicker when he adds a drive to the array). 

 

Joe L.'s  use of the 2K buffer certainly made things move alot faster than my experience.

 

 

 

Link to comment

somehow Tom does it much quicker when he adds a drive to the array. 

When Tom clears a drive prior to adding it to an array he is only writing to the drive, not reading both it and parity and writing it and parity. 

This trick to maintain parity while zeroing a drive will always be slower than the initial clearing of a drive prior to adding it to the array. 4 I/O operations per block vs. 1.

Joe L.'s  use of the 2K buffer certainly made things move alot faster than my experience.

I used 2048k as the buffer size, not 2k.  Much  bigger (2 Meg buffer vs. default size of 512 bytes), but it still took 11+ hours.

 

Joe L.

Link to comment
  • 7 months later...

Just for more data.... 

 

On my 300GB drive...  Using DD with no block size I get

 

27769+0 records out

14217728 bytes (14 MB) copied, 31.8698 s, 446 kB/s

 

66361+0 records out

33976832 bytes (34 MB) copied, 81.6211 s, 416 kB/s

 

If I do

dd if=/dev/zero bs=2048k of=/dev/md2 &

 

I get:

 

78+0 records out

163577856 bytes (164 MB) copied, 15.3426 s, 10.7 MB/s

 

1886+0 records out

3955228672 bytes (4.0 GB) copied, 363.529 s, 10.9 MB/s

 

2678+0 records out

5616173056 bytes (5.6 GB) copied, 516.283 s, 10.9 MB/s

 

So for me using the block size really helps!

 

FYI..

 

 

Link to comment
  • 5 months later...

Thanks for the original post. I used this procedure to remove two drives simultaneously without losing parity. Here is the procedure that I used.

 

1. umount /dev/md1

    umount /dev/md2

2. I opened another telnet session.

3. Telnet 1:  dd if=/dev/zero bs=2048k  of=/dev/md1

4. Telnet 2:  dd if=/dev/zero bs=2048k  of=/dev/md2

5. Wait ~5 1/2 hours (Both drives were 300GB Drives)

 

Telnet 1 output:

 

Tower:~# dd if=/dev/zero bs=2048k of=/dev/md1

dd: writing `/dev/md1': No space left on device

143095+0 records in

143094+0 records out

300090695680 bytes (300 GB) copied, 26049.5 s, 11.5 MB/s

 

Telnet 2 output:

Tower:~# dd if=/dev/zero bs=2048k  of=/dev/md2

dd: writing `/dev/md2': No space left on device

143095+0 records in

143094+0 records out

300090695680 bytes (300 GB) copied, 25847.2 s, 11.6 MB/s

 

 

6. Stopped the array. (Make sure commands from step 5 are complete)

7. Removed disk1,disk2 from array.

8. Restored Array. (Pressed Restore Button)

9. From Telnet 1: >mdcmd set invalidslot 99

10. Started Array (Start Button)

11. Checked Parity (Successful)

 

Whole process was smooth and no issues.

 

Link to comment
  • 4 months later...
  • 5 months later...

Hi, my 500gb samsung usb h.d stopped working when my O.Sw7 did. I;m not sure why- possibly a virus. I have since reinstalled W7 and i cant open the hard drive- in disc management it tells me its a healthy, active primary partition RAW but keeps asking me to format it. I downloaded a data recovery program which shows me all my files are still on there but i need to reformat it without losing those files. i have been told i can reformat it to a NTFS format from the command prompt without losing data but im a little nervous about it, any ideas?

I do not believe you can reformat ANY drive without losing the files on it.  Sorry but I think you were mis-informed.

 

You probably can set the partition type without losing the data in the partition, but that does not format the drive.

Link to comment
  • 1 month later...

Would this be possible to do safely on two drives at the same time? I have two 1 TB drives that I wish to remove. I've already copied their data onto a single 2 TB drive in the array. I want to replace the slots of those drives with another 2 TB drive to expand the array, and keep the next slot for preclearing other new 2(+) TB  drives. I have several 1 TB drives to replace with 2(+) TB ones, so repeating this process over time, will always ensure I have slots to preclear further drives.

Link to comment
  • 6 months later...

All right then I know this is a 4.3 thread but I was wondering would this still work on 4.5.6?

thanks

 

This technique should work for 4.x versions, and ~5.0b8+.  The "set invalidslot" command works a little differently with the 5.0 betas.

Link to comment

Hello all.

Really need your help.

Followed the instructions but doing "initconfig" (unraid 4.7) doesn't turn the disks blue. instead - i'm stuck at "Missing Disk" status.

any help would be appreciated.

 

Did you refresh the web gui?

 

When you ran initconfig, did it ask that you enter the word "Yes" (capital "Y" followed by lower case "es")?  Did you do it?

Link to comment

Has anyone tested this and confirmed the steps necessary with 5.x?

 

I need to start replacing the 3+ year old drives in my main array, and want to do it retaining parity.  Since I have several drives to do it to, I was planning to write a 5.0 plugin to accomplish it safely, but I don't want to reinvent the wheel.

 

 

Link to comment

Hello all.

Really need your help.

Followed the instructions but doing "initconfig" (unraid 4.7) doesn't turn the disks blue. instead - i'm stuck at "Missing Disk" status.

any help would be appreciated.

 

Did you refresh the web gui?

 

When you ran initconfig, did it ask that you enter the word "Yes" (capital "Y" followed by lower case "es")?  Did you do it?

Wrote "Yes" at initconfig, and it even indicated "Completed".

refreshing the web gui had no effect on the drives. they were all green except the one i've remove from the slot (which remains missing).

any ideas ?

Link to comment

As bjp999 reported in this thread:

 

    http://lime-technology.com/forum/index.php?topic=13567.0

 

I can confirm that under 5.0beta8, and beta10, the "mdcmd set invalidslot 99" does not work.

On one post made by Tom he said to NOT refresh the browser after running it on those versions since the browser refresh re-loads the "md" device and negates the invalid slot command. I guess you basically have to go on faith the array is initialized.
Link to comment
Guest
This topic is now closed to further replies.