NAS

EMPTY and Remove a Drive Without Losing Parity

195 posts in this topic Last Reply

Recommended Posts

@-Daedalus @garycase

 

Gary -

 

I think you're missing the point. If unRAID seeks to move more into appliance mode and away from having a skilled IT person operating it, features like "I want to remove this disk" would make sense, without a lot of "you have to do this first" stuff a person would just have to know to do or find knee deep in the forums.

Share this post


Link to post

I understand what you're saying ... I just don't think it's even close to a common function.  WHY would you want to remove a disk that had data on it ??   Replace it? -- Yes.   But remove it?   Folks who manipulate their data -- moving it around to different disks; restructure their shares; etc. aren't likely in the "naïve appliance user" category.   And for them to simply move all the data off a disk they want to remove from the array isn't a daunting chore at all.

 

Personally, if I DID want to remove a disk, I'd simply remove it and do a New Config.   (Obviously I'd confirm all was well before doing that)

 

I simply don't think a "Remove a disk" function is needed -- but I agree it would be a nice function for those cases where somebody wants it.   Zeroing a drive with DD and then doing a New Config with the "parity is already valid" box is definitely more complex than a simple "Remove an empty drive" function.    I just don't think that "emptying the drive" needs to be a GUI function => in fact, there's already a plugin that will do this [ UnBalance ].

 

Share this post


Link to post

It is a common ask. I agree with you that most experienced users do not need it often or at all. But every newbie asks (if not on the forum, in his head). And there is one use case that is relatively common. The desire to use a recently added disk to rebuild a failing disk. The answer "you shouldn't have added it in the first place" is not exactly helpful. And typically the user is convinced to buy a new one. But the removal method we've discussed (emptying then zeroing) would enable that as painlessly as possible. And there is no other way I know.

Share this post


Link to post

It could also be argued that it's somewhat expected. I'll grant you removing a disk is far less common than adding one, but unRAID's big callout is how easily disks can be added, and that you don't have to faff about with matches drives, pools, etc. like you do with something like ZFS. By this, you'd think an obvious extension would be easily adding, removing, and replacing disks.

Share this post


Link to post
14 hours ago, -Daedalus said:

you'd think an obvious extension would be easily adding, removing, and replacing disks.

Adding and replacing are already easy.

Share this post


Link to post

 

4 hours ago, trurl said:

Adding and replacing are already easy.

 

I'm aware.

 

18 hours ago, -Daedalus said:

By this, you'd think an obvious extension would be easily adding, removing, and replacing disks.

 

Share this post


Link to post
6 hours ago, -Daedalus said:

I'm aware.

 

So you meant this

On 7/29/2017 at 5:13 PM, -Daedalus said:

an obvious extension would be easily removing disks.

 

Share this post


Link to post
7 hours ago, trurl said:

 

So you meant this

 

 

No, actually I meant just what I said. An extension of the current approach "Easily add and replace disks", would be "Easily add, replace, and remove disks". These are the main operations a user would expect to be able to do with something designed around easy data management, and it stands to reason all options should be (roughly) equally easy.

Share this post


Link to post
3 hours ago, -Daedalus said:

 

 it stands to reason all options should be (roughly) equally easy.

No, removing a disk is much more difficult to engineer to be idiot proof, and it's much more rarely used. The first tenant of unraid is data safety and integrity. Adding blank capacity is easy, replacing disks is a little more rigorous, because you have to ensure that the data is safely transferred from the old to the new. REMOVING disks is risky. How do you deal with the data currently on the disk to be removed? You have to take into account many more scenarios, all much more risky than a simple add or replace.

 

Many of the possible methods are outlined in this forum, but none are completely without risk.

 

Removing disks comes up so rarely compared to adding and replacing, it's not a high priority for limetech to develop an easy button, especially when there are methods already in place to accomplish it if the user really needs to remove a disk.

Share this post


Link to post
4 hours ago, jonathanm said:

No, removing a disk is much more difficult to engineer to be idiot proof, and it's much more rarely used. The first tenant of unraid is data safety and integrity. Adding blank capacity is easy, replacing disks is a little more rigorous, because you have to ensure that the data is safely transferred from the old to the new. REMOVING disks is risky. How do you deal with the data currently on the disk to be removed? You have to take into account many more scenarios, all much more risky than a simple add or replace.

 

Many of the possible methods are outlined in this forum, but none are completely without risk.

 

Removing disks comes up so rarely compared to adding and replacing, it's not a high priority for limetech to develop an easy button, especially when there are methods already in place to accomplish it if the user really needs to remove a disk.

 

I agree with some of this but respectfully disagree with the conclusion.

 

The priority of the enhancement should certainly be a consideration, but so should the level of effort. This is not like implementing dual parity, or figuring out how to make Ryzen c-states work. It is well defined, the techniques are clear, and by and large the code already exist. This would not be a lengthy enhancement.

 

There are three parts:

1 - Copy the data off - a lot like the mover script, but moving from the "drive to remove" to the array. It could get snagged up if there is not enough space. It would take some testing to confirm everything got moved off. Not trying to say trivial, there are some exception cases that would have to be thought through. But very doable. And if things got too complex, it could punt the job back to the user.

2 - Zero the disk - dead simple

3 - Remove the disk from the config - maybe a little tricky if the array is not brought offline. But I think the stop array button could be used to stop the array and remove the disk from the array.

 

And I DO think it would be used. I already gave one use case ("I added a drive to the array and now want to use it to rebuild a failed disk"). But here is another maybe more satisfying use case.

 

Say a user buys a new 8T drive, and wants to use it to replace 2 3T drives (freeing a slot and adding 2T of free space). He can replace one of the disks with the 8T drive. But now he wants to distribute the contents of the other disk to the array and remove it. Today he would be using unbalance or Krusader, performing a parity check, doing a new config, performing a parity build. and if they are a good citizen, doing one more parity check at the end. If this feature were in place, push the remove drive button, and the whole operation would happen with parity maintained. Stop the array and the disk is free. People would use this feature!

  • Upvote 1

Share this post


Link to post

Long time I'm not active on this forum, unfortunately no much free time, but just want to say that after all this time I still fully agree that this should be implemented, and in fact I'm surprised that it wasn't yet after all this time and all changes on unRaid? I remember Tom mentioning that such feature could lead to mistakes, but there are certainly ways to prevent them, like only allowing to do this for a drive with empty file system or only for a file system with a single file with specific filename just to fully avoid any possible mistake.... but IMO it is definitively logic that users may want to remove a drive from the array without needing do a parity resync (that would unprotect the array while doing it) or without the need to use command line to do it, that is much more prone to mistakes (i.e. if user zeros the wrong drive by mistake).

Share this post


Link to post
2 hours ago, bjp999 said:

Copy the data off - a lot like the mover script, but moving from the "drive to remove" to the array.

 

Perhaps the latest versions of the mover script already have the capabilities in place.

 

Share this post


Link to post
15 hours ago, bjp999 said:

There are three parts:

1 - Copy the data off - a lot like the mover script, but moving from the "drive to remove" to the array. It could get snagged up if there is not enough space. It would take some testing to confirm everything got moved off. Not trying to say trivial, there are some exception cases that would have to be thought through. But very doable. And if things got too complex, it could punt the job back to the user.

2 - Zero the disk - dead simple

3 - Remove the disk from the config - maybe a little tricky if the array is not brought offline. But I think the stop array button could be used to stop the array and remove the disk from the array.

 

Basically agree, but as I noted earlier it's more complex if you consider the non-NAS functions that are now common in UnRAID.   Dockers and/or VM's that use the disk to-be-removed would have to be either shut down during the process or somehow notified to not use that disk.

 

Excluding those, it is indeed a fairly simple function => step (1) is already available in UnBalance, which could perhaps be invoked to empty the disk (and STOP the process with a "NOT ENOUGH SPACE"  if there isn't sufficient space to do it);  step (2) is indeed very simple (albeit LONG) -- but would also require the system be set to NOT do any further writes to the disk (this would perhaps mitigate the Dockers/VM issue as far as the removal goes ... but could generate questions as to "why my Docker isn't working anymore);  and step (3) should be virtually instantaneous (just modifying the config) ... although as bjp999 noted it may be tricky with the array online.

 

Share this post


Link to post

I know the P+Q parity scheme is a lot more complex than just a simple XOR.

 

Is the manual procedure in the first post valid with dual parity?

Share this post


Link to post
3 minutes ago, nick5429 said:

I know the P+Q parity scheme is a lot more complex than just a simple XOR.

 

Is the manual procedure in the first post valid with dual parity?

Anytime you write to an md device, parity is maintained.

Share this post


Link to post
4 minutes ago, nick5429 said:

I know the P+Q parity scheme is a lot more complex than just a simple XOR.

 

Is the manual procedure in the first post valid with dual parity?

 

It's valid, just make sure you keep the other disks assignments after the new config, i.e., say you remove disk3 from a 5 disk array, you need to leave that slot empty or parity2 won't be valid no more, also don't forget to enable turbo write before starting.

  • Like 1

Share this post


Link to post
6 minutes ago, trurl said:

Anytime you write to an md device, parity is maintained.

Of course.  But can I then remove that device, do a "New Config", and tell unraid "trust me, the parity is still good even though I removed a device" with dual parity mode active?

 

The answer is trivially yes in single parity mode where P is a simple XOR; I didn't see this directly addressed for dual parity where the calculations are much more complex (and the procedure was defined before dual parity mode existed), so wanted to ask.

Edited by nick5429

Share this post


Link to post

As Johnnie noted, as long as you maintain all of the same slots for the other disks, it will still be valid.    This does, however, add another element of risk to using this procedure instead of simply doing a New Config and letting parity rebuild -- which is still how I remove disks.  [A very rare process in my case]

 

Share this post


Link to post

I agree that this would be a nice feature to have.  I've actually been wanting to downgrade my server to use SSDs instead of the big HDDs that are in it right now; I don't need as much storage as I once did and I'd love an easy way to replace them all one-by-one without starting entirely from scratch on the whole thing.

Share this post


Link to post

I know this is an older thread but wanted to add something. I am a newish unRaid user and made the mistake of dumping every drive I had lying around (yes, I know now that it was stupid) including 3 tiny 320GB drives. These drives are old and haven't been written to yet. I want to pull them and drop in a 5TB drive. At the same time I want to add more SSDs to my cache pool.

 

Now I could just leave them all in but it will push me past my 12 drive limit and I don't want to go to a pro license. I for one, would think a simple button solution would be great!

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


Copyright © 2005-2018 Lime Technology, Inc.
unRAID® is a registered trademark of Lime Technology, Inc.