data rebuild question

So, I haven't had a failed drive a year or two... rebooted my unraid and disk1 won't come back to life.

 

Swap in a brand new disk that I had precleared, set as XFS and it started to rebuild.

 

Should I see data on the /mnt/disk1 as it's rebuilding?  When I telnet into unraid and look at /mnt/disk1 there is nothing there.

 

thanks

So, I haven't had a failed drive a year or two... rebooted my unraid and disk1 won't come back to life.

 

Swap in a brand new disk that I had precleared, set as XFS and it started to rebuild.

 

Should I see data on the /mnt/disk1 as it's rebuilding?  When I telnet into unraid and look at /mnt/disk1 there is nothing there.

 

thanks

Please list exactly what you did. What format was disk1 before it failed? Rebuilding puts back the original file system exactly as it was, you can't switch file systems with the data intact. If you formatted the new drive, you erased it.

http://lime-technology.com/forum/index.php?topic=37490.0

well crap.

 

It was riserfs and was redballed.

I removed the drive, installed new precleared disk.  Formatted (xfs) and started the rebuild.

 

 

well crap.

 

It was riserfs and was redballed.

I removed the drive, installed new precleared disk.  Formatted (xfs) and started the rebuild.

As soon as you hit the format button, it erased the contents of the emulated drive prior to it being rebuilt.

 

You may not be completely out of luck however.  Just because the original drive redballed does not necessarily mean that it is dead.  Drives outright dying are a fairly rare occurance. 

 

Let the rebuild finish, and then we can try and salvage the files that are on the original drive.

how should I go about connecting the original drive back to see if my data is salvageable?

 

 

 

 

well crap.

 

It was riserfs and was redballed.

I removed the drive, installed new precleared disk.  Formatted (xfs) and started the rebuild.

reiserfs is extremely resilient, you may still be able to recover the data from the new disk after the rebuild is complete. DO NOT WRITE ANYTHING TO DISK1.

Rough outline of what needs to happen to start to try to recover disk 1's data.

1. Finish rebuild

2. Restart array in maintenance mode

3. Run reiserfsck --check /dev/md1

4. Post output of that command here on the forum and solicit further advice

well crap.

 

It was riserfs and was redballed.

I removed the drive, installed new precleared disk.  Formatted (xfs) and started the rebuild.

reiserfs is extremely resilient, you may still be able to recover the data from the new disk after the rebuild is complete. DO NOT WRITE ANYTHING TO DISK1.

Rough outline of what needs to happen to start to try to recover disk 1's data.

1. Finish rebuild

2. Restart array in maintenance mode

3. Run reiserfsck --check /dev/md1

4. Post output of that command here on the forum and solicit further advice

 

even though It formated XFS when it was originally reiserfs? 

 

 

Is it actually doing a rebuild? unRAID should not allow you to format a drive in this situation, but I'm not sure that it will not. If you replace a drive and don't change filesystems, will it let you format the replaced drive?

 

So many people think that a drive needs to be formatted before it can be used that if unRAID actually allows you to format a drive when it needs to be rebuilt instead then I would think this mistake would happen all the time.

 

Since the OP was only an hour ago it certainly hasn't had time to complete a rebuild. Is it still working on it?

well crap.

 

It was riserfs and was redballed.

I removed the drive, installed new precleared disk.  Formatted (xfs) and started the rebuild.

reiserfs is extremely resilient, you may still be able to recover the data from the new disk after the rebuild is complete. DO NOT WRITE ANYTHING TO DISK1.

Rough outline of what needs to happen to start to try to recover disk 1's data.

1. Finish rebuild

2. Restart array in maintenance mode

3. Run reiserfsck --check /dev/md1

4. Post output of that command here on the forum and solicit further advice

And if Reiserfsck doesn't get you anywhere, then hopefully you still have the original hard drive which odds on is still fully readable to work with.

Is it actually doing a rebuild? unRAID should not allow you to format a drive in this situation, but I'm not sure that it will not. If you replace a drive and don't change filesystems, will it let you format the replaced drive?

Sounds like you're volunteering to test that out. 

Is it actually doing a rebuild? unRAID should not allow you to format a drive in this situation, but I'm not sure that it will not. If you replace a drive and don't change filesystems, will it let you format the replaced drive?

Sounds like you're volunteering to test that out. 

I have done a rebuild several times not because a disk failed but because I wanted to use a bigger disk. I don't remember if unRAID will give you the option to format when you replace a drive or not. It certainly should not.

Is it actually doing a rebuild? unRAID should not allow you to format a drive in this situation, but I'm not sure that it will not. If you replace a drive and don't change filesystems, will it let you format the replaced drive?

Sounds like you're volunteering to test that out. 

I have done a rebuild several times not because a disk failed but because I wanted to use a bigger disk. I don't remember if unRAID will give you the option to format when you replace a drive or not. It certainly should not.

I agree.  Was just making a joke.

 

I think that the issue here is that we've got two things happening at once.  When you change a drive as soon as you start the array it starts to rebuild.  However at the same time, unRaid sees that the user also changed the format of the drive, so it brings up the format button.  There should be more error checking in that situation.

 

But, I'm not going to volunteer to see if this is what actually happens.

 

 

At least here the OP noticed the problem right away.  As long as no data is written to the drive during or after the rebuild, the odds of recovery are good.  And the original drive is more than likely still readable.

 

So.. The question I have is what about the XFS format when the original was reiserFS?

 

For the record while I still remember this is the detail of what happened:

1) rebooted unraid

2) Disk1 was redballed (ResierFS)

3) shut down array powered off and replaced disk 1 (2tb) with new 3tb precleared disk

4) booted unraid

5) selected new disk as disk 1 placement

6) started array figuring it will trigger the rebuild and it showed unformatted disk

7) I selected the checkbox to format and unraid started formatting disk1 as XFS  <--apparently.. BAD! UGH!

8 ) Array showed drive rebuilding

 

If this doesn't go well...and disk1 is empty, can I add back the original redball as a disk in the array?  How should I go about adding it back to the system to try and copy off data?

 

 

 

Is it still rebuilding?

So.. The question I have is what about the XFS format when the original was reiserFS?

 

For the record while I still remember this is the detail of what happened:

1) rebooted unraid

2) Disk1 was redballed (ResierFS)

3) shut down array powered off and replaced disk 1 (2tb) with new 3tb precleared disk

4) booted unraid

5) selected new disk as disk 1 placement

6) started array figuring it will trigger the rebuild and it showed unformatted disk

7) I selected the checkbox to format and unraid started formatting disk1 as XFS  <--apparently.. BAD! UGH!

8 ) Array showed drive rebuilding

Because as far as unRaid was concerned, based upon you selecting the format of the drive to be XFS and replacing the drive at the same time it assumed that's what you wanted to do.  Format the emulated drive as XFS, and then rebuild it onto the replacement disk.

 

Its not a bug in the system (it only did what you told it to do), but there should be more error checking because I can't think of any situation where you would actually want it to do this kind of procedure, and it should have thrown out a whack of "Are you sure?" "Are you really sure?" "Ask someone else if they're sure?" messages.  Let the rebuild complete, and then follow jonathanm instructions. 

 

If we can't get the data back through that, then we'll have to go back to the bad drive and see what can be done.  Why exactly did you replace it?  Just because of a redball, or did you investigate the reasons for the redball?

 

As jonathanm said, Reiser is amazingly resilient, and the odds are good that your data is recoverable.  Just don't write ANYTHING to the server at all, because any writes will probably go to the new (xfs) drive and will impair the recovery attempts.

 

 

so no go on the reiserFSchk...

 

Can I add the old "disk1" as disk6 to the array?

 

 

so no go on the reiserFSchk...

 

Can I add the old "disk1" as disk6 to the array?

What was the result from reiserfsck?

 

You could add the old disk1 as disk 6, but then if it actually is marginal and ready to drop at a moment's notice then its a big PITA to remove the drive from the system and still maintain parity.

 

Myself, I would mount it via S.N.A.P.

Or as a cache drive if you don't already have one

so no go on the reiserFSchk...

what exactly did you type, were you in maintenance mode, and what was the output of the command?

So, I did the reiserFSchk this morning, can't recall the message but basically it had nothing.  I am sure I hosed it up badly.

 

At any rate, my new disk1 and array is fully online and I was able to mount the old disk1 with SNAP and recover my most critical data that was on the drive.  Now I am recovering all the "tier 2" stuff that I wouldn't be so upset about losing.

 

 

TBH, I caused my own nightmare by not paying attention and not having replaced a failed drive in over a year or two.  I do think unraid should give some extra "Hey idiot.. you may not want to do that" warning, but honestly, this was all on me

 

Thanks all for the help!

 

Mike

 

 

 

Ok.  Np.  BTW, the command you were supposed to run is not reiserFSchk,

 

but

 

reiserfsck --check /dev/md1

Ok.  Np.  BTW, the command you were supposed to run is not reiserFSchk,

 

but

 

reiserfsck --check /dev/md1

 

Thanks Squid, that is exactly what I ran and the results were no volume or something basically "nothing to process".  Can't recall

 

A few thoughts after reading this thread ...

 

The fundamental problem was that you used the wrong steps to initiate the rebuild.

 

Looking at the steps you outlined ...

... this is the detail of what happened:

1) rebooted unraid

2) Disk1 was redballed (ResierFS)

3) shut down array powered off and replaced disk 1 (2tb) with new 3tb precleared disk

4) booted unraid

5) selected new disk as disk 1 placement

6) started array figuring it will trigger the rebuild and it showed unformatted disk

7) I selected the checkbox to format and unraid started formatting disk1 as XFS

8 ) Array showed drive rebuilding

 

After step 2, what you SHOULD have done was stopped the array; unassigned disk1; restarted the array so it showed disk2 missing; and THEN shut down the array and replaced disk1 with the new disk.

 

Then when you rebooted; assign the new disk as disk1, and then the rebuild would have proceeded just fine.

 

For whatever reason, when you missed that step, you were presented with a Format option in Step 6 that should not have been there -- and, as already noted, when you selected Format you reformatted the emulated disk ... and that's what was then rebuilt.

 

 

There WAS a way to recover and start the rebuild over as long as you had NOT written any new data to the array; but since you've already been able to recover your data by mounting the old disk it seems you're essentially okay.    Any data you can't recover from the failed disk can simply be copied from your backups.

 

A few thoughts after reading this thread ...

 

The fundamental problem was that you used the wrong steps to initiate the rebuild.

 

Looking at the steps you outlined ...

... this is the detail of what happened:

1) rebooted unraid

2) Disk1 was redballed (ResierFS)

3) shut down array powered off and replaced disk 1 (2tb) with new 3tb precleared disk

4) booted unraid

5) selected new disk as disk 1 placement

6) started array figuring it will trigger the rebuild and it showed unformatted disk

7) I selected the checkbox to format and unraid started formatting disk1 as XFS

8 ) Array showed drive rebuilding

 

After step 2, what you SHOULD have done was stopped the array; unassigned disk1; restarted the array so it showed disk2 missing; and THEN shut down the array and replaced disk1 with the new disk.

 

Then when you rebooted; assign the new disk as disk1, and then the rebuild would have proceeded just fine.

 

For whatever reason, when you missed that step, you were presented with a Format option in Step 6 that should not have been there -- and, as already noted, when you selected Format you reformatted the emulated disk ... and that's what was then rebuilt.

What difference would that have made?  In between steps 5 and 6 he selected the drive to be xfs.  Wouldn't the same result would have happened anyways? Nevermind, I read your response wrong.  As long as he never changed the format of the drive (like he did somewhere between 5 & 6) everything would have been hunky-dory.  (Which we all already know)

 

There WAS a way to recover and start the rebuild over as long as you had NOT written any new data to the array; but since you've already been able to recover your data by mounting the old disk it seems you're essentially okay.    Any data you can't recover from the failed disk can simply be copied from your backups.

For my own reference, what is that procedure?  By the time the OP was posted, the format procedure was long over and the rebuild already in progress.  It can't get any worse by continuing.  Stopping the rebuild after the format is over will still have the same formatted data on the emulated drive.

 

Also, I'm curious as to exactly when we would have switched the disk settings from XFS back to Reiser in order to salvage the data.  Prior to running reiserfsck or after?

I put up a defect report for this. I thought someone else was going to on one of the previous incarnations of this, but I didn't find one. There have been too many of these, and if you follow the links in my defect report, it seems not all of these were caused by trying to change file systems. Not always clear what the user did in some cases. I don't have a test array so I will not be trying to reproduce this myself, maybe someone else can.