reiserfsck problem outstanding - I have a block error after --rebuild-tree :/

I have checked through the support section and it has put me onto running reiserfsck.  This on disk 3 found an error and recommended me to run --rebuild-tree which is taking FOREVER now...

I think I may have prematurely done this.  At the time I looked at it I noticed in the red it says not to run this unless with advice from an expert member

This process is taking FOREVER and it hasn't even gone above 0% yet.

But there are files I am sure are on other drives from my array that the array won't let me access either.

This has happened after updating to version 5.

I have run the prematurely haven't I... ?

Trying hard to be kind here.

Did you follow the directions when you upgraded? Specifically the part that talks about permissions, and running the new permissions script?

What EXACTLY did you type at the command line? Posting a picture of the screen showing the command you ran and the progress so far may be helpful.

Trying to be kind here? Like I'm an idiot?  Of course I'm an idiot at this compared to some others... it's kind of offensive to see stuff like that in support posts.  I have had my server running like a dream for months and it started stuffing up so I upgraded.

I followed the instructions on the Main page (not the forum or the tech support area) and those instructions really should be taken away because they apply to probably less than 1% of people.  I deleted my entire flash drive including the key.  I know that was stupid now and it has taken a lot of work to recover but when the instructions specifically tell you to do that and mention nothing about what to worry about then you think it's okay.  But I was stupid to so blindly follow those instructions and I realised after I've had all of these problems that is the case.

I don't know anything about a new permissions script, so that could very well be the problem.

These instructions - which are linked from the download page, should be replaced with a link to the forum.  They are surely going to cause more harm than good.

http://lime-technology.com/unraid-server-installation/

Man... I have really screwed things up. I know it's my fault, but I need to do something to fix it and I'm going to need help, so please be gentle with me because I'm touchy about this at the moment because I'm fearful of losing my data.  Wouldn't you be?

The drive that I ran reiserfsck --rebuild-tree on is saying it's unformatted now.

Now in the upgrade instructions I have done the permissions thing on the utils page, still no access to SOME of the directories.

I also can't seem to be able to ad a user, when I complete the form in the users page the new user just doesn't appear?

The instructions also say to Carefully examine the Identification strings for each disk.  How do I do this? 

So at this point I think I might have to recover the data from the bung drive from the parity?? But is that possible if the security model isn't fixed properly?

Post a screenshot of the main webGUI page so we can be sure we understand your situation.

Also, what was the exact reiserfsck command you ran?

It sounds like there is a good chance that your data is OK, except for the unformatted drive. If every other drive is good it will be possible to rebuild it from parity.

Don't do anything else without further instruction.

The string that I ran was reiserfsck /dev/md3 --rebuild-tree because when I did the check it came

Up with an error and suggested that method.

I actually think this drive may be dying anyway, it's a 1tb drive and I have a spare 2tb, if the security is what is stopping me from accessing files will the parity still have saved the data?

Can someone please provide me with instructions I should do next? I am

Happy to try and recover the data on that drive by changing the Hd...

So what I need to know is what do I need to do to make the data accessable and then how do I recover the bung drive? I'll look through the support forums for the data recovery info.

Okay so here are the images requested...

I also tried to attach my syslog but even zipped it is too big...

All it was that I hadn't added a user to the users page.  I've done that now and I can access all the files - however the problem with drive 3 is still there.

It's unformatted and when I just re-started the array after I added the user it started a bloody parity sync.  Thus - I have lost 1TB of data right?

Pretty much the only thing that I am thinking is going to help is if  reiserfsck /dev/md3 --rebuild-tree didn't totally destroy the drive and it can be recovered because I can't recover that data with the parity now.... unless there is a way to go back to the previous parity sync before this one started and I cancelled it?

It probably started the parity sync because you had an unsafe shutdown. Did you power down without stopping the array? That will always result in a parity sync.

Maybe a new screenshot since the one you already posted is not current now.

I used the stop button and restart option in the webgui to restart.  It restarted fine... until the parity sync started automatically again... I had turned off auto start as well.  Farkin.  I don't think anything crucial was on that drive, but I'll not know exactly what it was until I go looking for something that was there and then is gone I imagine.

The thing that I'm pissed off at myself the most about is that all of this could have been easily avoided if I had have looked in the right places and realised that a user name can't be created with a description that has spaces in it

Edit:  What I believe I need to do to recover the drive is in this thread - http://lime-technology.com/forum/index.php?topic=25006.0

However I am confused by the process that happened for him.  How did he re-build tree on a new drive and have it turn out okay?? His Parity was used to rebuild right?

Should I run the reiserfsck --check again?

Edit 2: Okay, I'm a little less freaked that I think I might be able to save this drive with the reiserfsck tool.  I'm running the --check now.  If I recommends to run the --rebuild-tree to fix it again I will try doing that again.

That screenshot has parity not installed and drive3 unformatted. Is this correct?

Yes... I took the parity offline so that it couldn't try and do a parity sync again in the hope that the previous parity sync will be saveable (but I don't think it is because it says "invalid" when it is mounted".

Drive 3 is the one that is saying unformatted which happened after running the reiserfsck /dev/md3 --rebuild-tree

Okay I got the results back from the reiserfsk --check it gave me this message:

"Bad root block 0. (--rebuild-tree did not complete)

Aborted (core dumped)"

I assume this means that the --rebuild-tree didn't complete (of course) and that it needs to be run again.  I'm not sure if I should wait for the go-ahead from you guys or just do it.  Not really anything else I can do right?

Can you access files on the other drives now?

Someone with more experience with reiserfsck will have to help you recover the data on that drive.

Yes, I can access the files on all the other drives.  All it took was creating a user... such a simple solution and I might have screwed a drive over it.

This is what I got from the check...

Looks like that drive is dying.  Can someone please walk me through what I need to do to try and do the bad blocks or -b option?  I don't know what the correct commands are and the info above doesn't state so.  I only want to do this so that I can re-build the parity and then replace the drive...

The only thing I can find now that even remotely seems to be the solution is spinrite??  This is not part of unraid though and I am weary of going anywhere near this as I have never run anything outside unraid before.  Can it be run through the telnet session?

I have looked back at your recent posts and it's unfortunate that you didn't seek or get help sooner. Almost all of the people on the forum are just trying to help fellow unRAID users and sometimes you have to be patient.

I think you probably had a plugin problem back on July 12 and instead of getting a good diagnosis (a syslog would have helped), you proceeded to upgrade unRAID, which would have probably been OK and we could have worked through any problems you had with that.

I think you really jumped the gun with reiserfsck. I have been using unRAID for 3+ years and I have never needed to use reiserfsck. It is an advanced technique and should only be done as a last resort when the more conventional approaches don't work. It is normally not needed to recover data. That is what rebuilding from parity is for, a technique I have used many times to upgrade to larger disks. It's not clear that there was any data that needed recovery anyway. If I ever get to a point where I think I need to use reiserfsck I will definitely be asking for help.

I really urge you to not do anything else without further instructions. Please get us a syslog. Then we can get a better idea of exactly what your problems are and how to proceed. We will probably need smart reports from some of the drives but get us a syslog first so we can see what drives, if any, are having problems.

Thanks for telling me all that, I'm already kicking myself for rushing into I know I screwed up big time...

I need to know how to fix it now. I don't think any of those extra details are going to help. I've waited 2 days for a response to the reiserfsck question now and still nada. I know I can't expect anything and it was stupid of me to do it but I'd still like to at least TRY and fix it.

I have to re-build problem blocks on that drive, and the drive had been getting errors for a while I think so I don't know for sure or not but I do believe this drive needs to be replaced which I'd love to do but I am doubting I will be able to now. 

I can't blame anyone but myself BUT in my defense those upgrade instructions on the main page are woefully inadequate and cause problems.

All I need to know is what string to put after reiserfsck with the info I provided previously because I've either messed the drive with --rebuild-tree or somehow the blocks are failing on it.  But it doesn't look like anyone else is going to weigh in... Is that just because I seem like a complete helpless idiot?

I say all that because everything else has been solved, I've just got one drive that I screwed up and is showing un formatted because I ran --rebuild-tree on it.

I want to know if I can do any MORE damage but running --rebuild-sb on the drive.

There are no instructions for this in the support area it specifically says to come and ask on the forum.

Since nobody has responded I will bump this for you.

One thing about the way you used the reiserfsck command that I noticed but thought might be TMI at the time. You used it on an md device instead of an sd device. Using the md device means that any changes made to the data drive also updated parity. See here.

Hopefully someone will see this now that I have bumped it and you will get some help.

These instructions - which are linked from the download page, should be replaced with a link to the forum.  They are surely going to cause more harm than good.

 

http://lime-technology.com/unraid-server-installation/

It is worth pointing out that these instructions are nothing to do with upgrading - they are for creating a new installation.

The actual upgrade instructions are linked to from the first post for each release, and also included as a readme.txt file in the download.

These instructions - which are linked from the download page, should be replaced with a link to the forum.  They are surely going to cause more harm than good.

 

http://lime-technology.com/unraid-server-installation/

It is worth pointing out that these instructions are nothing to do with upgrading - they are for creating a new installation.

 

The actual upgrade instructions are linked to from the first post for each release, and also included as a readme.txt file in the download.

True but it is linked from here: http://lime-technology.com/download/ 

These instructions - which are linked from the download page, should be replaced with a link to the forum.  They are surely going to cause more harm than good.

 

http://lime-technology.com/unraid-server-installation/

It is worth pointing out that these instructions are nothing to do with upgrading - they are for creating a new installation.

 

The actual upgrade instructions are linked to from the first post for each release, and also included as a readme.txt file in the download.

 

True but it is linked from here: http://lime-technology.com/download/

It seems pretty clear o me from Context that the link given there is for a new installation.

These instructions - which are linked from the download page, should be replaced with a link to the forum.  They are surely going to cause more harm than good.

 

http://lime-technology.com/unraid-server-installation/

It is worth pointing out that these instructions are nothing to do with upgrading - they are for creating a new installation.

 

The actual upgrade instructions are linked to from the first post for each release, and also included as a readme.txt file in the download.

 

True but it is linked from here: http://lime-technology.com/download/

It seems pretty clear o me from Context that the link given there is for a new installation.

This is pointless... Why do you feel the need to keep pointing out the fact of my errors and my mistakes and where and how I screwed up?? I'm completely aware of that now and perhaps I'm just dumb but I made the mistakes that I made... rather than telling me things that seem like well "ner ner serves you right" I'd prefer to actually hear something that might help...

I made that mistake, I've said that probably more than five times already.  I realise how stupid that was now but at the time I thought it wasn't stupid, but I realise it was now.  Even so,  I don't think it's such a stretch to think it's upgrade instructions considering it says to format the drive etc, if you can't see how I came to the conclusions I did then don't bother replying.

I'm sorry if that comes off as rude, but I've not received any help from the forum so far at all, which is exactly why I ran into problems by trying things myself and screwing things up myself.  I've following the troubleshooting from the site but reiserfsck says come to the forum for help which I've done and I'm not getting any help I'm just being told that I screwed up over and over again.  It's really disheartening.

If you are getting an I/O error trying to 'reiserfsck --rebuild-tree /dev/md3' then that physical disk is probably not recoverable via reiserfsck.  I do not see any reason to try and rewrite the superblock - the reiserfsck is not suggesting that it cannot read the superblock as far as I can see (in fact it would not have found the transaction log if the superblock could not be found)

However, one thing that is not clear is whether you think you have valid parity?    If so you might be able to recover what should be on the /dev/md3 device by physically removing the actual physical disk from the system so that is emulated using the combination of parity and the other data drives.  If that is the case then you can retry running the 'reiserfsck --rebuild-tree /dev/md3' command against the emulated drive to see if that allows it to be recovered (it is not always obvious that reiserfsck can be run against an emulated drive).  If that works you would then be able to copy the data off the emulated drive before trying to replace it with a replacement good drive.

This is pointless... Why do you feel the need to keep pointing out the fact of my errors and my mistakes and where and how I screwed up?? I'm completely aware of that now and perhaps I'm just dumb but I made the mistakes that I made... rather than telling me things that seem like well "ner ner serves you right" I'd prefer to actually hear something that might help...

 

I made that mistake, I've said that probably more than five times already.  I realise how stupid that was now but at the time I thought it wasn't stupid, but I realise it was now.  Even so,  I don't think it's such a stretch to think it's upgrade instructions considering it says to format the drive etc, if you can't see how I came to the conclusions I did then don't bother replying.

 

I'm sorry if that comes off as rude, but I've not received any help from the forum so far at all, which is exactly why I ran into problems by trying things myself and screwing things up myself.  I've following the troubleshooting from the site but reiserfsck says come to the forum for help which I've done and I'm not getting any help I'm just being told that I screwed up over and over again.  It's really disheartening.

Colonial -

Sorry you are having a problem with your array. I would suggest, though, that you not abuse those responding to your query. Whenever we have a serious array issue we get frustrated and get defensive. The key to solving a problem is clear thinking, understanding where we went wrong, and ultimately coming up with a solution (if one is possible). Everyone makes mistakes and no one is calling you stupid. But when you get abusive, that makes you look 100x worse than a user that made a mistake. Every one of us has been that user and every one of us has needed to reach out for help.

The most important thing in reporting a problem is an accurate accounting of what you did. More than anything else, and as a lessons learned, you need to be clear. I have read this thread a number of times trying to figure out what you did and how to help. But it is very confusing and others were involved. You lived it and your brain may be able to fill in the missing pieces, but to me (and probably others), we are only getting bits and pieces. For example, at some point you said you started the array and a parity sync started. And you canceled it. First, a parity sync is BUILDING parity. It is what happens when you first define an array and install a parity disk. It reads all of the disks and creates parity. This is different than a parity CHECK, which is what happens if you have a dirty shutdown and is used to verify that parity is correct. During a parity check, it will report sync errors. The most important thing for you to have noticed is were there any parity check mismatches, and if so how many. Also, how far into the parity check did it get. Was it a minute, 15 minutes, an hour? None of these details were captured or reported. But I don't know if it was a parity sync or parity check, or have any details. This would be very important information.

But, one way or another we are here. My understanding of the current state is that the array is back online but that one disk is being reported as unformatted, and attempts to run reiserfsck are failing due to a physical disk error. Your parity is in question. If parity were accurate, you could rebuild the disk onto a fresh disk and access the data. If parity is close to accurate, you could still do the rebuild and run reiserfsck against the result and it could likely recover all or most of your data (or run reiserfsck against the simulated disk as was offered - sort of the same thing). But if parity is badly hosed none of this will bear fruit, and your best chance is with the failing disk.

Once, long ago, I had a problem that you can read about HERE. I got an I/O error running reiserfsck similar to what you got. I ran spinrite to clear it. There may be other, newer, tools that can do the same thing, but once I overcame the issue, which was self inflicted if you read the story, I was able to get reiserfsck to do its job and recovered most of my data.

Short of that, the rebuild option is you next best hope. You could try that first as clearing your I/O error may be a challenge and require a lot of research to figure out how to do it. The rebuild and then running reiserfsck (if needed) is likely the best next step. I like it better than running it on the simulated disk as it is less stressful on the system. If that doesn't work, you'll still have the disk that is failing and can still pursue fixing your I/O error and running reiserfsck on that disk.

A couple of notes. Your array looks like it is running hot. I see disks running at 50C. Although there is conflicting info, I have always found that hot is bad and try to keep my disks running in the 30s or low 40s. To each his own but mentioning it.

I also see in one of the screenshots that you never ran a parity check after parity was built. This may be a result of the steps to redefine the array. But if you are not running parity checks you need to start. It is the only way we can gauge your ability to recover.

Hope this is helpful. Post back if I can help further.