Superblock cannot be found

Long story short, I have been having problems with my array for several months now.  Suffice it to say, it appears the power supply was causing the problems.  Unfortunately, before I made this realization, 3 of my drives got zapped.  After a full and complete panic attack and a rewiring of the power, I tried bringing the machine up again.  Two of the drives came back, but one did not. 

Below is a list of things I tried.  I know I tried more than this, but I wasn't thinking about capturing it so the rest scrolled out of my PuTTY buffer.  And yes, I realize disk 12 has corruptions and I am in the process of repairing it (again)

I tried running

reiserfsck --check /dev/md10

and received the error:

reiserfs_open: the reiserfs superblock cannot be found on /dev/md10. 
Failed to open the filesystem. 

So, I dug into the forums.  I found another topic (http://lime-technology.com/forum/index.php?topic=19433.0) where Joe L. suggested looking at the physical device so I ran:

reiserfsck --check /dev/sdo1

and it reported:

No corruptions found

I tried running the following:

reiserfsck --scan-whole-partition --rebuild-tree /dev/md10

which produced:

reiserfs_open: the reiserfs superblock cannot be found on /dev/md10. Failed to open the filesystem

running this:

reiserfsck --rebuild-sb /dev/md10

produced:

reiserfs_open: the reiserfs superblock cannot be found on /dev/md10.

what the version of ReiserFS do you use[1-4]
        (1)   3.6.x
        (2) >=3.5.9 (introduced in the middle of 1999) (if you use linux 2.2, choose this one)
        (3) < 3.5.9 converted to new format (don't choose if unsure)
        (4) < 3.5.9 (this is very old format, don't choose if unsure)
        (X)   exit
1

Enter block size [4096]:


No journal device was specified. (If journal is not available, re-run with --no-journal-available option specified).
Is journal default? (y/n)[y]: y

Did you use resizer(y/n)[n]: n
rebuild-sb: no uuid found, a new uuid was generated (94a514ec-862c-462d-a863-b856b39d2b73)

rebuild-sb: You either have a corrupted journal or have just changed
the start of the partition with some partition table editor. If you are
sure that the start of the partition is ok, rebuild the journal header.
Do you want to rebuild the journal header? (y/n)[n]: y
Reiserfs super block in block 16 on 0x90a of format 3.6 with standard journal
Count of blocks on the device: 488378368
Number of bitmaps: 14905
Blocksize: 4096
Free blocks (count of blocks - used [journal, bitmaps, data, reserved] blocks): 0
Root block: 0
Filesystem is NOT clean
Tree height: 0
Hash function used to sort names: not set
Objectid map size 0, max 972
Journal parameters:
        Device [0x0]
        Magic [0x0]
        Size 8193 blocks (including 1 for journal header) (first block 18)
        Max transaction length 1024 blocks
        Max batch size 900 blocks
        Max commit age 30
Blocks reserved by journal: 0
Fs state field: 0x1:
         some corruptions exist.
sb_version: 2
inode generation number: 0
UUID: 94a514ec-862c-462d-a863-b856b39d2b73
LABEL:
Set flags in SB:
Mount count: 1
Maximum mount count: 30
Last fsck run: Thu May 10 21:30:45 2012
Check interval in days: 180
Is this ok ? (y/n)[n]: y
The fs may still be unconsistent. Run reiserfsck --check.

So, I tried:

reiserfsck --scan-whole-partition --rebuild-tree /dev/md10

and got this again:

reiserfs_open: the reiserfs superblock cannot be found on /dev/md10.
Failed to open the filesystem.

Thinking I may have messed things up I tried this again:

reiserfsck --check /dev/sdo1

and got:

###########
reiserfsck --check started at Thu May 10 21:33:43 2012
###########
Replaying journal: Trans replayed: mountid 73, transid 62942, desc 5680, len 1, commit 5682, next trans offset 5665
Trans replayed: mountid 73, transid 62943, desc 5683, len 1, commit 5685, next trans offset 5668
Trans replayed: mountid 73, transid 62944, desc 5686, len 1, commit 5688, next trans offset 5671
Replaying journal: Done.
Reiserfs journal '/dev/sdo1' in blocks [18..8211]: 3 transactions replayed
Checking internal tree.. finished
Comparing bitmaps..finished
Checking Semantic tree:
finished
No corruptions found
There are on the filesystem:
        Leaves 452420
        Internal nodes 2790
        Directories 772
        Other files 62551
        Data block pointers 449337607 (218 of them are zero)
        Safe links 0
###########
reiserfsck finished at Thu May 10 23:16:15 2012
###########

So it looks like everything is still there.

But, running:

reiserfsck --check /dev/md10

produced this again:

reiserfs_open: the reiserfs superblock cannot be found on /dev/md10.
Failed to open the filesystem.

I have looked at the SMART reports on the drive and all of the "trouble" indicators appear to be clear: Reallocated_Sector_Ct, Reallocated_Event_Count, Current_Pending_Sector, Offline_Uncorrectable, UDMA_CRC_Error_Count, and Multi_Zone_Error_Rate are all 0.  At the moment I don't have access to the actual report on the drive; I checked it immediately after restarting the system.  The repair of disk 12 is still running and I don't know how to get the actual report to attach to this post without using the web ui, which is down because of the repair.  If I need to post it I will as soon as I can get to it.

I am a linux newbie so I am unfamiliar with how to make the leap from the sdo1 working to the md10 not working and get something that will let me recover my data.  I have attached the syslog if it will help.  Can someone help me get my superblock back so I can get the array back up and running?

Any, and all, help is greatly appreciated.

Thank you.

syslog.zip

Is the array started?

True.  You won't see /dev/mdX devices unless the array is started.

The reiserfsck you ran shows no corruptions on the file system.  Occe you start the array and the disk is mounted, you should be in operation.

Joe L.

Yes, the array is running.  Sorry, I should have mentioned that at the beginning. 

I have restarted the system several times now and each time disk 10 is red-balled and says it is unformatted.

To be 100% certain I just ran reiserfsck --check //dev/md10 and got the "reiserfs_open: the reiserfs superblock cannot be found on //dev/md10." error.  I tried running reiserfsck --rebuild-sb /dev/md10 and got the same messages I had gotten originally when I ran this command.  So it looks like it might have worked, but it doesn't seem to have done anything. I say that because then I ran the --check again and got "reiserfs_open: the reiserfs superblock cannot be found on //dev/md10." again.

Is there some way to "force" the OS to see that the drive is "ok"?

Thank you.

reiserfsck.zip

OK, so I tried doing a little digging and tried to manually mount the disk.  Here is the transcript:

root@Tower:~# mkdir /mnt/disk10
root@Tower:~# mount -t reiserfs /dev/md10 /mnt/disk10
mount: wrong fs type, bad option, bad superblock on /dev/md10,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

So, no joy on that either.  Like I mentioned before, Linux is not my strongest OS and I'm not sure what else to try.  Any and all help is greatly appreciated.

Thank you.

What does Main show? Attach a screenshot.

At the moment my internet connection is down. I'm having to use my phone to send this reply. As for disk 10, it has a red ball and under the used column it says unformatted. The size is correct if it matters. I will post a snapshot once my connection is restored.

Thank you.

Here is the image:

I moved the cables around and the drive is now sdc instead of sdo, in case anyone was wondering why the letters changed.

Also, I realize my array is unprotected at the moment.  It's not been real fun the last few days.   Once I get this drive back up and running I'm going to get the parity rebuilt.

Thank you again.

Try this:

mkdir /mnt/mydisk10
mount -t reiserfs /dev/sdo1 /mnt/mydisk10
ls /mnt/mydisk10

Awesome! That worked!  Now, how do I get the server to it?

Thank you.

Another question... Admittedly not as important as the main one, but why didn't my attempt to do this (see Reply #4, above) work? 

Thank you again for all of your help.

YES!!!

I got it working!!! 

I took a leap of faith and assumed that since I could successfully mount the drive now that the array should be able to do it.  So, I shut the array down and ran initconfig from a PuTTY window, refreshed the main page, reassigned all of my drives to it and fired it back up.  All 12 drives came online and the parity started rebuilding.  AWESOME!!

THANK YOU FOR ALL OF YOUR HELP!!

Remember to check parity after it's built.

Absolutely. Thank you again for your help.