System never starts

[Bossman]

Hello,

I have an unRAID Plus box running 4.2.1.  It was unresponsive the other day and someone just hit the power button.

The system rebooted, and now you can get to the web interface, but it says "starting".... for over 36 hours now.  The system is using the previously standard ASUS motherboard and has 4 drives in it.  One Seagate 1TB and 3 x 500 GB Western Digitals.

Any suggestions on how to get this to finish starting or ??

I have not spent lots of time, but I cannot seem to find info on if I can SSH or telnet into it or if I have to plug in a keyboard and mouse.

Thanks in advance.

[Joe L.]

I have not spent lots of time, but I cannot seem to find info on if I can SSH or telnet into it or if I have to plug in a keyboard and mouse.

Thanks in advance.

You are right... you did not spend much time

Instructions are in the wiki: http://lime-technology.com/wiki/index.php/Telnet

and here:

http://lime-technology.com/wiki/index.php/Troubleshooting

[Bossman]

Thanks Joe.

Any thoughts on why it hasn't started in over 36 hours or is this one of those problems that no one has heard of and needs to be pulled apart?  Would an upgrade to 4.4.x or 4.5.x be of any help if I can get it shut down and pull the USB key?

[Joe L.]

Thanks Joe.

Any thoughts on why it hasn't started in over 36 hours or is this one of those problems that no one has heard of and needs to be pulled apart?  Would an upgrade to 4.4.x or 4.5.x be of any help if I can get it shut down and pull the USB key?

What would help is for you to telnet to the server and to get a copy of the syslog.   At this point, your question, without more detail, is exactly like me calling a random auto-mechanic and saying "Something is wrong with my car, can you tell me what part number I need to buy?" 

Even knowing the symptom (car won't start) will not translate to a specific part number unless we know a lot more...

To help you more we must see the syslog.  As far as it saying "Starting"  it will say that until you refresh the browser... have you tried a "refresh" of the browser to see if the status changed?

At this point in time, no upgrade of the OS is needed, and it would only complicate your problems.   Whatever you do, DO NOT press the button labeled "Restore" as it does not restore data.  It resets your drive configuration to before parity was calculated.  If you press it when a drive has failed, and without specific instruction, you will lose the data on the failed drive.  Also, DO NOT press the "Format" button if you see a drive is "Unformatted"  That just indicates a drive cannot be mounted.   Provide the syslog, and people will help you through getting the server back on-line.

If you cannot telnet in, then you must plug in a keyboard and monitor...

Joe L.

[Bossman]

Thanks,

So in my telnet session I'm looking at the usage stats.

There are 54 tasks total, 2 running & 52 sleeping.
CPU is at 99.7: - 100% at the id marker.

When I refresh, it still says starting. and the read and writes go up with no errors, all hard drives look to be identified correctly and are all showing green.  When I look in the shares tab, they are all showing, but I can only access the flash across the network.



The end of my syslog is below.  Complete syslog.txt attached.


Aug 19 11:55:30 VMUnRaidRack81 emhttp[1159]: unRAID System Management
Utility version 4.2.1
Aug 19 11:55:30 VMUnRaidRack81 emhttp[1159]: Copyright (C) 2005-2007, Lime
Technology, LLC
Aug 19 11:55:30 VMUnRaidRack81 emhttp[1159]: Plus key detected, registered
to: VM Systems
Aug 19 11:55:30 VMUnRaidRack81 emhttp[1159]: shcmd (1): cp
/boot/config/group /etc
Aug 19 11:55:30 VMUnRaidRack81 emhttp[1159]: shcmd (2): cp
/boot/config/passwd /etc
Aug 19 11:55:30 VMUnRaidRack81 emhttp[1159]: shcmd (3): cp
/boot/config/smbpasswd /etc/samba/private
Aug 19 11:55:30 VMUnRaidRack81 emhttp[1159]: Device inventory:
Aug 19 11:55:30 VMUnRaidRack81 emhttp[1159]: pci-0000:00:05.0-scsi-0:0:0:0
(sda) scsi-SATA_ST31000340NS_9QJ1X9PL
Aug 19 11:55:30 VMUnRaidRack81 emhttp[1159]: pci-0000:00:05.1-scsi-1:0:0:0
(sdd) scsi-SATA_WDC_WD5000YS-01_WD-WCANU2342220
Aug 19 11:55:30 VMUnRaidRack81 emhttp[1159]: pci-0000:00:05.1-scsi-0:0:0:0
(sdc) scsi-SATA_WDC_WD5000YS-01_WD-WCANU2327222
Aug 19 11:55:30 VMUnRaidRack81 emhttp[1159]: pci-0000:00:05.0-scsi-1:0:0:0
(sdb) scsi-SATA_WDC_WD5000ABYS-_WD-WCAPW1755899
Aug 19 11:55:30 VMUnRaidRack81 emhttp[1159]: shcmd (4): rmmod md-mod
>>/var/log/go 2>&1
Aug 19 11:55:30 VMUnRaidRack81 emhttp[1159]: shcmd (4): exit status: 1
Aug 19 11:55:30 VMUnRaidRack81 emhttp[1159]: shcmd (5): modprobe md-mod
super=/boot/config/super.dat slots=8,0,8,16,8,32,8,48,0,0,0,0
>>/var/log/go 2>&1
Aug 19 11:55:30 VMUnRaidRack81 kernel: [  139.934150] md: unRAID driver
0.92.0 installed
Aug 19 11:55:30 VMUnRaidRack81 kernel: [  140.277330] md: xor using
function: p5_mmx (7010.000 MB/sec)
Aug 19 11:55:30 VMUnRaidRack81 emhttp[1159]: shcmd (6): /usr/sbin/hdparm
-S244 /dev/sda >/dev/null
Aug 19 11:55:30 VMUnRaidRack81 emhttp[1159]: shcmd (7): /usr/sbin/hdparm
-S244 /dev/sdb >/dev/null
Aug 19 11:55:30 VMUnRaidRack81 emhttp[1159]: shcmd (8): /usr/sbin/hdparm
-S244 /dev/sdc >/dev/null
Aug 19 11:55:31 VMUnRaidRack81 emhttp[1159]: shcmd (9): /usr/sbin/hdparm
-S244 /dev/sdd >/dev/null
Aug 19 11:55:31 VMUnRaidRack81 emhttp[1159]: shcmd (10): killall -w smbd nmbd
Aug 19 11:55:32 VMUnRaidRack81 emhttp[1159]: shcmd (11): /usr/sbin/nmbd -D
Aug 19 11:55:32 VMUnRaidRack81 emhttp[1159]: shcmd (12): /usr/sbin/smbd -D
Aug 19 11:55:32 VMUnRaidRack81 emhttp[1159]: driver cmd: start STOPPED
Aug 19 11:55:32 VMUnRaidRack81 kernel: [  141.707917] mdcmd (3): start
Aug 19 11:55:32 VMUnRaidRack81 kernel: [  141.708918] unraid: allocated
9078kB
Aug 19 11:55:32 VMUnRaidRack81 kernel: [  141.709052] md1: running, size:
488386552 blocks
Aug 19 11:55:32 VMUnRaidRack81 kernel: [  141.709065] md2: running, size:
488386552 blocks
Aug 19 11:55:32 VMUnRaidRack81 kernel: [  141.709077] md3: running, size:
488386552 blocks
Aug 19 11:55:32 VMUnRaidRack81 emhttp[1159]: driver cmd: check
Aug 19 11:55:32 VMUnRaidRack81 kernel: [  141.861079] mdcmd (5): check
Aug 19 11:55:32 VMUnRaidRack81 kernel: [  141.861087] md: recovery thread
got woken up ...
Aug 19 11:55:32 VMUnRaidRack81 emhttp[1159]: shcmd (13): udevsettle
Aug 19 11:55:32 VMUnRaidRack81 kernel: [  141.862859] md: recovery thread
has nothing to resync
Aug 19 11:55:32 VMUnRaidRack81 emhttp[1216]: shcmd (14): mkdir /mnt/disk1
Aug 19 11:55:32 VMUnRaidRack81 emhttp[1216]: shcmd (15): mount -t reiserfs
-o noatime,nodiratime /dev/md1 /mnt/disk1  >/dev/null 2>&1
Aug 19 11:55:32 VMUnRaidRack81 kernel: [  141.867038] ReiserFS: md1: found
reiserfs format "3.6" with standard journal
Aug 19 11:55:32 VMUnRaidRack81 kernel: [  141.867055] ReiserFS: md1: using
ordered data mode
Aug 19 11:55:32 VMUnRaidRack81 emhttp[1220]: shcmd (14): mkdir /mnt/disk2
Aug 19 11:55:32 VMUnRaidRack81 emhttp[1220]: shcmd (15): mount -t reiserfs
-o noatime,nodiratime /dev/md2 /mnt/disk2  >/dev/null 2>&1
Aug 19 11:55:32 VMUnRaidRack81 kernel: [  141.871363] ReiserFS: md2: found
reiserfs format "3.6" with standard journal
Aug 19 11:55:32 VMUnRaidRack81 kernel: [  141.871384] ReiserFS: md2: using
ordered data mode
Aug 19 11:55:32 VMUnRaidRack81 emhttp[1224]: shcmd (14): mkdir /mnt/disk3
Aug 19 11:55:32 VMUnRaidRack81 emhttp[1224]: shcmd (15): mount -t reiserfs
-o noatime,nodiratime /dev/md3 /mnt/disk3  >/dev/null 2>&1
Aug 19 11:55:32 VMUnRaidRack81 kernel: [  141.875638] ReiserFS: md3: found
reiserfs format "3.6" with standard journal
Aug 19 11:55:32 VMUnRaidRack81 kernel: [  141.875658] ReiserFS: md3: using
ordered data mode
Aug 19 11:55:32 VMUnRaidRack81 kernel: [  141.885232] ReiserFS: md3:
journal params: device md3, size 8192, journal first block 18, max trans
len 1024, max batch 900, max commit age 30, max trans age 30
Aug 19 11:55:32 VMUnRaidRack81 kernel: [  141.885747] ReiserFS: md3:
checking transaction log (md3)
Aug 19 11:55:32 VMUnRaidRack81 kernel: [  141.885877] ReiserFS: md2:
journal params: device md2, size 8192, journal first block 18, max trans
len 1024, max batch 900, max commit age 30, max trans age 30
Aug 19 11:55:32 VMUnRaidRack81 kernel: [  141.886466] ReiserFS: md2:
checking transaction log (md2)
Aug 19 11:55:32 VMUnRaidRack81 kernel: [  141.886668] ReiserFS: md1:
journal params: device md1, size 8192, journal first block 18, max trans
len 1024, max batch 900, max commit age 30, max trans age 30
Aug 19 11:55:32 VMUnRaidRack81 kernel: [  141.887253] ReiserFS: md1:
checking transaction log (md1)
Aug 19 11:55:32 VMUnRaidRack81 kernel: [  142.006196] ReiserFS: md3: Using
r5 hash to sort names
Aug 19 11:55:32 VMUnRaidRack81 kernel: [  142.006247] ReiserFS: md2: Using
r5 hash to sort names
Aug 19 11:55:32 VMUnRaidRack81 kernel: [  142.032222] ReiserFS: md1: Using
r5 hash to sort names
Aug 19 11:55:32 VMUnRaidRack81 kernel: [  142.136173] can't shrink
filesystem on-line
Aug 19 11:55:32 VMUnRaidRack81 kernel: [  142.136386] can't shrink
filesystem on-line
Aug 19 11:55:32 VMUnRaidRack81 kernel: [  142.160847] can't shrink
filesystem on-line
Aug 19 11:55:32 VMUnRaidRack81 emhttp[1159]: shcmd (14): mkdir /mnt/user
Aug 19 11:55:32 VMUnRaidRack81 emhttp[1159]: shcmd (15): shfs /mnt/user 0
Aug 19 11:56:08 VMUnRaidRack81 in.telnetd[1239]: connect from
192.168.100.6 (192.168.100.6)
Aug 19 11:56:28 VMUnRaidRack81 telnetd[1239]: ttloop: peer died: EOF
Aug 19 11:59:32 VMUnRaidRack81 shfs: make_link: symlink: No space left on
device
Aug 19 11:59:32 VMUnRaidRack81 emhttp[1159]: synchronizing with shfs...
Aug 19 12:00:03 VMUnRaidRack81 last message repeated 30 times
Aug 19 12:00:33 VMUnRaidRack81 last message repeated 30 times
Aug 19 12:00:33 VMUnRaidRack81 in.telnetd[1261]: connect from
192.168.100.1 (192.168.100.1)
Aug 19 12:00:34 VMUnRaidRack81 emhttp[1159]: synchronizing with shfs...
Aug 19 12:01:05 VMUnRaidRack81 last message repeated 31 times
Aug 19 12:01:07 VMUnRaidRack81 last message repeated 2 times
Aug 19 12:01:08 VMUnRaidRack81 in.telnetd[1263]: connect from
192.168.100.6 (192.168.100.6)
Aug 19 12:01:08 VMUnRaidRack81 emhttp[1159]: synchronizing with shfs...
Aug 19 12:01:28 VMUnRaidRack81 last message repeated 19 times
Aug 19 12:01:28 VMUnRaidRack81 telnetd[1263]: ttloop: peer died: EOF
Aug 19 12:01:29 VMUnRaidRack81 emhttp[1159]: synchronizing with shfs...
Aug 19 12:01:33 VMUnRaidRack81 last message repeated 4 times
Aug 19 12:01:34 VMUnRaidRack81 login[1262]: ROOT LOGIN  on `pts/0' from
`192.168.100.1'
Aug 19 12:01:34 VMUnRaidRack81 emhttp[1159]: synchronizing with shfs...
Aug 19 12:02:05 VMUnRaidRack81 last message repeated 31 times
Aug 19 12:03:06 VMUnRaidRack81 last message repeated 60 times
Aug 19 12:04:07 VMUnRaidRack81 last message repeated 61 times
Aug 19 12:05:08 VMUnRaidRack81 last message repeated 60 times
Aug 19 12:06:07 VMUnRaidRack81 last message repeated 59 times
Aug 19 12:06:08 VMUnRaidRack81 in.telnetd[1296]: connect from
192.168.100.6 (192.168.100.6)
Aug 19 12:06:08 VMUnRaidRack81 emhttp[1159]: synchronizing with shfs...
Aug 19 12:06:28 VMUnRaidRack81 last message repeated 19 times
Aug 19 12:06:28 VMUnRaidRack81 telnetd[1296]: ttloop: peer died: EOF
Aug 19 12:06:29 VMUnRaidRack81 emhttp[1159]: synchronizing with shfs...
Aug 19 12:06:54 VMUnRaidRack81 last message repeated 25 times
Aug 19 12:06:54 VMUnRaidRack81 emhttp[1159]: get_var: shareCount not found
Aug 19 12:06:55 VMUnRaidRack81 emhttp[1159]: synchronizing with shfs...
Aug 19 12:07:09 VMUnRaidRack81 last message repeated 14 times
Aug 19 12:07:09 VMUnRaidRack81 emhttp[1159]: shcmd (16): killall -HUP smbd
Aug 19 12:07:10 VMUnRaidRack81 emhttp[1159]: synchronizing with shfs...
Aug 19 12:07:41 VMUnRaidRack81 last message repeated 31 times

[Joe L.]

OK...

Very interesting...
One line that stands out is:
Aug 19 11:59:32 VMUnRaidRack81 shfs: make_link: symlink: No space left on device

So, to figure out what device it might be referring to, type:
df
at the telnet prompt

You might try "Stopping your array" if you can click on the button, then turning off "user-shares", then re-starting the array.  It might let you get to the files via the disk shares.

As it is, this line in the syslog seems to indicate that samba has been killed, so no shares will be seen on the LAN.
Aug 19 12:07:09 VMUnRaidRack81 emhttp[1159]: shcmd (16): killall -HUP smbd

[Bossman]

ok.... the df command gives me the following information:

/dev/sde1 use = 7%
/dev/md2 use = 78%
/dev/md3 use = 78%
/dev/md1 use = 78%
shfs use = 78%

This is consistent with the Disk status numbers in the main window of the GUI.

The system still says Starting... and only the refresh button is available, BUT... the shares are now appearing and we can access data.

Everything else as far as CPU information etc. looks to be the same but there are now 55 tasks total instead of 54

--- Edit ----

Despite the usage numbers reported, when I try to copy a shortcut (1 KB file) It tells me there is not enough disk space.  I am going to try to delete some backups.

So a quick delete now shows Disk 3 has 125 GB free (instead of 108 GB) and I can copy the shortcut.  5 minutes later... it still hasn't started.

[Bossman]

Update... 2 days later, but it still says starting.

People are using the server though. An automated backup ran and there are new files on the server.

3 drives each with over 100GB free and I cannot even copy a shortcut again.  So the system obviously is not reading the drive space properly.  Any suggestions on how to work towards a fix on this?

I'm in the process of deleting old backups.  I'm up to almost 500 GB free now.

[Joe L.]

Update... 2 days later, but it still says starting.

What browser are you using to see the web-management interface?  What Operating System?

If people are using the server, it cannot be "starting"  It must be "Started"

The screen will NOT refresh itself.  If your browser is caching the results, you might need to clear its cache.

You are apparently on version 4.2.1 of unRAID.  Do you realize the version of unRAID you are using is 25 versions ago...   

If anything, you might consider using a more current version as there are some major bugs that were fixed.  (The parity-swap feature is broken in your version... there is no way for you to swap in a new larger parity drive and use the existing parity drive to replace a failed drive in your array)

The entire user-share file system was re-written in the 4.3.x version...   You are fighting very old bugs.

I'm using 4.5beta6 and have no issues with it.  If you don't like to use "beta" versions, at least upgrade to 4.4.2.  It is 20 versions from the one you are running.

To upgrade all you need to do is to replace two files on the flash drive and reboot.  All your files and folders will be intact...

You can even rename the two existing files on your flash drive, bzroot and bzimage to bzroot.421 and bzimage.421, just in case you felt a need to revert to the older release... you can put them back to their old names and reboot.

Upgrade instructions are here: http://lime-technology.com/forum/index.php?topic=3032.0

Joe L.

[Bossman]

Thanks Joe.  I will try to upgrade and see what is happening then.

I'm browsing via IE 6 and have never had a problem with it caching the pages. I am also manually refreshing the page and the free space DOES increase as I delete files, but the command area continues to say starting.  I'm browsing from a Server running W2K3

I Have 2 boxes and will upgrade them both.  I will report back.

----------

The first sever came up perfectly after upgrading to 4.5b6.  The free drive space was the same, but it started right away and everything has been working perfect so far.

Thanks for your patience and time Joe.  It is appreciated.  I'm going to try to add unmenu now.  :-)