server locking up yet again...

After a new usb flash drive, a completley fresh install of EVERYTHING....a completed parity check, i am trying to set up couch potato to work with deluge..so i went to dockers and stop both of those dockers, i tried to restart them and now im locked out, no telnet access, no GUI access...getting fed up with this now...if i hold down the power button its going to start a parity check again..how can i shut it down safely..my appdata folder is on the array and set to cache only, i have noticed however that if i open the share on the array and the same share on the cache disc that some of the folders are different, ie there are folders that appear in the deluge directory on cache drive that are not in the same directory on the array. I have enjoyed using unraid for some time now but this is not fun..i seem to be constantly getting screwed.

we really need a syslog.  Please post a full syslog.

How do I get it off with the said problems stopping me... will it still be there after a hard reset ??..not sure how to get a full sys log off.

Attach a monitor and a keyboard and log into console locally?

Then you should be able to copy the syslog to your thumbdrive.

For those kind of situations there is a clean powerdown script.

We know that unRAID, especially the GUI, can lock up or go unresponsive.

If you don't have the powerdown script you will always trigger the parity check

when pressing the power button.

So I can access the console via keyboard and monitor where I can't access via telnet session and can't ping it ??.. I have always shut down the server using a short single press of the power button and has always worked in shutting down and not invoking a parity check. How do I go about copying the sys log to the usb,  what do I need to write on the command line if the keyboard/monitor let's me in ?

If you have an issue with your network, of course telnet and anything else won't work.

Yes, you should be able to log into console locally.

I can't remember where the log is located exactly - I will dig a bit and come back.

Edit:

How to get a syslog.

I have always shut down the server using a short single press of the power button and has always worked in shutting down and not invoking a parity check.

It is not always working. When emhttp is ging unresponsive the array will not get unmounted.

If telnet in and call powerdown or press the button, the stock powerdown will not unmount the array because of the locked up emhttp and cause an unclean shutdown.

This results in a parity check on next power-up.

If it worked until now, it means emhttp was OK.

so another 9 hour parity check later...finding one error as it has done the last few times, probably the disks dying where they have been hard shut down so many times. an hour after completion and i am trying to source some films, its locked me out yet again, i tried plugging in the keyboard and monitor but nothing, maybe it should have been in place prior to the crash ?..any way, i have no gui access, no putty access, no console access and no windows shares access..nothing..so i have hard shut down again, upon restart the parity is showing up now as a new device and is doing a parity sync, the drives have not been touched, i have doubl check the serials, its the same drive..this is what it was doing before i replaced the usb stick..just seem to be going around in circles here..

Don't worry, shutting down the server is not killing your drives.

You don't need to run the full parity check every time if you don't write to the server.

It is important to find out if your whole server locks up or only the network stack.

Please plug the keyboard before you start the server.

You may want to try to log into the local console when your server is working - just to get a feeling

how it works.

The local console should be accessible any time, except the machine freezes completely.

In that case you might have a serious hardware issue - but lets find out step by step.

ok, It seems to be behaving itself since it saw the parity drive as a new device and done a sync (have been here before though), it is currently saying valid configuration but has not yet done a check, the only writes to the arrays would be a few tv programmes it grabbed whilst in a fluid state, im beginning to think its happening after a check has been done, though I may be wrong. Anyway, I left it on and started watching a film on plex, it stopped working and I couldn't access anything, thinking it had crashed I then discovered that I could still access shares and dockers but cant access the gui, this is the first time this has happened, usually its a complete lock out so I have attached the sys.log, even I can see that plex has done something but I the only thing I have no access to is the GUI, I have been using IE as opposed to chrome the last few days.

Its to big to upload here and I only have a free pastebin account which has a limit of 512kb so I have uploaded to my google drive and added link to it, after I reboot server by hard reset ill install powerdown plugin..i cant access GUI to install powerdown script right now.

syslog with unresponsive GUI

EDIT: After hard resetting, it doesn't seem to want to boot...it fires up but the flash drive just blinks once then it just sits there with the fans humming..if I press the power button it just shuts off instantly.

Don't know if this applies to your situation, but you might want to check out this thread also

http://lime-technology.com/forum/index.php?topic=36688.msg349503#msg349503

For future reference, your syslog zipped to only 133KB, well below the maximum individual size of 192KB allowed on this forum.

syslog_47.zip

EDIT: After hard resetting, it doesn't seem to want to boot...it fires up but the flash drive just blinks once then it just sits there with the fans humming..if I press the power button it just shuts off instantly.

Check the thumbdrive on a windows machine.

Maybe corrupted due to the hard reset.

Seems to be fine, suffered another partial lock out which turned into a full lock out, this time i had the keyboard attached but could type anything as writing was scrolling at super speed..i have attached a video, for some reason its uploaded upside down, you will need to set the settings to HD to be able to make out the writing, its moving so fast its difficult to make out...if i get another lock up ill see if its the same. I installed the power down plugin for v6, am i right to understand that if i now press the power button it will invoke a proper shut down if i have no gui or access to anything else ?..i watched the console as i pressed ctrl+alt+del and i saw it it invoke powerdown 2.13 and shut itself down.

still getting unresponsive gui and dockers..however i can log into console..but then that just stagnates before i can pull off a syslog...wtf is going on...

i have sat now nearly two weeks with this keep crashing....i cant get a syslog after the crash happens because the console becomes unresponsive...lots of data appears on the screen and i cant type anything to dump a log...so what is happening ?? all the disks smart status is ok and i have tried different ram..what can i do next..im almost in two minds to buy a new motherboard and power supply as i am tearing out my hair with this..really not enjoying UNraid anymore feel all on my own with this..i have attached the last syslog i could dump but everything was working ok then.

syslog.txt

i have sat now nearly two weeks with this keep crashing....i cant get a syslog after the crash happens because the console becomes unresponsive...lots of data appears on the screen and i cant type anything to dump a log...so what is happening ?? all the disks smart status and i have tried different ram..what can i do next..im almost in two minds to buy a new motherboard and power supply as i am tearing out my hair with this..really not enjoying UNraid anymore

It might be worth getting back to as close to a virgin install as possible and seeing if it still crashes.  I was think something along the lines of backing up the contents of the USB stick (so you can get back to the current state easily) ; reformat it back to an empty state; extract the required release onto the flash; run the make_bootable file to make the flash bootable; copy back your license key;  copy back the config files (or redefine the array) and reboot the system.

This will eliminate any possibility of an add-on causing a problem.

This sounds an awful lot like the reiserfs problem some people are having in unRAID 6 b10+.  The fix is to convert your data drives to XFS.

Here are some relevant threads:

• http://lime-technology.com/forum/index.php?topic=37311.0
  
• http://lime-technology.com/forum/index.php?topic=35788.0
  
• http://lime-technology.com/forum/index.php?topic=37316.0
  

Just for the record, I had the same issue on the server I manage for my sister as well.  I converted it to XFS three weeks ago and haven't had any problems since.

...

These may also be relevant:

• http://lime-technology.com/forum/index.php?topic=37864.0
  
• http://lime-technology.com/forum/index.php?topic=38218.0
  
• http://lime-technology.com/forum/index.php?topic=38370.0
  

I have 7TB of data drives not including the parity. What sort of time is it going to take to convert them to xfs ?...also I am very much leaning to the idea that firstly converting the cache to xfs might resolve the problem given that it seems to start the crash when the docker images are doing things and they reside on the cache...

Converting the cache drive to XFS is fairly easy (because there isn't as much data to move around) so I started with that too.  But in my case it wasn't enough to stop the crashing.

Moving 7TB will take a few days at least.  Unless you can free up one of your drives you'll have to add a new drive to your array first. Format it as XFS (using the unRAID gui), then move everything off one of your existing drives to the new one.  When that drive is empty, format it as XFS and start moving data from the next one.  Lather, rinse, repeat. 

For me it seemed that the system was most unstable when it was under heavy use, so I shut down all my dockers while I was moving data around.  I didn't want it to crash during the move. 

I just used mc at the console to move data from one disk to another, but other people recommend using rsync to copy and verify the data before deleting it (there should be example syntax in the forums somewhere).  I had everything backed up in Crashplan, so I wasn't *too* worried about it.

Just be sure you do everything on the disk shares and not the user shares.  If you copy from a user share to a disk share you can lose data.

im going to need a full plan for this procedure...crash plan for starters..i will buy a new 3tb hdd and do as you suggested if the cache conversion doesnt do it

This Docker runs Crashplan great:

  http://lime-technology.com/forum/index.php?topic=33864.0

but unless you have an incredible internet connection, it will take the better part of a year to backup 7tb to the cloud, so that isn't really going to be helpful at this point.

well i have a new WD 3tb dive o order. i hav backed up my cache drive to the array using mc, i then stopped the array to to set the cache drive to XFS at which point after restarting prompted that it needed reformatting which i allowed..however, i noticed that it seem to hang at 'reormatting' then after a few click arounds it settled down..i now seem to keep seing this 'appdata' folder i had created still visible via smb and mc..it wont let me delete it saying it is not empty..i turn off the setting for that folder to make it cahe only but no joy..im not convinced it has foramtted properly..when i browse to the folder it has wierd files in it named fuse`hi..not sure whats going on

If you copied or moved the appdata folder from cache to the root of one of your array disks, then that is what is showing up in your user shares. Just move it back to cache and make sure there are no appdata folders still on your array disks.

If that doesn't fix it, post the results of

ls -al /mnt/user

and

ls -al /mnt/user0

Ok. . I reformatted the cache drive to xfs... weird thing is the back up I made of my app data folder for the docker images didn't copy any of the configuration files over for cp,  sabnzb etc so I had to redo settings for all of them, having said that, the problem of the server crashing and locking me out seems to be resolved, I say that with my breath held. Not quite ready to close as resolved yet but from what I see, changing the file system to xfs has cured it.

knew i should have kept my mouth shut.. piece of shit ... so now do i spend days of preclearing the data drives one by one ??and converting to XFS and waste more time ??..i completley removed all the dockers and did fresh installs...REALLY getting me pi**ed now