How To: Restart emhttp WebUI without rebooting

I've seen this asked many times but never answered or the answers didn't work for me. Anyhow, here is what I do to fix my WebUI when it stops responding.

Step One: Clear your browser cache and try connecting with a different browser. If neither of these work your emhttp process might have become unresponsive and here's what you can do to shut down your array safely, restart emhttp, and bring it back up.

Disclaimer: The following procedure works well for me, however I am by no means an unRAID expert so DO THIS AT YOUR OWN RISK! I read in other places in the forum that emhttp is NOT DESIGNED TO BE RESTARTED so this procedure is UNSUPPORTED by UNRAID / Lime Tech. Please note that I am running unRAID 5.0-rc11. Needless to say, this may or may not work for other versions.

Also, please let me know if you find any mistakes here or if you have any suggestions for improvement.

This guide assumes you have unMenu installed as well as screen and clean powerdown, which you can install via the unmenu package manager.

Step Two: Figure out what processes are accessing files on the array. The fuser command shows processes that access dirs and subdirs of the /mnt directory where the array is mounted. The lsof command lists files on the array opened by processes, often times this makes is very clear what add-ons are actually running.

# fuser -mv /mnt/*/* 
# lsof /mnt/*	

Step Three: Stop any processes that access any files on the array using the runscripts found in /etc/rc.d, e.g:

# /etc/rc.d/rc.sabnzbd stop
# /etc/rc.d/rc.couchpotato_v2 stop
# /etc/rc.d/rc.sickbeard stop

, etc.

Step Four: Re-issue the above fuser and lsof commands (step two). If you stopped everything, they will not return anything.

Step Five: Stop the array and kill the unresponsive emhttp process. The rc.unRAID runscript is part of unmenu clean powerdown and can be invoked to stop the array. It also gracefully shuts down vmware, samba, and nfsd and then tries to kill any other processes that are accessing files on the array, and may do so in a not so graceful way. However, if you used runscripts to shut down all processes accessing the array and confirmed as I explained in steps three and four there will be no processes left accessing the array. The script also saves a copy of your current syslog to your flash drive.

# /etc/rc.d/rc.unRAID stop

# killall emhttp

Step Six: List all mount points of the array and cache drive (if you have one). Unmount all of them.

# mount	| grep /mnt

# umount /mnt/disk1
# umount /mnt/disk2
# umount /mnt/disk3
# umount /mnt/cache

, etc.

Step Seven: Clean up /mnt. This is what unRAID expects when it starts the array. Don't worry, this will not delete any files in those directories. (If at this point you have any files in these directories, rmdir will throw an error ... and it would mean that you probably didn't unmount all of your drives)

# rmdir /mnt/disk* /mnt/user* /mnt/cache

Step Eight: Restart emhttp. If you restart emhttp directly within your telnet session it will be a CHILD process of your telnet shell, which means that it will be KILLED when your telnet session ends. One way around this is to run emhttp within a screen session. Screen can be installed through the unmenu package manager. Open a screen session and start emhttp like so:

# screen
# /usr/local/sbin/emhttp &

Step Nine: Done. You can detach the screen session by typing Ctrl-A d (i.e. first Ctrl-A then separately, 'd'). The screen process and all of its children will persist even after the telnet session ends.

You should now be able to browse to the WebUI and start the array.

Again, do this at your own risk. Please let me know if you find any mistakes here. Go easy on me though - this is my first how-to guide.

Thanks,

Stephan K

I have to ask: with the amount of time it would take to do all that why would not not just reboot?

Probably takes longer to tell it than to do it. If you don't have many drives or plugins it might not be too bad. I may try this next time I need it. I do wonder about restarting emhttp in screen though. I guess if you are logged in as root it would be the same as doing it from the console and I am usually headless.

It doesn't take that long to do this and it saves you a trip to the server and the anxiety over whether or not it will start back up.

I also just don't like the idea that I have to physically restart the computer in order to get a Linux application back up and running.

I have to ask: with the amount of time it would take to do all that why would not not just reboot?

A. Rebooting means all programs stop, E.G. RSync, sabnzbd, transmission, plex, cp, mv, any other scripts/programs that are running

B. Rebooting takes FOREVER, at-least for me, about 40 minutes (Although, my go script is huge, way too many scripts I should probably use screen/& for)

C. Refer back to A.

Basically, A.

It doesn't take that long to do this and it saves you a trip to the server and the anxiety over whether or not it will start back up.

 

I also just don't like the idea that I have to physically restart the computer in order to get a Linux application back up and running.

There's something called a "Restart" which means that it turns off, then, turns back on without requiring to push the button on the front of the case, it doesn't require any physical access? And for the people with IPMI, if it doesn't start up just open a ssh tunnel/VPN/port forward into IPMI and work out what's wrong.

Although, I still do concur, restarting is pretty damn annoying, even more so when remotely and I will give you a point for the anxiety part.

I have to ask: with the amount of time it would take to do all that why would not not just reboot?

 

A. Rebooting means all programs stop, E.G. RSync, sabnzbd, transmission, plex, cp, mv, any other scripts/programs that are running

B. Rebooting takes FOREVER, at-least for me, about 40 minutes (Although, my go script is huge, way too many scripts I should probably use screen/& for)

C. Refer back to A.

 

Basically, A.

I don't understand why it takes so long for you to reboot. Even when I was running tons of plugins on the server from the time I hit the stop array button to the time I was back up and running was 10 minutes tops.

I have to ask: with the amount of time it would take to do all that why would not not just reboot?

 

A. Rebooting means all programs stop, E.G. RSync, sabnzbd, transmission, plex, cp, mv, any other scripts/programs that are running

B. Rebooting takes FOREVER, at-least for me, about 40 minutes (Although, my go script is huge, way too many scripts I should probably use screen/& for)

C. Refer back to A.

 

Basically, A.

 

I don't understand why it takes so long for you to reboot. Even when I was running tons of plugins on the server from the time I hit the stop array button to the time I was back up and running was 10 minutes tops.

A. Loading OS bare (~ 4 minutes)

B. Waiting for internet connection (~ +1 minute, assuming that it's turning on because of a power cut. Modems has to request IP from ISP and router has to request IP from modem and unraid has to request IP from router)

C. Installing loads of plugins (~ +8 minutes)

D. Launch wake on lan script (~ +0 minutes)

E. Send a couple of emails, play a few beeps throughout the whole go script (All included) (~ +2 minutes)

F. Periscope (+ ~25 minutes)

G. Rsync (+ ~20 minutes)

Basically periscope and rsync, and, yes, I know about screen & telling bash not to wait for a script (&) however, I'm keeping it my way.

Your instructions are flawed in that rc.unRAID is NOT a part of unRAID as distributed by lime-technology.

See the screen capture of the actual distribution of /etc/rc.d/ as viewed through the 7zip archive manager looking at the contents of bzroot.

Joe L.

Your instructions are flawed in that rc.unRAID is NOT a part of unRAID as distributed by lime-technology.

 

See the screen capture of the actual distribution of /etc/rc.d/ as viewed through the 7zip archive manager looking at the contents of bzroot.

 

Joe L.

Well, I have it. I presume simplefeatures in that case?

Your instructions are flawed in that rc.unRAID is NOT a part of unRAID as distributed by lime-technology.

 

See the screen capture of the actual distribution of /etc/rc.d/ as viewed through the 7zip archive manager looking at the contents of bzroot.

 

Joe L.

 

Well, I have it. I presume simplefeatures in that case?

I think it's part of the powerdown package/plugin

"... Your instructions are flawed in that rc.unRAID is NOT a part of unRAID as distributed by lime-technology. "  ==>  No, the instructions are fine.

I would agree with Joe's comment EXCEPT ... the instructions CLEARLY note:

"... This guide assumes you have unMenu installed - if you don't have it you should get it. "

"... Your instructions are flawed in that rc.unRAID is NOT a part of unRAID as distributed by lime-technology. "  ==>  No, the instructions are fine.

 

I would agree with Joe's comment EXCEPT ... the instructions CLEARLY note:

 

"... This guide assumes you have unMenu installed - if you don't have it you should get it. "

(and most certainly does not by default install /etc/rc.d/rc.unRAID )

As mentioned, that file is part of the clean-powerdown package.  However, it is not installed unless you install it. 

In my opinion, it is best to not use it to stop your add-ons, since it does NOT cleanly stop them, it just issues KILLS to processes with open files.

Joe L.

Thanks for the (loud) clarification ==> so the author's comment that "... The rc.unRAID runscript is part of unmenu and can be invoked to stop the array ..." is incorrect unless Clean Powerdown is installed.

I suspect Clean Powerdown and APC are the two most-used packages in UnMenu (Screen is probably the only potential competitor for that title) ... but nevertheless, I agree it should be noted that Clean Powerdown must be installed for this set of instructions to work.

unMENU DOES NOT (by default) install anything. 

(and most certainly does not by default install /etc/rc.d/rc.unRAID )

As mentioned, that file is part of the clean-powerdown package.  However, it is not installed unless you install it.

In my opinion, it is best to not use it to stop your add-ons, since it does NOT cleanly stop them, it just issues KILLS to processes with open files.

Thanks for catching this mistake - I modified the instructions to reflect that clean-powerdown is required. The idea behind rc.unRAID stop is to get rid of any unresponsive processes after you're tried to gracefully shut down all services manually using rc.* commands.

Stephan K

unMENU DOES NOT (by default) install anything. 

(and most certainly does not by default install /etc/rc.d/rc.unRAID )

As mentioned, that file is part of the clean-powerdown package.  However, it is not installed unless you install it.

In my opinion, it is best to not use it to stop your add-ons, since it does NOT cleanly stop them, it just issues KILLS to processes with open files.

 

Thanks for catching this mistake - I modified the instructions to reflect that clean-powerdown is required. The idea behind rc.unRAID stop is to get rid of any unresponsive processes after you're tried to gracefully shut down all services manually using rc.* commands.

 

Stephan K

something like

fuser -ik /dev/md* /mnt/cache

should do it.  (it will interactively ask you if you wish to send SIGKILL to the processes it detects)

I think the breakthrough you've discovered is that if the array is stopped, and all the disks un-mounted and all the mount points removed you can re-start emhttp.    You can probably use

nohup /usr/local/sbin/emhttp &

to restart emhttp where it will remain running after you log off.

Unfortunately, if you've stopped the array, and un-mounted the disks, and removed the /mnt/disk* mount points, it is just as easy to reboot and, in fact, probably better, since if emhttp crashed it probably ran out of memory, and you've done nothing directly to fix that. (indirectly perhaps, but only if one of your add-ons was using it all until killed)

Typically, when you lose control of emhttp and wish to restart it, it is because you wish to regain control of your server after emhttp stops responding.

Most often, that occurs when emhttp is killed in an out-of-memory event.  (since it was the process idle the longest )

Joe L.

I still think it's useful to be able to restart the Web GUI without requiring a reboot.    Not really necessary;  but just nice to know it can be done -- and it may allow an orderly shutdown that won't force a post-boot parity check.

I had a similar issue a couple months ago with my media server (running v4.7) ... and after posting a question about it, decided to try the sequence that some suggested => it worked perfectly.  Details are here:  http://lime-technology.com/forum/index.php?topic=26476.msg231685#msg231685

... I never did bother to reboot it; but have run 3 parity checks since then;  copied a couple hundred gigs to the server; and streamed numerous movies, so I'm confident it's working fine.    As I noted then, I generally only restart after (a) a version update; or (b) a power-failure that results in a shutdown via the APC UPS package.    Current runtime is 134 days (since a power outage in December).  I think v4.7 has only been booted 3 times (possibly 4).

My 2nd server runs v5 RC12a, and has never had the Web GUI stop working, so I've not had to try anything.    Based on the segfault issues outlined in the posts I read before, my plan would have been to just reboot if that ever happened.    But this thread provides a nice set of instructions that could avoid the need for the reboot.    It sounds like the "fuser -ik /dev/md* /mnt/cache" command is perhaps an improvement in the process.  Exactly WHERE in the sequence should this be used?    Is it a replacement for "Step Five" ??  ... and if so, do you still need the # killall emhttp  command?

I would use fuser for that.  Much easier

something like

fuser -ik /dev/md* /mnt/cache

should do it.  (it will interactively ask you if you wish to send SIGKILL to the processes it detects)

 

I think the breakthrough you've discovered is that if the array is stopped, and all the disks un-mounted and all the mount points removed you can re-start emhttp.    You can probably use

nohup /usr/local/sbin/emhttp &

to restart emhttp where it will remain running after you log off.

 

Unfortunately, if you've stopped the array, and un-mounted the disks, and removed the /mnt/disk* mount points, it is just as easy to reboot and, in fact, probably better, since if emhttp crashed it probably ran out of memory, and you've done nothing directly to fix that. (indirectly perhaps, but only if one of your add-ons was using it all until killed)

 

Typically, when you lose control of emhttp and wish to restart it, it is because you wish to regain control of your server after emhttp stops responding.

Most often, that occurs when emhttp is killed in an out-of-memory event.  (since it was the process idle the longest )

 

Joe L.

Thanks, nohup is a great idea. I will try that next time emhttp stops responding and update the how-to guide then.

My issue with fuser -ik is that I want to encourage people to safely shut down their services before sending out SIGKILLs. I suppose invoking fuser -ik could be presented as an alternative or second line approach.

FYI, I've had emhttp stop responding a couple times on my server, and there was nothing about segfault or out-of-memory in the syslog (my server has 12 GB of RAM). Restarting using my method allowed me to regain control over my server.

I agree that it might be better to restart the computer but like many others I think rebooting the server is annoying, anxiety-provoking, and I feel like it's just not something you should have to do on a Linux-based system.

What I'd really like to see is a run control script for emhttp. Part of the reason why I decided to post these instructions was to facilitate the implementation of an rc script that takes all of the necessary steps for safely restarting the webui.

Thanks again,

Stephan K

 

... Part of the reason why I decided to post these instructions was to facilitate the implementation of an rc script that takes all of the necessary steps for safely restarting the webui.

 

Thanks again,

Stephan K

GREAT Idea.  Especially for us non-Linux guys who'd like to have a simple (and Safe) way to  just

(a) Telnet to UnRAID,

(b) logon, and

© type a command

to restart the Web GUI if ever needed 

Now if someone would just create that script !!