Just for the developers... Refining the process of starting and stopping add-ons

OK... in several threads we've been discussing where to collect the scripts needed to start and stop add-on scripts on the unRAID array.

Typically, most will need to be stopped before the array can be stopped.  In the same way, most will need to be re-started when the "Start" button is pressed once the array is online.

Today, unRAID does not supply a pre-stop-array event for us to use as a trigger.  Likewise, it does not supply a post-array-start trigger.  It is expected those will be available in unRAID 5.0.

Until they are we can simulate their existence and refine how they will be used.

So far, I expect this sequence of events when stopping the array

[pre]

User presses the "Stop" button

    The pre-stop-event occurs.

        The /etc/rc.d/rc.unRAID script is invoked with a "stop" argument

                Each of the add-on rc.* scripts in /etc/rc.d/unraid.d directory are invoked with the "stop" argument

    Once the /etc/rc.d/rc.unRAID script exits, the array can be stopped as all add-ons should be terminated.

    unRAID (emhttp) will then un-mount the drives as needed, remove the /mnt/disk mount and stop the array

[/pre]

This sequence seems to indicate that /etc/rc.unRAID "stop" should NOT stop the array, nor un-mount disks, as they will be un-mounted by emhttp and the array stopped by emhttp.

Now, we do want /etc/rc.unRAID to invoke /etc/rc.d/unraid.d/rc.* "stop" to get the add-on processes terminated. We do want the disks un-mounted, and the array stopped when we issue a /sbin/powerdown command... but not when used for an array "stop"  by the user pressing the "Stop" button on emhttp.

To me, that seems to indicate that /etc/rc.d/rc.unRAID needs new arguments of "powerdown" (or "shutdown") rather than "stop" for when it will perform all the tasks and emhttp is not involved at all.

rc.unRAID "stop" will invoke all the /etc/rc.d/unraid.d/rc.* stop scripts.  It will not need to stop the array as emhttp will do that.

rc.unRAID "powerdown" will invoke all the /etc/rc.d/unraid.d/rc.* stop scripts AND then proceed to un-mount the disks, stop the array, log as needed, and power down.

rc.unRAID "shutdown" might invoke all the /etc/rc.d/unraid.d/rc.* stop scripts AND then proceed to un-mount the disks, stop the array, log as needed but not  power down.

In the same way, a post-start event would work something like this:

[pre]

User presses "Start"

    unRAID (emhttp) starts the array mounting the disks, etc.

        A post-start event is generated once the disks are on-line.

            the /etc/rc.d/rc.unRAID "start" is invoked by the post-start event.

                Each of the /etc/rc.d/unraid.d/rc.* start scripts are invoked in turn by /etc/rc.d/rc.unRAID.

[/pre]

Comments? Feedback?

To help with this, the attached script will allow WeeboTech and others to experiment.  It is able to accurately detect the array starting and had a "stub" to invoke /etc/rc.d/rc.unRAID start. (commented out)

It is also accurately able to detect the pre-array-stop event and again I have a "stub" to invoke /etc/rc.d/rc.unRAID stop. (also commented  out)

I cannot use the existing /etc/rd.unRAID as it does too much when when invoked with "stop" (It is currently doing what is needed when invoked with "powerdown", and, at least in my version, is apparently invoking "stop" events in the unRAID_start() function)

The attached unraid_addon_control.sh script can be invoked with the "-F" option to run in the foreground (and print to the screen as it detect the events), or leave it off to have it detach from the terminal and run itself in the background.  It logs to the syslog as it works in either case.  Terminate it with the "-q" option.  It is written as a finite-state-machine.

You can leave it running and play starting and stopping your array using the web-management console.

Again, this is for the developers out there... it may crash your server if the stubs are un-commented and it is invoked on the current rc.unRAID.  Let WeeboTech work up a compatible version of rc.unRAID first.  It will cause no harm as attached, as it only prints to the syslog what it would do, but not actually do it.

Oh yes, this script will NOT work with older versions of unRAID.  You must be on one of the later 4.5 versions (4.5beta7 and newer) or anything from 4.6 onward . I used a sneaky method to force the detection of the pre-stop event and force unRAID to wait for us to process our add-on stop events.  I purposely keep a data disk busy by changing directory to it. This results in an error message we can detect in the syslog when the stop button is pressed and the disk un-mount is attempted by emhttp.

Edit: I just attached the newest version of the unraid_addon_control.sh script.  It is now fully working code but still waits for an updated version of the powerdown script.  It can be used without the powerdown script's rc.unRAID assistance by using the "-u" option when invoking it.

Joe L.

unraid_addon_control.zip

The problem I see is that by preventing emhttp shutdown by holding a disk open, nothing tells emhttp to try again after you release the disk.

Plus. there has been some prior discussion of "stop now dammit" code requested to be included in emphttp, to clear any pesky open files or processes with open files, so a shutdown will succeed.... the logic here was most problems with shutting down are due to open files that users were unaware were open.  I was hopeful that the "stop now dammit" code would be added to emhttp in unRAID 5.x (or sooner).

I was under the impression that emhttp sits in a retry loop until the disks are no longer busy, as of 4.5beta7 or there abouts.

The problem I see is that by preventing emhttp shutdown by holding a disk open, nothing tells emhttp to try again after you release the disk.

This logic is in place now in emhttp.  Has been there since 4.5-beta7.

Plus. there has been some prior discussion of "stop now dammit" code requested to be included in emphttp, to clear any pesky open files or processes with open files, so a shutdown will succeed.... the logic here was most problems with shutting down are due to open files that users were unaware were open.  I was hopeful that the "stop now dammit" code would be added to emhttp in unRAID 5.x (or sooner).

You can see his response to me in this post and the few following it in the thread

http://lime-technology.com/forum/index.php?topic=4484.msg40363#msg40363  He also says the solution will be in 5.0 (implied with the pre-array-stop trigger)

I think we will need to add the "stop now dammit" code to our logic if appropriate.  Perhaps after several "stop" attempts on processes we are aware of.  This still does not help when a user has logged in and "cd'd to a disk.  They do not have an affiliated  rc.* stop event.

As I said.  This thread is to discuss the issues, and possible fixes.  We know specific tasks must be performed.  Others will be more evident once we start using the discovery process.  We've all had the "powerdown" script installed for a time now.  Did you know the "start" argument to it stopped things... probably not, since we've never used that code.

In the interim, the script I attached provides stubs for two triggering events.  We can use them and learn what we'll need and be ready, or not.    I want to be ready.

Joe L.

I was under the impression that emhttp sits in a retry loop until the disks are no longer busy, as of 4.5beta7 or there abouts.

Joe L.

This logic is in place now in emhttp.  Has been there since 4.5-beta7.

I didn't catch that.... That's what I get for keeping 4.4.2 on my production machine and using my own "stop not damnit" script.

OK... in several threads we've been discussing where to collect the scripts needed to start and stop add-on scripts on the unRAID array.

 

Typically, most will need to be stopped before the array can be stopped.  In the same way, most will need to be re-started when the "Start" button is pressed.

Refine that: ...In the same way, most will need to be (re)started after the array is fully started.

I'll play with your script, and post my comments tomorrow.

It occurs to me.... Limetech will probably be posting a new 4.5.x build soon to fix the 'unformatted' problem plaguing 4.5.3...

So I am thinking, it may be worth the shot if you email Tom and ask if he could possibly add one tiny little thing to emhttp in the new build -- simple syslog entries that look like this:

[...]
Mar 11 11:13:10 Tower emhttp: start_array completed.
[...]
Mar 11 11:19:26 Tower emhttp: initiating stop_array procedures...
[...]

I know it's a long shot, but who knows?

It occurs to me.... Limetech will probably be posting a new 4.5.x build soon to fix the 'unformatted' problem plaguing 4.5.3...

So I am thinking, it may be worth the shot if you email Tom and ask if he could possibly add one tiny little thing to emhttp in the new build -- simple syslog entries that look like this:

[...]
Mar 11 11:13:10 Tower emhttp: start_array completed.
[...]
Mar 11 11:19:26 Tower emhttp: initiating stop_array procedures...
[...]

I know it's a long shot, but who knows?

 

This was a good idea, I sent a note.

ps. My hand is getting better and I'm learning to type with only one finger on the left hand hahahahaha! 

Edit: I just attached the newest version of the unraid_addon_control.sh script to the first post in this thread. 

It is coded to either work through the /etc/rc.d/rc.unRAID script (and have it deal with the files in /etc/rc.d/unraid.d) , or, to work directly on the files in /etc/rc.d/unraid.d.

I've been using it with the "-u" option to work directly on the files in /etc/rc.d/unraid.d

I've currently got two scripts in that /etc/rc.d/unraid.d folder.  One is to start and stop the cache_dirs program.  The other to start and stop the spinup_when_accessed.sh program.

For now, I've put a file in /boot/packages named rc_files.auto_install

In it are these commands.    They create the unraid.d directory and copy to it the rc.cache_dirs and rc.spinup_when_accessed scripts.

mkdir -p /etc/rc.d/unraid.d

cp /boot/custom/etc/rc.spinup_when_accessed /etc/rc.d/unraid.d/rc.spinup_when_accessed

chmod +x /etc/rc.d/unraid.d/rc.spinup_when_accessed

cp /boot/custom/etc/rc.cache_dirs /etc/rc.d/unraid.d/rc.cache_dirs

chmod +x /etc/rc.d/unraid.d/rc.cache_dirs

The .auto_install files are all invoked by a single line added to the end of my config/go script. That line was added by the unMENU package manager the first time I requested a package ge auto-installed on re-boot:

cd /boot/packages && find . -name '*.auto_install' -type f -print | sort | xargs -n1 sh -c

The /boot/custom/etc/rc.cache_dirs file looks like this:

root@Tower:/boot/custom/bin# cat /boot/custom/etc/rc.cache_dirs

case $1 in

start)

  [ -x /boot/custom/bin/cache_dirs ] &&

  /boot/custom/bin/cache_dirs -w -e data -e Pictures -B

;;

stop)

  [ -x /boot/custom/bin/cache_dirs ] &&

  /boot/custom/bin/cache_dirs -q

;;

esac

The /boot/custom/etc/rc.spinup_when_accessed file looks like this:

root@Tower:/boot/custom/bin# cat /boot/custom/etc/rc.spinup_when_accessed

case $1 in

start)

  [ -x /boot/custom/bin/spinup_when_accessed.sh ] &&

  /boot/custom/bin/spinup_when_accessed.sh -u -e data -e Pictures -m "192.168.2.251"

;;

stop)

  [ -x /boot/custom/bin/spinup_when_accessed.sh ] &&

  /boot/custom/bin/spinup_when_accessed.sh -q

;;

esac

Both files have a "case" statement looking for a "start" and "stop" argument.  Both check to ensure the command being invoked is executable (using "test -x" before invoking it.)

I also have a /boot/packages/unraid_addon_control.auto_install file.  Its contents are:

/boot/custom/bin/unraid_addon_control.sh -u

When I boot, the two rc. scripts are copied from /boot/custom/etc to /etc/rc.d/unraid.d and the unraid_addon_control.sh script is invoked with the -u argument... directly operating on the files in /etc/rc.d/unraid.d  in other words, this is entirely independent of the powerdown command when used in this way.  Eventually, it should be able to be invoked with the "-t" option and that will operate through the /etc/rc.d/rc.unRAID script, invoking it with a "start" and "stop" argument as needed.

This has been in place for a few weeks now.  I've started and stopped my array dozens of times.  Each time it properly stops and re-starts the two scripts for caching directories and for spinning up drives when folders are accessed.

This is still primarily for the developers, but the technique of invoking rc.* scripts, and not being held forever if one does not exit in the expected time is really important.  No one rc.* script should keep the server from starting, or stopping.  the unraid_addon_control.sh script attached to the first post on this thread will run one rc script at a time as long as they take 30 seconds or less each.  Those taking more than 30 seconds are allowed to continue, but the next rc script is run in parallel.  there is a script timer, and affiliated inter-process semaphore file for each rc script that prevents any script from completely stopping the start of stop of the unRAID server.

When we eventually get a true before-stop, and after-start event trigger from unRAID 5.0 the same addon_processes function in unraid_addon_control.sh can be used with the same rc.* scripts in /etc/rc.d/unraid.d  In fact, I expect the same function, or something very like it will be incorporated into the rc.unRAID file WeeboTech created as part of the powerdown package.  (once he can type once more) 

I'll be ready for 5.0...

Joe L.

First of all thanks for all the hard work you put into your scripts Joe. This one is just what I need so I can stop using the very clumsy cron jobs.

I tried out this script and put a few rc scripts in the /etc/rc.d/unraid.d directories for it to run (crashplan, mediatomb, cache_dirs, rsyncd).

Now is it correct that this script will always think the array is stopping if there is a stop event in the last 50 lines of the log file? The first time I ran this script to test (unraid_addon_control.sh -u -F) it was starting then stopping all the rc scripts every 30 seconds or so. Got really confusing with crashplan trying to check its files and cache_dirs didn't close properly when it was stopped right after being started. I know this script is new and still being worked on. Is it mainly there to just stop things properly before a shutdown (ie. not when the array is manually stopped then started again)?

Again, thanks for this script. I'm not complaining or anything, just trying to understand how it works and how I am supposed to use it.

Now is it correct that this script will always think the array is stopping if there is a stop event in the last 50 lines of the log file? The first time I ran this script to test (unraid_addon_control.sh -u -F) it was starting then stopping all the rc scripts every 30 seconds or so. Got really confusing with crashplan trying to check its files and cache_dirs didn't close properly when it was stopped right after being started. I know this script is new and still being worked on. Is it mainly there to just stop things properly before a shutdown (ie. not when the array is manually stopped then started again)?

This was something that was bothering me a bit. Maybe a new approach of using a program called logtail.

logtail reads a file, but remembers the last location it read. So the next time it is run. it will pickup from the last read position.

This might make it easier to weed out an old status.

It's small and I can compile it statically. Thoughts.

Now is it correct that this script will always think the array is stopping if there is a stop event in the last 50 lines of the log file? The first time I ran this script to test (unraid_addon_control.sh -u -F) it was starting then stopping all the rc scripts every 30 seconds or so. Got really confusing with crashplan trying to check its files and cache_dirs didn't close properly when it was stopped right after being started. I know this script is new and still being worked on. Is it mainly there to just stop things properly before a shutdown (ie. not when the array is manually stopped then started again)?

This was something that was bothering me a bit. Maybe a new approach of using a program called logtail.

logtail reads a file, but remembers the last location it read. So the next time it is run. it will pickup from the last read position.

 

This might make it easier to weed out an old status.

 

It's small and I can compile it statically. Thoughts.

Yes, there will always be issues for any script that is still starting when it is asked to stop.  It might not be listening yet, but as I said, I think I learned a bit from my earlier versions.  All in all, this script is just until we get actual events for pre-stop and post-start.

It runs the rc.* scripts once when in the pre-stop state, and once in the post-start state.  Other than that, it just sits and monitors the tail end of the syslog.

I haven't looked here for a while. Is this still the way to start stuff when the array is ready? I searched around the forums and found talk of 5.0 supporting this a different way but nothing definite.

I haven't looked here for a while. Is this still the way to start stuff when the array is ready? I searched around the forums and found talk of 5.0 supporting this a different way but nothing definite.

Ok, thanks. I'll fiddle around with this script a little and try to get it working for me. Before, I was not able to get it working properly. It looks like this script is made to only work if you do not use the user share (/mnt/user). It would sometimes start the crashplan service before /mnt/user was mounted in which case crashplan will create a new /mnt/user/crashplan directory, preventing the real /mnt/user from mounting properly.

Probably just need to change it to look for /mnt/user mounted instead of any /mnt/disk# and search the log for "Retry unmounting user share" instead of "Retry unmounting disk share" or whatever it is currently doing. Please correct me if I am wrong.

Ok, thanks. I'll fiddle around with this script a little and try to get it working for me. Before, I was not able to get it working properly. It looks like this script is made to only work if you do not use the user share (/mnt/user). It would sometimes start the crashplan service before /mnt/user was mounted in which case crashplan will create a new /mnt/user/crashplan directory, preventing the real /mnt/user from mounting properly.

 

Probably just need to change it to look for /mnt/user mounted instead of any /mnt/disk# and search the log for "Retry unmounting user share" instead of "Retry unmounting disk share" or whatever it is currently doing. Please correct me if I am wrong.

It is best for the script that starts crashplan to perform that test.  yes, you would want to wait for /mnt/user/crashplan to exist. 

(Obviously, I do not use crashplan here)

Just add a few lines like this to the top of the rc.crashplan script that starts crashplan

case $1 in

start)

# loop until /mnt/user/crashplan directory exists

  while true 

  do

    sleep 5

    if [ -d /mnt/user/crashplan ]; then break;

  done

  #commands to start crashplan go here

;;

stop)

  #commands to stop crachplan go here

;;

esac

Joe L.