System performance / graphs

I have Sysstat (sar) installed and working on unRAID to collect system performance statistics, and a web interface in php to graph them.

 

 

Anyone else interested in this type of stuff?  If so, I'll try to clean it up and get it into BubbaRaid... otherwise I'll keep it on the back burner as a tool for me in its current "quick and dirty" glory.

Interesting...  Would this have some diagnostic potential, for example aligning those red spikes with stutter?  What is the real difference between Idle and IO Wait?  What actually was happening between roughly 11:05 and 11:08?

That graph captures the time when a 4.5GB ISO file was copied to the unRAID server.

 

Diagnostic potential is indeed the expected use.

 

Sysstat give you data for NIC, drives, and a host of other parameters on the server.

 

I usually run two collection routines... one at 10 minute intervals, and retain the data for 35 days. and another at 30 second intervals, and keep it for 2 days.  To get the realtime graph in the OP, I launched a hi-res collection that runs at 5 second intervals.  That gives 120 data points for a 10 minute graph, which is what you see above.  That's a fairly good compromise resolution, without the data collection and graphing work itself causing a spike in CPU.

 

 

If we have PHP installed is there any way to incorporate this into unMenu?

If we have PHP installed is there any way to incorporate this into unMenu?

 

Not unless you find a way to serve up large images with awk.

 

And to answer Rob J's question, IO Wait is time the CPU was idle because it was waiting for IO to finish (usually disk IO) Higher IO Wait points to your disk IO as a bottleneck.

If we have PHP installed is there any way to incorporate this into unMenu?

What do you want to invoke uising "php" ?

 

An un-menu plug-in can be written in any scriptable language, php included.

 

If a "php" file will provide a web-page, the unmenu can send it onward to your browser.

 

Or... if you like, you can invoke the "php" as a button on the "User-Scripts" plug-in.

 

I did exactly that to invoke the smart logging "php" script.  My "User-Script file" is named 50-unmenu_user_script_smarthist-graph

 

root@Tower:/boot/unmenu_final# cat 50-unmenu_user_script_smarthist-graph

#define USER_SCRIPT_LABEL SMART History Report

#define USER_SCRIPT_DESCR Run SMART parameter tracking database Report

cd /boot/smarthistory

php smarthistory.php -wake ON -output HTML -graph IMAGE -report ALL

 

If you wanted this as a menu choice, it would look like this:

cat 50-unmenu-smart_history.cgi

#ADD_ON_MENU Smart History

#ADD_ON_URL smart_history

cd /boot/smarthistory

php smarthistory.php -wake ON -output HTML -graph IMAGE -report ALL

 

As long as the rendered html does not include any images from the unmenu server, you will be fine.  The php code that performs Smart history graphing gets its images from Google, so it works perfectly.

 

I am assuming that this System performance graph might also be created by google-graphs, so it too is easily possible from in unmenu using "php"

 

Joe L.

Unfortunately, Google Chart will not presently work for the sysstat graphs.  The PHP code creates a png file on the server, which is used in the BubbaRaid web server (lighttpd).

 

I am going to split lighttpd out of BubbaRaid, and make it an independent add-in with a set-up wizard.  That way others can have a lightweight web server for developing in w/o having to use a full BubbaRaid package.

I have Sysstat (sar) installed and working on unRAID to collect system performance statistics, and a web interface in php to graph them.

 

 

Anyone else interested in this type of stuff?  If so, I'll try to clean it up and get it into BubbaRaid... otherwise I'll keep it on the back burner as a tool for me in its current "quick and dirty" glory.

 

Yes - extremely interested and very useful.

 

You may (if you havn't already) want to have a look at munin.

 

It will graph similar metrics (and / or much more and user plugins are easy to create) and will remove the reliance on php (output is static html). However it does need perl, rrdtool and a bunch of perl modules so may not be much easier!!

 

http://munin.projects.linpro.no/

I originally tried rrdtool and sar2rrd, and the perl scripts.... HORRIBLY slow and cumbersome.  This is a very lightweight php library, even generating the graphics every 5 seconds there is practically no cpu impact, even with the sar overhead.

 

php  == good

 

perl  != good

I originally tried rrdtool and sar2rrd, and the perl scripts.... HORRIBLY slow and cumbersome.  This is a very lightweight php library, even generating the graphics every 5 seconds there is practically no cpu impact, even with the sar overhead.

 

php  == good

 

perl  != good

 

Sorry should have made clear - munin uses its own plugins to get the data, totally unconnected to sar or sar2rrd. Although it is a big bunch of perl to generated the rrd's then the individual plugins can be in whatever language the author desired (usually shell script but sometimes python, perl or even php).

 

Worth a look - probably not as lightweight as you are currently but will get you more metrics and is also easily extensible for custom unraid interests.

 

I'll leave the perl v php debate alone

 

 

The problem when trying to plot high res data on CPU usage, you can spike the graph you are generating with the CPU used to capture the data and graph it.

The problem when trying to plot high res data on CPU usage, you can spike the graph you are generating with the CPU used to capture the data and graph it.

 

rrdtool uses rolling averages - so spikes are smoothed out. It takes a sustained period of cpu usage to build an actual spike on the graph.

 

Your point is valid as it does of course show more cpu load than should be - but most systems won't show it / the additional load is just smoothed out over time as opposed to the true load which makes up the bulk of the graphed data.

 

Sorry - I didn't want to get into an argument! Whatever you come up with will be extremely useful to have - I just thought I would throw munin out there to save you re-inventing the wheel in some places if possible.

 

 

It's not an argument... just brainstorming.

 

Averaging is OK for some things, and not for others.... it can be particularly inappropriate when you are looking for transients that are interrupting real-time streaming of media (which is a primary use of unRAID).

It's not an argument... just brainstorming.

 

phew

 

Averaging is OK for some things, and not for others.... it can be particularly inappropriate when you are looking for transients that are interrupting real-time streaming of media (which is a primary use of unRAID).

 

If you see this as an on the spot 'somethings going on right now, what's the problem?' then I understand - your more granular approach will be vastly more informative.

 

For long term trends etc then you might have to rrd (or similar) at some point just to trim the dataset - depending on the amount of data generated for the graph in your example.

 

Of course perhaps it could just be a 'enable/disable this detailed logging and graphing' option for when you need to gather information. (akin to Suns analytics? http://ctistrategy.com/2008/12/17/sun-storage-7000-analytics-overview/ )

 

Maybe there's room for both.

 

The more stats and information about whats going on at the low(ish) level the better.

Anyone else interested in this type of stuff?  If so, I'll try to clean it up and get it into BubbaRaid... otherwise I'll keep it on the back burner as a tool for me in its current "quick and dirty" glory.

 

+1

Very interested!

Anyone else interested in this type of stuff?  If so, I'll try to clean it up and get it into BubbaRaid... otherwise I'll keep it on the back burner as a tool for me in its current "quick and dirty" glory.

 

+1

Very interested!

 

+2

If it keeps track of those stats and you can compare bandwidth coming through the Network cards versus writing on the SATA drives, that is awesome.

I do this manually now. It would be a good tool to benchmark an unraid server build / certify certain motherboards, controllers, drives. Throw in there drive temperatures. Then all you need is some kind of script which would be the same for everyone to run. In the end, we would see which setup has the best bang for money. All this to say I would like to see it in there, with ssh!

If it keeps track of those stats and you can compare bandwidth coming through the Network cards versus writing on the SATA drives, that is awesome.

I do this manually now. It would be a good tool to benchmark an unraid server build / certify certain motherboards, controllers, drives. Throw in there drive temperatures. Then all you need is some kind of script which would be the same for everyone to run. In the end, we would see which setup has the best bang for money. All this to say I would like to see it in there, with ssh!

You can see I/O and network statistics in the myMain plug-in of unMENU.  It does not show history, but rather as it is occurring and the page is refreshed.  You will need to install the "bwm-ng - Bandwidth Monitor NG (Next Generation)" package, but the package manager in unMENU has a config file that allows you to install that package.

 

Joe L.