Cache_dirs and S3 sleep.

Hi guys,

 

UnRAID server is up and running, small hiccups here and there but mainly because I'm using the lastest BETA so I can use AFP & Time Machine (which is working well, if not perfectly yet )

 

Two major issues at the moment:

 

Note: Using OS X and 5.0-beta6a and my command line is pretty limited.

 

1. Cache_dirs.

Setup using the guide on the wiki, there are 5 instances of cache_dirs in top (four data drives + cache, I assume).

 

However whenever I click on "Tower" in Finder all the disks spin up, which suggests it isn't working correctly.

 

How can I check cache_dirs is working correct/is there a bug?

 

2. Sleep

WOL works with my mobo and if I hit the button on the front it goes to sleep and wakes up again quite happily.

 

Now ideally i'd like it to go to sleep after a set amount of time (say an hour).

 

I've played around with the s3.sh script however it uses the /acpi/sleep command which the current kernal doesn't use.

 

I've looked through the forums to try and find the answer, but I've failed miserably.

 

What does "echo 3 > /proc/acpi/sleep" need to change to get the latest beta to sleep?

 

Thanks,

CR!

S3 - this is the only script that works fine for me. I've edited it slightly for me, but works well on 5b. I've changed the setup so it should sleep for you after about an hour.

 

in a folder on flash called S3.sh:

 

#!/bin/bash
# constants
yes="yes"
no="no"
DATE=$(date)

# [CONFIGURATION]

# before going to sleep
intrnlTimeoutTicks=2	# ticks after HDD spindown before checking for external activity
extrnlTimeoutTicks=30	# ticks of no external activity before sleep; only after spindown+internal countdown

# control of internal timeout
checkHDDs=$yes		# check if all HDDs are parked before counting down towards sleep
noCountdownHours=""	# only countdown towards sleep outside these hours
			# example: <noCountdownHours="07 08 19 20">
			# always countdown: <noCountdownHours="">

# control of external timeout 
checkTCP=$no		# check for TCP activity

pingIPs="$no" 		# do not sleep if <$pingsIPs> are pingable
			# example: <pingIPs="192.168.1.4 192.168.1.5">
			# no ping-check: <pingIPs="">

# after waking up from sleep
doDhcpRenewal=$no	# <$no> for servers w/static IP address
forceGb=$yes		# might not be needed; probably always safe

# [/CONFIGURATION]


# implementation stuff
ticklengthSecs=60		# probe hardware + count down every minute/60secs, aka a tick
noTCP='0.00'			# what constitutes absence of TCP activity
flash=/dev/`ls -l /dev/disk/by-label| grep UNRAID | cut -d"/" -f3 | cut -c 1-3` # automatic id of flash drive
check_hour() {
echo $(date +%H)
}
check_HDD_activity() {
if [ $checkHDDs = $yes ]
then
	# probe the flash drive at your peril
	HDDs=$((for d in $(ls /dev/[hs]d? | grep -v "$flash"); do hdparm -C $d | grep active ; done) | wc -l)
else
	HDDs=0
fi
echo $HDDs
}
check_TCP_activity() {
if [ "$checkTCP" = $yes ]
then 
	TCP=$(bwm-ng -o csv -c 1 -d 0 -T avg | grep eth0 | cut -d";" -f5)
else
	TCP="$noTCP"
fi
echo "$TCP"
}
check_IP_status() {
mp_online=$no  # initialize to "no" until we learn otherwise
       # ping each of the media servers to determine if online
if [ ! "$checkIPs" = $no ]; then
       for i in "$pingIPs"
       do
       	# ping the media server; if it answers, it is online
           	out=`ping -q -c 1 $i 2>/dev/null`
           	rec_count=`echo "$out" | grep received | cut -d " " -f4`
           	if [ "$rec_count" -eq 1 ]
           	then
               	mp_online=$yes
               	# if one is online, we do not need to ping
               	# any others, break out of the "for" loop.
               	break;
           	fi
       done
fi
       echo $mp_online
}
pre_sleep_activity() {

			echo "$DATE - Stopping services and sleeping"
			# Stop SMB
			/etc/rc.d/rc.samba stop 

			# Stop NFS
			/etc/rc.d/rc.nfsd stop

			# Stop AFP
			/etc/rc.d/rc.atalk stop

			# Stop AVAHI
			/etc/rc.d/rc.avahidaemon stop
			/etc/rc.d/rc.avahidnsconfd stop

			# Syncing Filesystems
			sync
}
post_sleep_activity() {

echo "$DATE - Starting services and waking"

# Force NIC to use gigabit networking
if [ "$forceGb" = $yes ]
then 
	ethtool -s eth0 speed 1000
fi

# Force a DHCP renewal (shouldn't be used for static-ip boxes)
if [ "$doDhcpRenewal" = $yes ]
then 
	/sbin/dhcpcd -n
fi

# Start SMB
/etc/rc.d/rc.samba start 

# Start NFS
/etc/rc.d/rc.nfsd start

# Start AFP
/etc/rc.d/rc.atalk start

# Start AVAHI
/etc/rc.d/rc.avahidaemon start
/etc/rc.d/rc.avahidnsconfd start

beep -f660 -l 100ms -D 100ms
beep -f660 -l 100ms -D 140ms
beep -f660 -l 120ms -D 150ms
beep -f510 -l 100ms -D 100ms
beep -f660 -l 120ms -D 160ms
beep -f770 -l 100ms -D 575ms
beep -f380 -l 150ms -D 575ms

}


# main
intrnlCountdown=$intrnlTimeoutTicks
extrnlCountdown=$extrnlTimeoutTicks
while [ 1 ]
do
# do not countdown during certain hours
	hour=`check_hour`
hourMatch=$(echo "$noCountdownHours" | grep "$hour" | wc -l)
if [ $hourMatch -eq 0 ]
then
	# count number of HDDs that are not parked
	HDDact=`check_HDD_activity`
 	if [ "$HDDact" -eq 0 ]
 	then
		# tick-tock for time since last spindown
		if [ $intrnlCountdown -gt 0 ]
  		then
  			intrnlCountdown=$[$intrnlCountdown-1]
		fi
 	else
		# reset countdown, following HDD activity
  		intrnlCountdown=$intrnlTimeoutTicks
 			extrnlCountdown=$extrnlTimeoutTicks
		echo "$DATE - Hard Drives active, resetting counter" | logger
 	fi

	if [ $intrnlCountdown -le 0 ]
 	then
		# check for persistent external activity
 		TCPact=`check_TCP_activity`
 		IPping=`check_IP_status`
 		if [ "$TCPact" = $noTCP -a "$IPping" = $no ]
		then
			if [ $extrnlCountdown -le 0 ]
 			then
	  		# Do pre-sleep activities
				pre_sleep_activity
	  			sleep 5
	  		
			# Go to sleep
	  			echo -n mem >/sys/power/state

	  		# Do post-sleep activities
				post_sleep_activity
	  			sleep 5
	  			intrnlCountdown=$intrnlTimeoutTicks
	  			extrnlCountdown=$extrnlTimeoutTicks
			else
				# tick-tock for persistent external activity
				if [ $extrnlCountdown -gt 0 ]
		  		then
		  			extrnlCountdown=$[$extrnlCountdown-1]
				fi
			fi
		else
			# reset countdown, following external activity
			extrnlCountdown=$extrnlTimeoutTicks
		fi
 	fi
fi

 # Wait a tick
	sleep $ticklengthSecs
done

 

In go script:

 

nohup /boot/path/to/s3.sh &

Excellent, thanks for that, it's a very comprehensive script.

 

I've tested the sleep command and that works and then it wakes up again after a WOL magic packet so all is good there.

 

What do I need to change so it goes to sleep X mins after the last HD goes to sleep?

 

Also I assume I just need to put my Mac Pro's IP in to the ping IP section: pingIPs="192.168.1.5"?

 

That would enable me to have a very short HD spindown to sleep timer as the MP is the main user of the array,.

Hi guys,

 

UnRAID server is up and running, small hiccups here and there but mainly because I'm using the lastest BETA so I can use AFP & Time Machine (which is working well, if not perfectly yet )

 

Two major issues at the moment:

 

Note: Using OS X and 5.0-beta6a and my command line is pretty limited.

 

1. Cache_dirs.

Setup using the guide on the wiki, there are 5 instances of cache_dirs in top (four data drives + cache, I assume).

 

However whenever I click on "Tower" in Finder all the disks spin up, which suggests it isn't working correctly.

 

How can I check cache_dirs is working correct/is there a bug?

No, no bug.  I suspect your "finder" is accessing files on the disks and not just listing the directories.    To see the actual files being accessed, install inotifywatch and see exactly what is being accessed.  I think you'll find a lot more than just directory listings going on.

 

Joe L.

Play with these two lines. You can skip IP activity if that suits, I don't use it.

 

 

	intrnlTimeoutTicks=2	# ticks after HDD spindown before checking for external activity
extrnlTimeoutTicks=30	# ticks of no external activity before sleep; only after spindown+internal countdown

Play with these two lines. You can skip IP activity if that suits, I don't use it.

 

 

	intrnlTimeoutTicks=2	# ticks after HDD spindown before checking for external activity
extrnlTimeoutTicks=30	# ticks of no external activity before sleep; only after spindown+internal countdown

 

Thanks.

 

Do I need to restart the server to get the script to kick in or does it read it every so often? [Just wondering as I will tinker with it until im happy!]

 

Hi guys,

 

UnRAID server is up and running, small hiccups here and there but mainly because I'm using the lastest BETA so I can use AFP & Time Machine (which is working well, if not perfectly yet )

 

Two major issues at the moment:

 

Note: Using OS X and 5.0-beta6a and my command line is pretty limited.

 

1. Cache_dirs.

Setup using the guide on the wiki, there are 5 instances of cache_dirs in top (four data drives + cache, I assume).

 

However whenever I click on "Tower" in Finder all the disks spin up, which suggests it isn't working correctly.

 

How can I check cache_dirs is working correct/is there a bug?

No, no bug.  I suspect your "finder" is accessing files on the disks and not just listing the directories.    To see the actual files being accessed, install inotifywatch and see exactly what is being accessed.  I think you'll find a lot more than just directory listings going on.

 

Joe L.

 

Bummer.

 

That's rather annoying, is there any [known] way round it?

 

Really no point in having a cache disk spinning all the time otherwise

 

Two new things:

 

1. Turning off the cache disk results in write speeds of... 250Kb/sec with bursts of ~5MB/sec. [Note: This is disabling cache disk for user shares and then not restarting the array/PC]

2. Cache disk enabled transfer rates jump from 80+MB/sec down to 20MB/sec for 10 seconds and then back up to 80MB/sec for 5 seconds, etc, what is going on here?

Direct to drive transfers can hit 80MB/sec+ constant.

 

Otherwise, cache disk transfers every two hours on the hour and time machine seems to be working.

 

Thanks,

CR!

Unsure! Try putting it to the minimum ticks to see if it works first. If not, restarts it is!

 

I've got a great setup with my hackintosh media center. The unRAID server & media center will go to sleep if there's no activity from 10pm onwards. My media center starts at 7:30am to host my blogs etc. Then the media center wakes unRAID server at 5 for when I come home from work. Media center mounts all the volumes and opens iTunes for me whenever unRAID is awake.

 

Loves it 

 

Script for your Mac below, to control wake and volume mounting:

 

http://lime-technology.com/forum/index.php?topic=10274.0

You should be getting a pretty solid 30MB/s without Cache. TimeMachine does backup slowly however, but thats because it's transferring many small files.

 

I'd be more inclined to enable TimeMachine on a share without cache. Sparsebundles tend to corrupt easily if the mover script doesn't do the job perfectly. I've got TM sitting on a lone drive right now, haven't had the time or courage to span across two disks yet.

 

Just think of a situation if your MacPro hard drive dies, and the latest backup is still sitting on Cache drive. The potential for something un-toward to happen is high.

Unsure! Try putting it to the minimum ticks to see if it works first. If not, restarts it is!

 

I've got a great setup with my hackintosh media center. The unRAID server & media center will go to sleep if there's no activity from 10pm onwards. My media center starts at 7:30am to host my blogs etc. Then the media center wakes unRAID server at 5 for when I come home from work. Media center mounts all the volumes and opens iTunes for me whenever unRAID is awake.

 

Loves it 

 

Script for your Mac below, to control wake and volume mounting:

 

http://lime-technology.com/forum/index.php?topic=10274.0

 

Thanks.

 

Does this script work when I wake the Mac Pro?

 

Ideally I'd like the following:

 

1. Mac runs a WOL script (I already have an applescript that works, just no easyish way of running it when it wakes).

Mac is NEVER shutdown, so I never login!

 

Any ideas for this one?

 

2. Server running whenever the Mac Pro is running (the ping command in the S3 script gives me this).

 

I'd like the Time Machine disk to be spinning as long as the Mac Pro is running (it's only ever accessed regularly by that).

 

Setting the spin down time to never would help, but then the script would never let it sleep because it checks all disks are asleep.

 

Can I tell it to ignore a disk/share when checking if the disks are spun down? (In this case disk 1...)

 

Then two mins after the MP shuts down, the server will shut down (assuming none of the other disks are spinning).

Combination of 1 min for HD shutdown check + 1 min external shutdown check.

 

 

3. When it goes to sleep, it wakes all the disks up, is that a UnRAID thing or a script thing? From a component and HD life PoV this isn't great!

Any solutions?

 

 

 

 

You should be getting a pretty solid 30MB/s without Cache. TimeMachine does backup slowly however, but thats because it's transferring many small files.

 

I'd be more inclined to enable TimeMachine on a share without cache. Sparsebundles tend to corrupt easily if the mover script doesn't do the job perfectly. I've got TM sitting on a lone drive right now, haven't had the time or courage to span across two disks yet.

 

Just think of a situation if your MacPro hard drive dies, and the latest backup is still sitting on Cache drive. The potential for something un-toward to happen is high.

 

I've turned the cache disk off for time machine now, I assume mover will move the last backup over still?

 

I won't ever have the need to span TM over two disks, 3TB drives will be used before that day arrives!

 

Cheers

CR

1. Mac runs a WOL script (I already have an applescript that works, just no easyish way of running it when it wakes).

Mac is NEVER shutdown, so I never login!

 

2. Server running whenever the Mac Pro is running (the ping command in the S3 script gives me this).

 

 

Essentially what you want to do is have my mount/WOL script hours changed to be 24 hours. The script runs perpetually, so when the MacPro is awake then the server will be woken up. This way you can forgo your Applescript. Then change the unRAID server to sleep whenever there's no ping activity on the MacPro, and the hard drives are spun down.

 

Don't worry about setting the TimeMachine disk to be spinning all the time. It only takes 5 seconds for it to be active again.

 

3. When it goes to sleep, it wakes all the disks up, is that a UnRAID thing or a script thing? From a component and HD life PoV this isn't great!

 

This is normal, unsure if it's unRAID, but not a script issue. My script additions turns off the array and syncs the filesystem before unRAID sleeps, which is the safest way before sleeping. Don't worry about it - your hard drives will continue to work just fine. Some of mine have gone through tens of thousands of spinup cycles without any issues.

 

Version: 5b10

 

Code:

 

#!/bin/bash
# constants
yes="yes"
no="no"
DATE=$(date)

# [CONFIGURATION]

# before going to sleep
intrnlTimeoutTicks=1	# ticks after HDD spindown before checking for external activity
extrnlTimeoutTicks=1	# ticks of no external activity before sleep; only after spindown+internal countdown

# control of internal timeout
checkHDDs=$yes		# check if all HDDs are parked before counting down towards sleep
noCountdownHours=""	# only countdown towards sleep outside these hours
# example: <noCountdownHours="07 08 19 20">
# always countdown: <noCountdownHours="">

# control of external timeout 
checkTCP=$no		# check for TCP activity

pingIPs="192.168.1.5 192.168.1.30 192.168.1.31" 		# do not sleep if <$pingsIPs> are pingable
# example: <pingIPs="192.168.1.4 192.168.1.5">
# no ping-check: <pingIPs="">

# after waking up from sleep
doDhcpRenewal=$yes	# <$no> for servers w/static IP address
forceGb=$yes		# might not be needed; probably always safe

# [/CONFIGURATION]


# implementation stuff
ticklengthSecs=60		# probe hardware + count down every minute/60secs, aka a tick
noTCP='0.00'			# what constitutes absence of TCP activity
flash=/dev/`ls -l /dev/disk/by-label| grep UNRAID | cut -d"/" -f3 | cut -c 1-3` # automatic id of flash drive
check_hour() {
echo $(date +%H)
}
check_HDD_activity() {
if [ $checkHDDs = $yes ]
then
# probe the flash drive at your peril
HDDs=$((for d in $(ls /dev/[hs]d? | grep -v "$flash"); do hdparm -C $d | grep active ; done) | wc -l)
else
HDDs=0
fi
echo $HDDs
}
check_TCP_activity() {
if [ "$checkTCP" = $yes ]
then 
TCP=$(bwm-ng -o csv -c 1 -d 0 -T avg | grep eth0 | cut -d";" -f5)
else
TCP="$noTCP"
fi
echo "$TCP"
}
check_IP_status() {
mp_online=$no  # initialize to "no" until we learn otherwise
# ping each of the media servers to determine if online
if [ ! "$checkIPs" = $no ]; then
for i in "$pingIPs"
do
# ping the media server; if it answers, it is online
out=`ping -q -c 1 $i 2>/dev/null`
rec_count=`echo "$out" | grep received | cut -d " " -f4`
if [ "$rec_count" -eq 1 ]
then
mp_online=$yes
# if one is online, we do not need to ping
# any others, break out of the "for" loop.
break;
fi
done
fi
echo $mp_online
}
pre_sleep_activity() {

echo "$DATE - Stopping services and sleeping"
# Stop SMB
/etc/rc.d/rc.samba stop 

# Stop NFS
/etc/rc.d/rc.nfsd stop

# Stop AFP
/etc/rc.d/rc.atalk stop

# Stop AVAHI
/etc/rc.d/rc.avahidaemon stop
/etc/rc.d/rc.avahidnsconfd stop

# Syncing Filesystems
sync
}
post_sleep_activity() {

echo "$DATE - Starting services and waking"

# Force NIC to use gigabit networking
if [ "$forceGb" = $yes ]
then 
ethtool -s eth0 speed 1000
fi

# Force a DHCP renewal (shouldn't be used for static-ip boxes)
if [ "$doDhcpRenewal" = $yes ]
then 
/sbin/dhcpcd -n
fi

# Start SMB
/etc/rc.d/rc.samba start 

# Start NFS
/etc/rc.d/rc.nfsd start

# Start AFP
/etc/rc.d/rc.atalk start

# Start AVAHI
/etc/rc.d/rc.avahidaemon start
/etc/rc.d/rc.avahidnsconfd start

beep -f 400 -l 100 -D 100
beep -f 800 -l 100 -D 100
beep -f 1200 -l 100 -D 100
beep -f 1200 -l 100 -D 100

}


# main
intrnlCountdown=$intrnlTimeoutTicks
extrnlCountdown=$extrnlTimeoutTicks
while [ 1 ]
do
# do not countdown during certain hours
hour=`check_hour`
hourMatch=$(echo "$noCountdownHours" | grep "$hour" | wc -l)
if [ $hourMatch -eq 0 ]
then
# count number of HDDs that are not parked
HDDact=`check_HDD_activity`
if [ "$HDDact" -eq 0 ]
then
# tick-tock for time since last spindown
if [ $intrnlCountdown -gt 0 ]
then
intrnlCountdown=$[$intrnlCountdown-1]
fi
else
# reset countdown, following HDD activity
intrnlCountdown=$intrnlTimeoutTicks
extrnlCountdown=$extrnlTimeoutTicks
echo "$DATE - Hard Drives active, resetting counter" | logger
fi

if [ $intrnlCountdown -le 0 ]
then
# check for persistent external activity
TCPact=`check_TCP_activity`
IPping=`check_IP_status`
if [ "$TCPact" = $noTCP -a "$IPping" = $no ]
then
if [ $extrnlCountdown -le 0 ]
then
# Do pre-sleep activities
pre_sleep_activity
sleep 5

# Go to sleep
echo -n mem >/sys/power/state

# Do post-sleep activities
post_sleep_activity
sleep 5
intrnlCountdown=$intrnlTimeoutTicks
extrnlCountdown=$extrnlTimeoutTicks
else
# tick-tock for persistent external activity
if [ $extrnlCountdown -gt 0 ]
then
extrnlCountdown=$[$extrnlCountdown-1]
fi
fi
else
# reset countdown, following external activity
extrnlCountdown=$extrnlTimeoutTicks
fi
fi
fi

# Wait a tick
sleep $ticklengthSecs
done

 

Does everything look ok here?

 

Bah.

 

Went to sleep after 2 minutes again.

 

Something isn't right

 

Mac Pro (192.168.1.5) is running.

 

It's like it isn't pinging anymore!?

checkTCP=$no # check for TCP activity

 

Change that to

 

checkTCP=$yes # check for TCP activity

Is that what is stopping the ping thing from working now?

 

It was working perfectly up until that point!

I thought someone else could find this useful: 

 

I altered the beep codes in the sleep script to play the first notes of the Super Mario Theme when the server wakes up:

 

  
  beep -f659.25 -l 150ms -D 5ms
  beep -f659.25 -l 180ms -D 80ms
  beep -f659.25 -l 180ms -D 80ms
  beep -f523.25 -l 150ms -D 5ms
  beep -f659.25 -l 180ms -D 80ms
  beep -f783.99 -l 180ms -D 400ms
  beep -f392 -l 180ms -D 80ms

I thought someone else could find this useful: 

 

I altered the beep codes in the sleep script to play the first notes of the Super Mario Theme when the server wakes up:

 

  
  beep -f659.25 -l 150ms -D 5ms
  beep -f659.25 -l 180ms -D 80ms
  beep -f659.25 -l 180ms -D 80ms
  beep -f523.25 -l 150ms -D 5ms
  beep -f659.25 -l 180ms -D 80ms
  beep -f783.99 -l 180ms -D 400ms
  beep -f392 -l 180ms -D 80ms

Bookmarked this!