cache_dirs - an attempt to keep directory entries in RAM to prevent disk spin-up

[starcat]

I am starting cache_dirs from my go script with the following command, basically delaying startup until all directories and mount points are created.

echo "/boot/scripts/cache_dirs -w -B -u -e /mnt/disk*/Data -e /mnt/user/Data" | at now + 10 minute

However I am still getting following error, am I doing something wrong?

ERROR: excluded directory "/mnt/disk*/Data" does not exist.
ERROR: excluded directory "/mnt/user/Data" does not exist.
cache_dirs process ID 16607 started, To terminate it, type: cache_dirs -q

[Joe L.]

I am starting cache_dirs from my go script with the following command, basically delaying startup until all directories and mount points are created.

echo "/boot/scripts/cache_dirs -w -B -u -e /mnt/disk*/Data -e /mnt/user/Data" | at now + 10 minute

However I am still getting following error, am I doing something wrong?

ERROR: excluded directory "/mnt/disk*/Data" does not exist.
ERROR: excluded directory "/mnt/user/Data" does not exist.
cache_dirs process ID 16607 started, To terminate it, type: cache_dirs -q

You can only exclude top level directories using the "-e" option.

You can use the -a option though and add a "prune" command to the "find" being invoked, and it would look like this:
cache_dirs  -w -B -u -a '-noleaf -name Data -prune -o -print'

It was discussed a few post ago.
http://lime-technology.com/forum/index.php?topic=4500.msg61520#msg61520

Looking closer at your example, Data is a top level directory.  You only need give the share name, not the full path to it.

Try this:
echo "/boot/scripts/cache_dirs -w -B -u -e Data" | at now + 10 minute

also verify the folder does exist with the exact same capitilization as in the directory.
type
ls -l /mnt/disk*/Data

if it does not exist, then perhaps the directory name is lower case and only showing mixed case in Windows when viewed through SAMBA.

Joe L.

[rsd]

Hi Joe,

I could greatly use some help with cache dirs. I recently upgraded my mobo to a biostar 760+ and upgraded to unraid 4.54.

I call cache_dirs in my go script using: /boot/cache_dirs -d 5 -w -e "VMC"

Everything was working for a few weeks but now twice in the past 3 days I got some strange error that hard froze unraid (can't telnet to it, can't refresh the unmenu/web, when I manually hook up a monitor to it no commands are acknolwedged, can't see it on the network etc). It doesn't freeze after a reboot or anything but some random time afterwards (hours/days).

This is what the last few syslog lines read from Unmenu:

Jun 15 05:51:33 Tower kernel: [] ? sys_read+0x3b/0x60
Jun 15 05:51:33 Tower kernel: [] ? syscall_call+0x7/0xb
Jun 15 05:51:33 Tower kernel: Code: 55 f0 e8 8a 82 d5 c8 8d 50 01 29 f2 39 d3 7c 0b 8b 45 0c 89 d3 c7 00 01 00 00 00 8b 45 f0 89 d9 81 c6 a0 3d 3e f8 c1 e9 02 89 38 a5 89 d9 83 e1 03 74 02 f3 a4 5a 89 d8 5b 5e 5f 5d c3 55 89
Jun 15 05:51:33 Tower kernel: EIP: [] md_cmd_proc_read+0x41/0x54 [md_mod] SS:ESP 0068:c238bef4
Jun 15 05:51:33 Tower kernel: CR2: 00000000f83e8000
Jun 15 05:51:33 Tower kernel: ---[ end trace 6baff4c42c121ab7 ]--

I see some different error messages when I go to the unraid box itself, a lot of stuff but this is the part that makes me think it is something cache_dirs related:

/boot/cache_dirs/: Line 479: 30756 killed
$command "$i"/$share_dir" $depth > /dev/null 2>&1

I know my cache_dirs call is slightly modified from before (I didn't use to exclude the "VMC" folder), so I'll try just calling it with -w and see what happens.
Any help is greatly appreciated!

[Joe L.]

Looks more like you are running out of memory. 

cache_dirs attempts to set a server kernel variable to be more aggressive in keeping directory entries in cache memory.   
As part of its script, it invokes
sysctl vm.vfs_cache_pressure=10

You can override this with the "-p" option to cache_dirs.  As delivered by lime-technology the value of 100 is used.

So, you can try adding
-p 50
or
-p 100

or anything higher than 10.

When memory is running out Linux will terminate processes that are idle in order to attempt to free memory for programs requesting it.  Typically, the web-management interface is one of those idle, so it gets killed and you lose communications.  In the same was SAMBA also gets killed off.

You can re-start emhttp with
killall emhttp
/usr/local/sbin/emhttp &

You can re-start samba with
/root/samba restart

Usually this will allow you to re-gain communications with the server so you can cleanly shut it down and re-start it.

[rsd]

Thanks Joe, you're the best!

[NAS]

That exlains alot.

cat /proc/sys/vm/vfs_cache_pressure
10

is overriding my value of 0 in in go script.

I was scratching my head why disk were spinning up more. Now we have an answer.

[Joe L.]

That exlains alot.

cat /proc/sys/vm/vfs_cache_pressure
10

is overriding my value of 0 in in go script.

I was scratching my head why disk were spinning up more. Now we have an answer.

I've run out of memory with cache_pressure set to "0" on my server with 512Meg of RAM.  It is for that reason I set it to 10.   (and it is still possible to run out of RAM if I try hard enough)

Default value on Linux is 100.

Just use
-p 0
to keep your alternative value from being changed

[NAS]

Way ahead of you.

Purely based on casual observations setting 0 REALLY makes a difference for me.

Wouldn't it be nice if the kernel would allow you to reserve a portion of memory dedicated to this cache then swap based on cache pressure and all the other magic if more than this was needed.

Oh to know a kernel developer/

[Joe L.]

Way ahead of you.

Purely based on casual observations setting 0 REALLY makes a difference for me.

Wouldn't it be nice if the kernel would allow you to reserve a portion of memory dedicated to this cache then swap based on cache pressure and all the other magic if more than this was needed.

Oh to know a kernel developer/

how much RAM do you have in your server?  I know I've run out even with a swap file defined with cache pressure of 10.

[NAS]

2GB i believe.

I havent really looked at ram usage but from a user POV everything seems to just work

[lionelhutz]

OK, I just installed it and put cache_dirs -w in my go script and drives were still spinning up the same as before.

So, I stopped it and then ran it at the prompt like - cache_dirs -F -v
The below is the result;

Executing find /mnt/disk2/Car_Music -noleaf
Executing find /mnt/disk1/Movies -noleaf
Executing find /mnt/disk2/Music -noleaf
Executing find /mnt/disk1/New_Movies -noleaf
Executing find /mnt/disk2/New_Movies -noleaf
Executing find /mnt/disk2/Pictures -noleaf
Executing find /mnt/disk2/Storage -noleaf
Executing find /mnt/disk2/TV_Shows -noleaf
Executing find /mnt/disk3/TV_Shows -noleaf
Executed find in 0.287845 seconds, weighted avg=0.289340 seconds, now sleeping 6 seconds

So, it appears to be going through all the directories and it did not spin up any drives the first pass when run on the comand line so it also appears the version running from the go script was working. I left it last night and the drvies all spun down sometime early this morning and stayed spun down while I stopped it and ran it from the command line.

While it was running as above, the drives still spun up if I tried to navigate to almost any directory on them. I could go to the top level of the shares but that was about it. As I understand it, it will scan to a default maxium directory depth of 99 and I don't have anything beyond about 4 or 5 levels.

I am using W7 Windows Explorer. I specifically set the default view to "List" which should not try to gather any specific info about the file or create a thumbnail of the file. I also turned off the "Details" pane and the "Preview" pane as well.  Actually, the preview was off already.

I then tried the -M 1 option and it seems a little better but still spins up disks.also I also tried -p 1 option but not much change. However, after both of these options it seemed to me there was a higher chance the disk would spin up when I pressed the back button while surfing directories. But, I still didn't notice much if any pattern to it.

So, anyone have any ideas? Is this a Windows Explorer thing? Anyone else know if W7 Windows Explorer will still read file info no matter what.

Peter

[Eisi2005]

Hi,

i start cache_dir with -w -B. All works good, but one of my datadisk and my parity hdd spin up after some hours. Did cache_dirs write anythin on my share or my disk when i start it with -w -B, and if yes how could i turn off this feature ?

Greets
Eisi

[Joe L.]

Hi,

i start cache_dir with -w -B. All works good, but one of my datadisk and my parity hdd spin up after some hours. Did cache_dirs write anythin on my share or my disk when i start it with -w -B, and if yes how could i turn off this feature ?

Greets
Eisi

cache_dirs writes nothing to the disks, your PC might be scanning for new files,   If parity spun up, then one of your disks was written to.  Perhaps it is making thumbnail images.

Joe L.

[chanders]

Have been using this for a while and I love it. Fast as hell.

Quick question: If I add a new folder to an existing share, will this acknowledge the newly added folder?

[Joe L.]

Have been using this for a while and I love it. Fast as hell.

Quick question: If I add a new folder to an existing share, will this acknowledge the newly added folder?

yes.