cache_dirs - an attempt to keep directory entries in RAM to prevent disk spin-up

The only thing relating to cache dirs is my go file which is:

/boot/cache_dirs -w -d 4

 

You are running Dynamix, check its settings, perhaps it is set to run CacheDirs also?

No its not. Im going to remove cache dirs and the line from the go file and install it from dynamix, maybe that will work better. Thanks

I found this post to joeL a while ago:

 

--- End quote ---

The cache_dirs program will re-schedule itself when it finds no mounted disks. (Or thinks there are no mounted disks)

 

Why would I have or it think there are no mounted disks?

 

I wonder if this is related to the error I see in syslog when cache_dirs starts up on boot:

 

Sep 10 12:02:00 Tower cache_dirs: ==============================================
Sep 10 12:02:00 Tower cache_dirs: command-args=-w -m 1 -M 10 -d 6 -p 10 -U 50000 -i /mnt/ -B -a -noleaf
Sep 10 12:02:00 Tower cache_dirs: vfs_cache_pressure=10
Sep 10 12:02:00 Tower cache_dirs: max_seconds=10, min_seconds=1
Sep 10 12:02:00 Tower cache_dirs: max_depth=6
Sep 10 12:02:00 Tower cache_dirs: command=find -noleaf
Sep 10 12:02:00 Tower cache_dirs: version=1.6.9
Sep 10 12:02:00 Tower cache_dirs: ---------- caching directories ---------------
Sep 10 12:02:00 Tower cache_dirs: 
Sep 10 12:02:00 Tower cache_dirs: ----------------------------------------------
Sep 10 12:02:00 Tower cache_dirs: ERROR: included directory "/mnt/" does not exist.
Sep 10 12:02:00 Tower cache_dirs: cache_dirs process ID 5533 started, To terminate it, type: cache_dirs -q

 

I *think* the array is started by that time.  From all external evidence the /mnt/ volumes are being cached.

 

D

I wonder if this is related to the error I see in syslog when cache_dirs starts up on boot:

 

Sep 10 12:02:00 Tower cache_dirs: ==============================================
Sep 10 12:02:00 Tower cache_dirs: command-args=-w -m 1 -M 10 -d 6 -p 10 -U 50000 -i /mnt/ -B -a -noleaf
Sep 10 12:02:00 Tower cache_dirs: vfs_cache_pressure=10
Sep 10 12:02:00 Tower cache_dirs: max_seconds=10, min_seconds=1
Sep 10 12:02:00 Tower cache_dirs: max_depth=6
Sep 10 12:02:00 Tower cache_dirs: command=find -noleaf
Sep 10 12:02:00 Tower cache_dirs: version=1.6.9
Sep 10 12:02:00 Tower cache_dirs: ---------- caching directories ---------------
Sep 10 12:02:00 Tower cache_dirs: 
Sep 10 12:02:00 Tower cache_dirs: ----------------------------------------------
Sep 10 12:02:00 Tower cache_dirs: ERROR: included directory "/mnt/" does not exist.
Sep 10 12:02:00 Tower cache_dirs: cache_dirs process ID 5533 started, To terminate it, type: cache_dirs -q

 

I *think* the array is started by that time.  From all external evidence the /mnt/ volumes are being cached.

 

Using /mnt/ cannot work, because includes must be top level folders on any of the data drives or the cache drive.  The test is something like checking for the existence of /mnt/disk1//mnt/, which could not exist (note the double slash).

Using /mnt/ cannot work, because includes must be top level folders on any of the data drives or the cache drive.  The test is something like checking for the existence of /mnt/disk1//mnt/, which could not exist (note the double slash).

 

Thanks!  Somehow that was not clear to me.  For other interested parties, what worked is this command in /boot/config/go:

setsid /boot/config/plugins/cache_dirs -w -d 6 -i Media -i Misc -i Comedy -i Music\ Videos 

resulting in this log report:

Sep 10 15:03:00 Tower cache_dirs: ==============================================
Sep 10 15:03:00 Tower cache_dirs: command-args=-w -m 1 -M 10 -d 6 -p 10 -U 50000 -i Media -i Misc -i Comedy -i Music Videos -B -a -noleaf
Sep 10 15:03:00 Tower cache_dirs: vfs_cache_pressure=10
Sep 10 15:03:00 Tower cache_dirs: max_seconds=10, min_seconds=1
Sep 10 15:03:00 Tower cache_dirs: max_depth=6
Sep 10 15:03:00 Tower cache_dirs: command=find -noleaf
Sep 10 15:03:00 Tower cache_dirs: version=1.6.9
Sep 10 15:03:00 Tower cache_dirs: ---------- caching directories ---------------
Sep 10 15:03:00 Tower cache_dirs: Comedy
Sep 10 15:03:00 Tower cache_dirs: Media
Sep 10 15:03:00 Tower cache_dirs: Misc
Sep 10 15:03:00 Tower cache_dirs: Music Videos
Sep 10 15:03:00 Tower cache_dirs: ----------------------------------------------
Sep 10 15:03:00 Tower cache_dirs: cache_dirs process ID 5599 started, To terminate it, type: cache_dirs -q

Okay so I removed cache_dirs and the go file and then installed it from the Dynamix plugins page. I have some errors coming up. All I have done is set the depth to 4 folders deep in the settings. Bonienl sent this message so I really need joeL to look at it. I was quite happy running cache Dirs and the go file but was having problems so my thought was to remove it and run it from Dynamix control panel but Im still having errors. It looks like my original errors were to copying the go file command another user posted as I only wanted to scan 4 folders deep. This is the message Bonienl sent me on the Dynamix forum page and the image of my errors is posted as well:

 

The latest version of "Dynamix Cache Dirs" downloads the cache_dirs script made available here on the forum. The file is unzipped and placed in the /usr/local/sbin folder.

 

This approach ensures that always the latest available version of cache_dirs is used. Dynamix builds a GUI wrapper around it, which allows the user to enter parameters using the browser. Upon execution these are given as the actual CLI options for cache_dirs.

 

Looking at the source code of cache_dirs I see it does a hard coded search in /mnt/disk* and /mnt/cache, which explains the warning messages you get. These are harmless, but you may want to ask Joe to make a correction (e.g. test if Cache drive exists).

I don't care how I install cache dirs, either standalone or from Dynamix. But I need to know the correct go file command so it scans all drives and folders 4 deep. Thanks

Joe L, there's a bug in Cache_dirs, primarily affecting users with the -w option and no includes, no -i option.  I discovered it when comparing syslogs from the same user (and confirmed after viewing a number of other syslogs), where Cache_dirs would start at differing points of the drive mounting process.  In one syslog, Cache_dirs initiated during the mounting of Disk 10, within seconds of the mounting of Disk 1, resulting in a cached directory list of 3 folders, what it found on the first 9 disks only and not the Cache drive.  In another syslog of the same server, it initiated after the last drive and the Cache drive, resulting in a cached list of 7 folders.  When it initiates, it builds a cached directory list from what it can find at that moment, and never checks again until restarted.

 

I don't fully understand the problem (because I found exceptions), but it appears that the "at now + 1 minute" is the cause.  I assume someone thought that the "now + 1 minute" meant one full minute from now, but it actually means schedule it at the top of the next minute, which could be only seconds away.  So most Cache_dirs (but not all) are started at the top of a minute (eg. 17:48:00, not 17:48:13).  The simple way to fix that would be to change the command to something like "at now + 2 minutes", which would result in a minimum of 60 plus seconds.  My preference however would be 3 to 5 minutes, in case of long transaction replays.  A better solution though would be to test for the disks_mounted event, which occurs after all array disks are mounted, and the cache drive mounted, and the User Share system setup.

 

Since mounting is rather quick, this problem has probably not affected many users.  It should only affect users with a longer mounting time that started near the end of a minute.

 

After examining a number of syslogs, I would like to strongly recommend always using the -i option, to limit the folders cached to only those truly desired.  Using the -w option alone results in ALL top level folders on all array disk being included PLUS all top level folders on the Cache drive.  I saw a number of cases where almost certainly unwanted folders were being cached, which wastes memory (especially in 32 bit versions), and may cause the desired directory entries to be flushed out occasionally.

This is interesting RobJ as I can recall once when cache_dirs was running after a reboot I watched as chat_dirs read all my disks from 1 to 8 but missed disk7. After reading the disks and them all spinning down I browsed my openelec box and disk7 spun up but not the others. This could be in relation to the problem you have identified.

Joe L, there's a bug in Cache_dirs, primarily affecting users with the -w option and no includes, no -i option.  I discovered it when comparing syslogs from the same user (and confirmed after viewing a number of other syslogs), where Cache_dirs would start at differing points of the drive mounting process.  In one syslog, Cache_dirs initiated during the mounting of Disk 10, within seconds of the mounting of Disk 1, resulting in a cached directory list of 3 folders, what it found on the first 9 disks only and not the Cache drive.  In another syslog of the same server, it initiated after the last drive and the Cache drive, resulting in a cached list of 7 folders.  When it initiates, it builds a cached directory list from what it can find at that moment, and never checks again until restarted.

 

I don't fully understand the problem (because I found exceptions), but it appears that the "at now + 1 minute" is the cause.  I assume someone thought that the "now + 1 minute" meant one full minute from now, but it actually means schedule it at the top of the next minute, which could be only seconds away.  So most Cache_dirs (but not all) are started at the top of a minute (eg. 17:48:00, not 17:48:13).  The simple way to fix that would be to change the command to something like "at now + 2 minutes", which would result in a minimum of 60 plus seconds.  My preference however would be 3 to 5 minutes, in case of long transaction replays.  A better solution though would be to test for the disks_mounted event, which occurs after all array disks are mounted, and the cache drive mounted, and the User Share system setup.

 

Since mounting is rather quick, this problem has probably not affected many users.  It should only affect users with a longer mounting time that started near the end of a minute.

 

After examining a number of syslogs, I would like to strongly recommend always using the -i option, to limit the folders cached to only those truly desired.  Using the -w option alone results in ALL top level folders on all array disk being included PLUS all top level folders on the Cache drive.  I saw a number of cases where almost certainly unwanted folders were being cached, which wastes memory (especially in 32 bit versions), and may cause the desired directory entries to be flushed out occasionally.

Interesting observation.  (and I never gave much thought about the +1 minute, but of course you are correct, it is the next minute, which might only be seconds away.)

 

You are also correct in that it does not look for the top level folder once started.  It assumes they are in place and did not consider the issue when initially mounting disks.

 

Yes, the problem is when first starting cache_dirs from the config/go script.

Do something like this instead in the "go" script... (assuming cache_dirs resides at /boot/cache_dirs, otherwise, fix the path to where you put it.  you'll also need to add whatever " -i sharename" options you like to the line that creates the svcs_restarted script.):

 

mkdir -p /usr/local/emhttp/plugins/cache_dirs/event/

echo "/boot/cache_dirs -q" >/usr/local/emhttp/plugins/cache_dirs/event/stopping_svcs

echo "/boot/cache_dirs -w" >/usr/local/emhttp/plugins/cache_dirs/event/svcs_restarted

chmod +x /usr/local/emhttp/plugins/cache_dirs/event/stopping_svcs

chmod +x /usr/local/emhttp/plugins/cache_dirs/event/svcs_restarted

I have made some modifications to the cache_dirs script. I added the ability to adjust depth automatically based on whether scans are judged to cause disk access or not. It judges that a disk has been accessed during scan if scan takes a long time or if any recent disk access was made (and no recent disk access was made before scanning). The purpose being to avoid the situations where cache_dirs will continuously search through my files keeping disks busy all the time. Before it was also rather difficult to tell if cache_dirs was 'trashing' my disks, now its quite clear from the log if logging is enabled (though the log is rather large at the moment). If disks are kept spinning for some consequtive scans, the depth is decreased, and future rescan is scheduled at higher depth.

 

I havn't published it anywhere yet (except for a new public github account which I havn't shared yet), I wanted to hear you you Joe on how to proceed. You have been the creator and maestro of cache_dirs till now it appears with no messing around in your codebase, so I'm hesistant to share it before I get your approval. It would also be nice to just continue to have one version if you agree/accept on the changes.

 

Let me know if I should PM you a link, or just upload it here or what you think.

 

Best Alex

 

PS: I believe the changes are much less useful for x64 users with huge amounts of ram, but I'm still on x86 hence the sudden desire to implement my ideas.

I have made some modifications to the cache_dirs script. I added the ability to adjust depth automatically based on whether scans are judged to cause disk access or not. It judges that a disk has been accessed during scan if scan takes a long time or if any recent disk access was made (and no recent disk access was made before scanning). The purpose being to avoid the situations where cache_dirs will continuously search through my files keeping disks busy all the time. Before it was also rather difficult to tell if cache_dirs was 'trashing' my disks, now its quite clear from the log if logging is enabled (though the log is rather large at the moment). If disks are kept spinning for some consequtive scans, the depth is decreased, and future rescan is scheduled at higher depth.

 

I havn't published it anywhere yet (except for a new public github account which I havn't shared yet), I wanted to hear you you Joe on how to proceed. You have been the creator and maestro of cache_dirs till now it appears with no messing around in your codebase, so I'm hesistant to share it before I get your approval. It would also be nice to just continue to have one version if you agree/accept on the changes.

 

Let me know if I should PM you a link, or just upload it here or what you think.

 

Best Alex

 

PS: I believe the changes are much less useful for x64 users with huge amounts of ram, but I'm still on x86 hence the sudden desire to implement my ideas.

PM me a link.  Your changes sound interesting and useful.

 

I'll review and incorporate the improvements.

 

Joe L.

Does the default (9999) depth setting really read every file and directory on the array?

 

I used to run with depth of 3 or 4 in a previous version to limit memory usage; I have 8 gigs, so shouldn't be an issue, but still. Reading the post above it appears that you could leave it at default and get dynamic/smart caching without having to worry about depth setting.

I had 8Gb ram. when I was on unRaid 5 with only 32bits I could use depth 8 or 9, and any deeper and cache_dirs just kept my disks spinning and continously scanned drives, because cache was to small to keep all dirs in memory. on unRaid 6 I can go to depth 14. Its solely dependent on how many files you've got in that depth. If you have x64 you can probably just keep it on default.

 

Best Alex

Hello,

 

I installed this script into my server. It seems to work well, the folders are displayed without disk spin-up. However, when I checked this morning, I'm getting a bunch of errors displayed onto the console:

 

./cache_dirs: line 500: 28393 Segmentation fault      $command "$i"/"$share_dir" $args $depth > /dev/null 2>&1
./cache_dirs: line 500: 28620 Segmentation fault      $command "$i"/"$share_dir" $args $depth > /dev/null 2>&1
./cache_dirs: line 500: 28840 Segmentation fault      $command "$i"/"$share_dir" $args $depth > /dev/null 2>&1
./cache_dirs: line 500: 29049 Segmentation fault      $command "$i"/"$share_dir" $args $depth > /dev/null 2>&1
./cache_dirs: line 500: 29271 Segmentation fault      $command "$i"/"$share_dir" $args $depth > /dev/null 2>&1
./cache_dirs: line 500: 29482 Segmentation fault      $command "$i"/"$share_dir" $args $depth > /dev/null 2>&1
./cache_dirs: line 500: 29700 Segmentation fault      $command "$i"/"$share_dir" $args $depth > /dev/null 2>&1
./cache_dirs: line 500: 29923 Segmentation fault      $command "$i"/"$share_dir" $args $depth > /dev/null 2>&1

 

Any thoughts? Thank you!

 

 

I'm guessing its caused by out of memory. If you use x64 OS (unRaid 6) then cache_dirs is able to consume more memory, and defaults at a relatively large buffer. Possibly try reducing buffer by using param -U, see help from cachedirs ie this line:

 

echo " -U NN    =  set ulimit to NN to limit memory used (default=5000 on 32 bit, 50000 on 64 bit OS, '-U 0' sets no ulimit at all)"

 

best Alex

Thank you Alex. I tried that command:

 

root@winserver:/boot/config/plugins# cache_dirs -w -U 5000
sed: error while loading shared libraries: libc.so.6: failed to map segment from shared object: Cannot allocate memory
sed: error while loading shared libraries: libc.so.6: failed to map segment from shared object: Cannot allocate memory
sed: error while loading shared libraries: libc.so.6: failed to map segment from shared object: Cannot allocate memory
sed: error while loading shared libraries: libc.so.6: failed to map segment from shared object: Cannot allocate memory
./cache_dirs: xmalloc: make_cmd.c:100: cannot allocate 365 bytes (98304 bytes allocated)

 

It's a 64bit unRAID running on a machine with 16gigs RAM. I tried different -U values with the same result. What denominations are the values in? Bytes? Kilobytes? Thank you.

I don't know the unit, but 5000 is atmost 2mb as that was the max on 32 bit os, and I'm pretty sure its not too much. Though you can try

cache_dirs -w -U 5000 -d 5

for limited to max depth 5 (you can try different depths).

 

Do you have filled up your root filesystem, unRaids filesystem runs in memory, so you can exhaust your memory as far as I know.

'df' will not tell you root storage it seems. You can try

 

du -hs /*

 

(if you unmount the array it wont include you files).

 

It might also be a plugin which installs some bad libraries, but that sounds unlikely.

 

Otherwise I'm lost and have no further ideas for you.

 

Best Alex

 

Hello,

 

I am running unRAID 6.0 RC2 on the system described in my signature with 19 data drives of mostly media content, from TV shows and movies to music. I have have a client running XBMC/Kodi set to scan for new content regularly along with running the Subsonic docker on my server scanning every night my music share looking for new content.

 

I am very pleased with how everything is working and I'm finally getting to the last few tweaks to make my system just about perfect.

 

I would love to limit the amount of times my drives spin up so I'm thinking that using my cache drives to save my content during the day along with caching my directory content would limit any (most?) drive spin up to overnight when Subsonic scans for new music and the cache script move new content to the array. Well, except for when I'm actually watching content

 

So, I'm wondering if I should use the cache_dir script to cache my extensive media listing (40Tb+) to either memory or to a directory on my SSD cache drives to limit drive spin up to a minimum? Would this work? Is there just too much content to"index"?

 

Is the other way to use the XBMC/Kodi docker to save my content "index" on my cache drive and prevent my client machine "poling" the server regularly? Of course, this would also have the benefit of allowing a multiple XBMC/Kodi client environment to share one "index".

 

What would be best?

 

I would love to hear your suggestions!

 

TIA.

I have made some modifications to the cache_dirs script. I added the ability to adjust depth automatically based on whether scans are judged to cause disk access or not. It judges that a disk has been accessed during scan if scan takes a long time or if any recent disk access was made (and no recent disk access was made before scanning). The purpose being to avoid the situations where cache_dirs will continuously search through my files keeping disks busy all the time. Before it was also rather difficult to tell if cache_dirs was 'trashing' my disks, now its quite clear from the log if logging is enabled (though the log is rather large at the moment). If disks are kept spinning for some consequtive scans, the depth is decreased, and future rescan is scheduled at higher depth.

 

I havn't published it anywhere yet (except for a new public github account which I havn't shared yet), I wanted to hear you you Joe on how to proceed. You have been the creator and maestro of cache_dirs till now it appears with no messing around in your codebase, so I'm hesistant to share it before I get your approval. It would also be nice to just continue to have one version if you agree/accept on the changes.

 

Let me know if I should PM you a link, or just upload it here or what you think.

 

Best Alex

 

PS: I believe the changes are much less useful for x64 users with huge amounts of ram, but I'm still on x86 hence the sudden desire to implement my ideas.

PM me a link.  Your changes sound interesting and useful.

 

I'll review and incorporate the improvements.

 

Joe L.

 

Joe => Did you ever have a chance to look at these modifications and, if so, are they going to be incorporated?    I noticed on the first post in this thread that there haven't been any new releases of cache_dirs, so I assume this hasn't been done yet.    Just curious if it's in the works, as it does sound like a useful set of changes for x64 systems.

 

I have made some modifications to the cache_dirs script. I added the ability to adjust depth automatically based on whether scans are judged to cause disk access or not. It judges that a disk has been accessed during scan if scan takes a long time or if any recent disk access was made (and no recent disk access was made before scanning). The purpose being to avoid the situations where cache_dirs will continuously search through my files keeping disks busy all the time. Before it was also rather difficult to tell if cache_dirs was 'trashing' my disks, now its quite clear from the log if logging is enabled (though the log is rather large at the moment). If disks are kept spinning for some consequtive scans, the depth is decreased, and future rescan is scheduled at higher depth.

 

I havn't published it anywhere yet (except for a new public github account which I havn't shared yet), I wanted to hear you you Joe on how to proceed. You have been the creator and maestro of cache_dirs till now it appears with no messing around in your codebase, so I'm hesistant to share it before I get your approval. It would also be nice to just continue to have one version if you agree/accept on the changes.

 

Let me know if I should PM you a link, or just upload it here or what you think.

 

Best Alex

 

PS: I believe the changes are much less useful for x64 users with huge amounts of ram, but I'm still on x86 hence the sudden desire to implement my ideas.

PM me a link.  Your changes sound interesting and useful.

 

I'll review and incorporate the improvements.

 

Joe L.

 

Joe => Did you ever have a chance to look at these modifications and, if so, are they going to be incorporated?    I noticed on the first post in this thread that there haven't been any new releases of cache_dirs, so I assume this hasn't been done yet.    Just curious if it's in the works, as it does sound like a useful set of changes for x64 systems.

 

Bueller?

Hey guys

 

I want to update to unraid 6 tomorrow but I wanted to see if cache_dirs was still a seperate package since so many things have been packaged into the default system.

 

On a test rig I tried this plugin https://github.com/bergware/dynamix/blob/master/unRAIDv6/dynamix.cache.dirs.plg but it caused a really high cpu usage and a load over 2

 

I remember it happening on a different rig when I tried running the script manually on one of the early betas.

 

So what are people doing on Unraid 6?

You can read my recommendations for CacheDirs in the Upgrading to UnRAID v6 guide.

Hi,

 

is there any chance to exclude subdirectorys ? My Problem is that i have only one share called "server" under this share i have the folder "movie, picture, mp3 ....." But now cachedirs want to cache all the files. But my memory is with 8 gb to small to cache all these files, so the disks will spun up if i click through my dirs

 

 

Hi,

 

is there any chance to exclude subdirectorys ? My Problem is that i have only one share called "server" under this share i have the folder "movie, picture, mp3 ....." But now cachedirs want to cache all the files. But my memory is with 8 gb to small to cache all these files, so the disks will spun up if i click through my dirs

That feature does not exist as far as I know.  Most people tend to have multiple shares so it is not an issue.

 

Having said that I am surprised that you said cache_dirs cannot cache all the files with 8GB of RAM fitted.  Are you running other RAM hungry applications?