cache_dirs - an attempt to keep directory entries in RAM to prevent disk spin-up


Recommended Posts

is there any chance to exclude subdirectorys ? My Problem is that i have only one share called "server" under this share i have the folder "movie, picture, mp3 ....." But now cachedirs want to cache all the files. But my memory is with 8 gb to small to cache all these files, so the disks will spun up if i click through my dirs :(

 

You did not say whether you are running v5 or v6, and whether you are running the plugin or the script from the go file.  If you are running this in v5, then you are limited to low memory only, which is considerably less than 1GB.

 

I don't know your layout, but have you tried the depth parameter?  It defaults to unlimited, but you can set it to 3 or 5 or whatever, and limit it to only scanning that deep into the folder levels.  At the command line, it is "-d 5" (for example).  In the plugin, it is the Maximum scan level depth: parameter.

Link to comment

@RobJ

 

thanks for your answer. I´m running v5 with Dynamix Addon. I think depth parameter will not bring anything for me beacuse the movie structure is server\movie\A-C\Alien\Alien.mkv . All the other dirs called picture, series, mp3 have a smaller deep than the movie dir. and the movie dir is the dir i want to cache.

 

But when i understand you right with my bad english then i use in v5 only 1 gb for caching from my 8 gb memory ? Was it better for me and the caching to switch to v6 ? But why show me my system stat that from 8 gb there will be used 7 gb for cache ?

Link to comment

But ... i use in v5 only 1 gb for caching from my 8 gb memory ? Was it better for me and the caching to switch to v6 ? But why show me my system stat that from 8 gb there will be used 7 gb for cache ?

 

There are different caches for different things.  The cache for folder entries is in lower memory and as far as I know cannot use the high memory, like caching disk files can.  So that only gives you less than 500MB I believe, possibly much less.  Moving to the 64 bit v6 would give you access to all of your memory, as Alex said.

Link to comment

I think its probably more like 2-4Gb (maybe 4GB if the whole 32bit can be used and are not used elsewhere). I once tested memory usage of cachedyrs on x86 unRaid by filling ram by placing huge file on / and if my memory serves me it was back then more than 1GB cache_dirs consumed, though I'm not sure. But anyway, exact numbers are irrelevant I guess.

Link to comment

Not sure how many bytes it takes to cache a single file entry in the directory, but I'd assume it's not more than 256 bytes.    At 256 bytes/entry, that's 4 entries per k; 4,000 per MB, and 4,000,000 per GB.    I doubt many of us have 4 million files on our servers ... so a GB should be VERY adequate for caching the entire directory structure.  Even at 1k/entry you could still cache a million files in a GB.

 

 

 

Link to comment

Hi All -

 

posted this in the Dynamix plugins thread but thought I'd crosspost here since I'm having a problem with cache_dirs specifically.

 

I recently upgraded to v6, and was using cache_dirs without a problem in v5.  I have 4GB RAM, about 225K files (but not all are included in cache_dirs), and my folders are no deeper than five levels.  Using the same arguments I did with v5, my disks keep spinning up with the plugin version in v6.  I've tried messing with the min and max arguments (as discussed previously in the thread), the cache pressure, etc. but am not having any luck keeping my disks spun down.  The min scan time argument had the largest effect when bumped up to 10, which kept the disks spun down for about 10-15 minutes before they started to spin up.  If I disable folder caching, then the disks remain spun down.

 

A far more minor problem is that I can't cache one of my user shares named "3D Movies" that I didn't have a problem with in v5.  Other user shares with spaces in the name are cached successfully; it is only this particular share that doesn't remain selected in the GUI after being started.

 

Any advice would be appreciated.

Link to comment

Hi bonienl -

 

Thanks for the suggestion.  I've tried RobJ's suggestions in the wiki but unfortunately, the drives keep spinning up.  Puzzled as to what may be occurring as there are no other computers that are accessing the server.

 

Yeah that is the latest version. I am using that version as well on v6, and disks spin down as should be.

 

You may want to have a look at this excellent guide: http://lime-technology.com/wiki/index.php/Upgrading_to_UnRAID_v6

 

In here is a section about cache_dirs and some recommendations.

Link to comment

I have Dockers installed, and shared directories all reside on the cache drive, but disabled Dockers for the purposes of testing.  Last night, I started all Dockers but disabled folder caching and all drives remained spun down this morning.  After enabling folder caching, the drives started spinning up.  I gave it a few minutes, spun down all drives and leaving folder caching enabled, and the drives started spinning up again shortly afterwards.  Disabling folder caching, and spinning down all the drives kept the drives spun down.

 

Any Docker applications running?

Link to comment

I am a bit puzzled here, also because you are saying that folders with a space in their name don't work.

 

Can you open the log function (button top right), then stop/start cache_dirs, and post the output of the log part about cache_dirs starting together with its settings.

 

 

Link to comment

Sorry, to clarify:

 

1) Dockers can be enabled, and the drives when spun down remain spun down.

a. Anytime folders caching is enabled, drives start spinning up even after a forced spin down of all disks.

2) Only one folder that has a space in the name cannot be cached (even if selectable in GUI).  Other folders with spaces in the name can be cached without a problem.  Running cache_dirs manually to cache the problematic share also errors out.

 

Here's the part of the log after I quit cache_dirs and restart it:

 

Jul  5 22:53:43 Tower cache_dirs: killing cache_dirs process 9142
Jul  5 22:54:02 Tower cache_dirs: ==============================================
Jul  5 22:54:02 Tower cache_dirs: command-args=-w -B -i Movies\ A-D -i Movies\ E-J -i Movies\ K-O -i Movies\ P-S -i Movies\ T-Z -i TV -U 0 -p 100 -m 8 -d 5
Jul  5 22:54:02 Tower cache_dirs: vfs_cache_pressure=100
Jul  5 22:54:02 Tower cache_dirs: max_seconds=10, min_seconds=8
Jul  5 22:54:02 Tower cache_dirs: max_depth=5
Jul  5 22:54:02 Tower cache_dirs: command=find -noleaf
Jul  5 22:54:02 Tower cache_dirs: version=1.6.9
Jul  5 22:54:02 Tower cache_dirs: ---------- caching directories ---------------
Jul  5 22:54:02 Tower cache_dirs: Movies A-D
Jul  5 22:54:02 Tower cache_dirs: Movies E-J
Jul  5 22:54:02 Tower cache_dirs: Movies K-O
Jul  5 22:54:02 Tower cache_dirs: Movies P-S
Jul  5 22:54:02 Tower cache_dirs: Movies T-Z
Jul  5 22:54:02 Tower cache_dirs: TV
Jul  5 22:54:02 Tower cache_dirs: ----------------------------------------------
Jul  5 22:54:02 Tower cache_dirs: cache_dirs process ID 26320 started, To terminate it, type: cache_dirs -q

 

and a small screen showing that "3D Movies" is checked off.

 

dElQKRkl.jpg

 

I am a bit puzzled here, also because you are saying that folders with a space in their name don't work.

 

Can you open the log function (button top right), then stop/start cache_dirs, and post the output of the log part about cache_dirs starting together with its settings.

Link to comment

I have updated my extended version of the cache dirs. To spike your memory: In my version I added adaptive depth level, to prevent cache_dirs from thrashing disks when they are otherwise occupied and cache is evicted.

 

In the new version I bugfixed and improved it and tested on x64.

 

I think I should publish it under another name such as cache_dirs_adaptive if Peter does not have time or the desire to check/approve/edit the changes. Do you agree?

 

Best Alex

Link to comment

Sorry, to clarify:

 

1) Dockers can be enabled, and the drives when spun down remain spun down.

a. Anytime folders caching is enabled, drives start spinning up even after a forced spin down of all disks.

2) Only one folder that has a space in the name cannot be cached (even if selectable in GUI).  Other folders with spaces in the name can be cached without a problem.  Running cache_dirs manually to cache the problematic share also errors out.

 

Here's the part of the log after I quit cache_dirs and restart it:

 

Jul  5 22:53:43 Tower cache_dirs: killing cache_dirs process 9142
Jul  5 22:54:02 Tower cache_dirs: ==============================================
Jul  5 22:54:02 Tower cache_dirs: command-args=-w -B -i Movies\ A-D -i Movies\ E-J -i Movies\ K-O -i Movies\ P-S -i Movies\ T-Z -i TV -U 0 -p 100 -m 8 -d 5
Jul  5 22:54:02 Tower cache_dirs: vfs_cache_pressure=100
Jul  5 22:54:02 Tower cache_dirs: max_seconds=10, min_seconds=8
Jul  5 22:54:02 Tower cache_dirs: max_depth=5
Jul  5 22:54:02 Tower cache_dirs: command=find -noleaf
Jul  5 22:54:02 Tower cache_dirs: version=1.6.9
Jul  5 22:54:02 Tower cache_dirs: ---------- caching directories ---------------
Jul  5 22:54:02 Tower cache_dirs: Movies A-D
Jul  5 22:54:02 Tower cache_dirs: Movies E-J
Jul  5 22:54:02 Tower cache_dirs: Movies K-O
Jul  5 22:54:02 Tower cache_dirs: Movies P-S
Jul  5 22:54:02 Tower cache_dirs: Movies T-Z
Jul  5 22:54:02 Tower cache_dirs: TV
Jul  5 22:54:02 Tower cache_dirs: ----------------------------------------------
Jul  5 22:54:02 Tower cache_dirs: cache_dirs process ID 26320 started, To terminate it, type: cache_dirs -q

 

and a small screen showing that "3D Movies" is checked off.

 

dElQKRkl.jpg

 

I am a bit puzzled here, also because you are saying that folders with a space in their name don't work.

 

Can you open the log function (button top right), then stop/start cache_dirs, and post the output of the log part about cache_dirs starting together with its settings.

 

Can you try and rename "3D Movies" to "Movies 3D" (the difference: the name doesn't start with a digit)

 

Link to comment

 

Can you try and rename "3D Movies" to "Movies 3D" (the difference: the name doesn't start with a digit)

 

Hi bonienl,

 

I created a new share with that name, and it worked.  Here's a strange wrinkle, if I excluded (-e switch) all directories except for the media folders, "3D Movies" was cached without a problem.

 

Now if only I can figure out why my disks keep spinning up.  I think I'll go back to an older version of cache_dirs I had on v5 and see if that makes any difference.

 

Alex, I would be interested in trying out your version if you can share.

Link to comment
  • 3 weeks later...

Hi Guys,

 

I'm using this on unRAID 6 (latest release) and have realised it stops Plex from detecting folder changes, hence it will not automatically update the library when a new movie is added.

 

Would this make sense, can it be resolved as I really like the functionality of this plugin... :-/

 

Thank you in advance,

 

Mark

Link to comment

Hi Guys,

 

I'm using this on unRAID 6 (latest release) and have realised it stops Plex from detecting folder changes, hence it will not automatically update the library when a new movie is added.

 

Would this make sense, can it be resolved as I really like the functionality of this plugin... :-/

 

Thank you in advance,

 

Mark

Strange as Cache-Dirs does not actually change anything on the disks.  All it does is periodic listings of files/folders to try and keep the relevant directory entries in RAM for performance reasons to improve the experience of browsing the file system.  Cannot see off-hand how that would affect Plex.
Link to comment

Hi Guys,

 

I'm using this on unRAID 6 (latest release) and have realised it stops Plex from detecting folder changes, hence it will not automatically update the library when a new movie is added.

 

Would this make sense, can it be resolved as I really like the functionality of this plugin... :-/

 

Thank you in advance,

 

Mark

Strange as Cache-Dirs does not actually change anything on the disks.  All it does is periodic listings of files/folders to try and keep the relevant directory entries in RAM for performance reasons to improve the experience of browsing the file system.  Cannot see off-hand how that would affect Plex.

 

I agree it does seem odd, but having disabled the plugin, Plex is now detecting folder changes again.

 

As far as I can understand, Plex uses native features of the OS to detect folder changes, in my case Plex is installed on Windows Server 2012 R2. I can only assume that cache_dirs affects the ability to detect updates.

 

FYI, this is an example of how Windows detects directory changes - https://goo.gl/KSSxez

Link to comment
  • 1 month later...

I just installed the cache_dirs v1.6.9  plugin for unRAID V6.0.1.  When I try to include 4 directories and press Apply, the first directory gets unchecked/dropped.  So the result is that cached_dirs only caches 3 out of the four.  My directories are named as followes: Asia, Movies, PBN, Shows

 

To include the first directory, Asia, I have to use the User Defined options of -i Asia.

Link to comment

I just installed the cache_dirs v1.6.9  plugin for unRAID V6.0.1.  When I try to include 4 directories and press Apply, the first directory gets unchecked/dropped.  So the result is that cached_dirs only caches 3 out of the four.  My directories are named as followes: Asia, Movies, PBN, Shows

 

To include the first directory, Asia, I have to use the User Defined options of -i Asia.

 

This is not a cache_dirs problem. You need to post on the dynamix plugins for v6.  It's a dynamix issue.

Link to comment
  • 3 months later...

I and others have been having a problem with the cache_dirs script stopping child processes quickly.  This is becoming a real issue for me and others using the dynamix cache_dirs plugin on V6.  If an array stop is initiated, the cache_dirs process is stopped with -q, but the child processes continue to run until they have completed.  This can take some time and while it is happening, the drives cannot be unmounted.

 

I am proposing the following change to line 250 of the cache_dirs script:

      pkill -TERM -P "$lock_pid"

 

This kills the cache_dirs process and all child processes, rather than just the cache_dirs process alone.

 

Joe L.,

Can you review this change for appropriateness and incorporate the change into the cache_dirs script?

 

EDIT: I did some more testing on this and I don't think I have the complete answer yet.  There are still child 'find' processes running at times.  Hopefully Joe L. can come up with a better fix.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.