Spin down timers - are they in HDD firmware or stored in slackware

A more radical approach could be to monitor /proc/meminfo and when some condition is met issue a "echo 1 > /proc/sys/vm/drop_caches" which frees up all page cache and if i am redin this correctly does not effect dentrie and inode entries.

 

So what you are proposing is some daemon that should monitor the page cache and if it is going to flush the dentrie cache, choose to flush the page cache instead.

I'm not so sure this can be done efficiently at a user level.

What is needed is a kernel tuning that says to favor dentrie cache over page cache.

 

Perhaps what is needed agressive tuning in /etc/sysctl to flush the page cache more often.

 

 

see /etc/sysctl.conf

 

------

# This tunable is used to define when dirty data is old enough to be eligible for writeout
# by the pdflush daemons. It is expressed in 100'ths of a second. Data which has been dirty
# in-memory for longer than this interval will be written out next time a pdflush daemon wakes up.
# This was on 3000. This means that the file data should be first cached for 30 seconds before
# it is eligible to be written to disk.
# I set this to 100 so that after 1 second the data can be written to disk.
vm.dirty_expire_centisecs = 100

# The pdflush writeback daemons will periodically wake up and write old data out to disk.
# This tunable expresses the interval between those wakeups, in 100'ths of a second.
# This was on 500. This means that every 5 seconds the pdflush daemon is waken up to see if
# there is dirty data in the file buffer cache to be written to disk.
# I set this parameter to 50 to make pdflush more agressive to write data to disk.
vm.dirty_writeback_centisecs = 50

# Contains, as a percentage of total system memory, the number of pages at which a process which
# is generating disk writes will itself start writing out dirty data.
# This was on 40. This means that a process must have more then 40 percent of the physical memory
# of data cached before it starts writing data directly to disk. This is set to 10% which makes the
# processess in a earlier stage responsible for writing the data to disk. In this way you less
# depend on the pdflush daemon on flushing the data to disk.
vm.dirty_ratio = 10

# Contains, as a percentage of total system memory, the number of pages at which
# the pdflush background writeback daemon will start writing out dirty data.
# This was on 10, I changed to 5.
vm.dirty_background_ratio = 5

# Controls the tendency of the kernel to reclaim the memory which is used for
# caching of directory and inode objects.
# At the default value of vfs_cache_pressure=100 the kernel will attempt to
# reclaim dentries and inodes at a "fair" rate with respect to pagecache and
# swapcache reclaim.  Decreasing vfs_cache_pressure causes the kernel to prefer
# to retain dentry and inode caches.  Increasing vfs_cache_pressure beyond 100
# causes the kernel to prefer to reclaim dentries and inodes.
# This was 100, I changed to 10.  Consider changing to 0 (!).
vm.vfs_cache_pressure = 10

aggressive tuning is what I am experimenting on this now. I am lurking in ##kernel and when i am about long enough to wait for a answer I will ask the question for advice there. I have to be careful though reading some of the mailing list discussion this seems to be a religious topic for some developers.

 

Interesting though.... after watching a couple of avis my memory looks like:

 

more /proc/meminfo

MemTotal:      3624144 kB

MemFree:        119528 kB

Buffers:        193968 kB

Cached:        3070712 kB

SwapCached:          0 kB

Active:         137352 kB

Inactive:      3154080 kB

HighTotal:     2743872 kB

HighFree:         4020 kB

LowTotal:       880272 kB

LowFree:        115508 kB

SwapTotal:           0 kB

SwapFree:            0 kB

Dirty:               0 kB

Writeback:           0 kB

AnonPages:       26792 kB

Mapped:           5932 kB

Slab:           164200 kB

SReclaimable:   116024 kB

SUnreclaim:      48176 kB

PageTables:        440 kB

NFS_Unstable:        0 kB

Bounce:              0 kB

CommitLimit:   1812072 kB

Committed_AS:   135200 kB

VmallocTotal:   114680 kB

VmallocUsed:     14036 kB

VmallocChunk:   100572 kB

 

 

even though the box is idling this now.

 

Issuing:

 

echo 1 > /proc/sys/vm/drop_caches

 

And it now looks like:

 

more /proc/meminfo

MemTotal:      3624144 kB

MemFree:       3299176 kB

Buffers:         29268 kB

Cached:          82256 kB

SwapCached:          0 kB

Active:          84032 kB

Inactive:        54240 kB

HighTotal:     2743872 kB

HighFree:      2623396 kB

LowTotal:       880272 kB

LowFree:        675780 kB

SwapTotal:           0 kB

SwapFree:            0 kB

Dirty:               0 kB

Writeback:           0 kB

AnonPages:       26788 kB

Mapped:           5936 kB

Slab:           137448 kB

SReclaimable:   110164 kB

SUnreclaim:      27284 kB

PageTables:        440 kB

NFS_Unstable:        0 kB

Bounce:              0 kB

CommitLimit:   1812072 kB

Committed_AS:   135200 kB

VmallocTotal:   114680 kB

VmallocUsed:     14036 kB

VmallocChunk:   100572 kB

 

 

Quite a obvious difference although im not sure why Slab dropped as well.

 

Update: I should have ran sync before i issued the echo command incase someone else goes to replicate this.

Check these parms out too.

 

grep dentry /proc/slabinfo

cat  /proc/sys/fs/dentry-state

 

/proc/sys/fs/dentry-state is a new one to me will research it.

 

Not sure what this is though its likely relevant:

 

grep reiser_inode_cache /proc/slabinfo

Another excellent read explaining laptop_mode

 

http://www.linuxjournal.com/article/7539

 

Update:

 

I have spent many hours reading countless discussions and kernel notes along with samba stuff and anything else i can lay my hands on. The links Ive posted are the important ones ive negated to post about 100 that are less relevant. One thing I am now sure off is that the default kernel tunables are NOT suitable for the typical unRAID user (caveat: this is based on having a decent amount of memory). The defaults are tuned for users of desktops and a generic usage pattern that doesnt fit with unRAID.

 

I assume that the typical unRAID user:

 

accesses small number of large files

typically read way more than they write

have way more disks than most users.

consider heat a potential problem

want disks to stay spun down as much as possible

dont mind the lag when going to read a file on a disk that comes from disk spin up

want samba browsing to be instant but not at the expense of having disks online.

have a completely atypical user count to data storage ratio (normally it is assumed large data sets means large user sets)

 

We can definatly tune the kernel to favour this type of usage however I am unclear if we can tune it enough.

 

I am going to need so volunteers to test changes as we cannot make the assumption that my setup and usage matches everyones. There should be no real danger in this but normal testing disclaimers apply.

OK a small update. It seems the caching trick work much better when you are reading from unRAID than when you are writing. The investigation continues.

 

Question: apart from the ugliness can anyone think of any reason not to run a never ending script such as (in pseudo script):

 

ls -R blah

WAIT 5

repeat

 

Update: Journaling filesystems like ext3, reiserfs or xfs bypass the kernel's delayed write mechanisms.

Update2: Simplified down my changes to the ls cron, vfs cache pressue and laptop_mode changes. So far these changes are showing the most promising performance difference.

Had a thought that other may already have had.

 

Creating a large pagefile on a cahcedrive may help with this? One spun up cache drive is way better than 15 spun up data drives.

 

 

thoughts?

I don't know if dentry's would get paged out to swap vs just discarding them.

 

What I did learn recently is that unused portions of tmpfs and rootfs will page out to swap if necessary.

 

I'm still high on the possibility of a SAFE50 arrangement with unRAID.

Half of two drives in RAID0 for parity, the other half in RAID1 for Cache.  The best of both worlds.

Large parity and mirrored cache drive where you can put swap and a directory for unioning with the root ram filesystem.

 

just fyi

 

if you have duplicate entrys running any of these tricks to keep dentrys in RAM causes a syslog explosion.

 

root@TOWER:/proc# ls -alh /var/log/syslog

-rw-r--r-- 1 root root 1.4G Sep 14 18:07 /var/log/syslog

 

Keep in mind thats just 18 hours worth

just fyi

 

if you have duplicate entrys running any of these tricks to keep dentrys in RAM causes a syslog explosion.

 

root@TOWER:/proc# ls -alh /var/log/syslog

-rw-r--r-- 1 root root 1.4G Sep 14 18:07 /var/log/syslog

 

Keep in mind thats just 18 hours worth

I hope you have more than 1.8Gig of ram, otherwise, you're in for a hard crash any time now.

 

To truncate your syslog, I think you can type:

cp /dev/null /var/log/syslog

 

It should free up the space you've used. (You'll lose the older contents of the syslog, but on a running system with no other issues, that is not a big issue.)

 

Joe L.

Just run logrotate

 

/etc/cron.daily/logrotate

 

It's set to rotate the log once it is > 1M.

You may need to tweak it.

 

 

Maybe you need to move it to your hourly directory.

 

Also you may want to uncomment the following line

 

# uncomment this if you want your log files compressed:

#compress

 

 

 

root@Atlas [3] /mnt/disk1/bittorrent>more /etc/cron.daily/logrotate 
#!/bin/sh
/usr/sbin/logrotate /etc/logrotate.conf
root@Atlas [3] /mnt/disk1/bittorrent>

root@Atlas [3] /mnt/disk1/bittorrent>more /etc/logrotate.conf 
# /etc/logrotate.conf
#
# logrotate is designed to ease administration of systems that generate large
# numbers of log files.  It allows automatic rotation, compression, removal, and
# mailing of log files.  Each log file may be handled daily, weekly, monthly, or
# when it grows too large.
#
# logrotate is normally run daily from root's crontab.
#
# For more details, see "man logrotate".
#
# tmm - modified for unRAID

# keep 2 copies
rotate 2

# create new (empty) log files after rotating old ones:
create

# uncomment this if you want your log files compressed:
#compress

# Rotate /var/log/wtmp:
/var/log/wtmp {
    size 10K
    create 0664 root utmp
    rotate 1
}

# Rotate /var/log/btmp:
/var/log/btmp {
    size 10K
    create 0600 root root
    rotate 1
}

# Note that /var/log/lastlog is not rotated.  This is intentional, and it should
# not be.  The lastlog file is a database, and is also a sparse file that takes
# up much less space on the drive than it appears.

# system-specific logs may be also be configured below:

/var/log/syslog {
    size 1M
    postrotate
        /bin/kill -HUP cat /var/run/syslogd.pid 2>/dev/null 2>/dev/null || true
    endscript
}

 

Thank goodness for 4GB of RAM.

 

The main thing is ive always had doubts about the usefulness of dumping the duplicate log warnings to syslog. They dont tell you enough info to do anything about the files easily (since they dont tell you what the file is duplicated with). And more to the point they just keep saying the same thing over and over. For normal unRAID users the way this is implemented is not so useful and could do with some improvement.

 

I can fix my problem by routing out all the dupes.. I have the means to do this but a log saying xx is duped with yy would help most people.

I can fix my problem by routing out all the dupes.. I have the means to do this but a log saying xx is duped with yy would help most people.

 

Agreed, then again, you can do a

find /mnt -name ${nameinquestion} -ls

to find the files that have the same name.

 

Probably would be better if you ran locate scanning all the files first, then use the locate command to find all the files.

Less work on the system rather then traversing the whole tree.

 

Try something like this (all on one line):

grep "duplicate object" /var/log/syslog | cut -d" " -f8- | sed -e 's/\([^\/]*\/\)/*\//g' -e "s/^\/*\*/ls -l /" | sort -u | sh -

It is not too bad once you get a hang for regular expressions.  Just need to know where to put the backslashes.

 

On my system it prints:

-rwx------ 1 root root 899414 Oct 24  2006 /mnt/disk1/data/SetupDVDDecrypter_3.5.4.0.exe

-rwx------ 1 root root 899414 Nov 12  2006 /mnt/disk5/data/SetupDVDDecrypter_3.5.4.0.exe

-rwx------ 1 root root 899414 Oct 24  2006 /mnt/user/data/SetupDVDDecrypter_3.5.4.0.exe

It shows the two parallel files, and also the "user" file one of the two is affiliated with.  (probably the one with the same date)

 

Joe L.

 

That was cool, but it did not work on mine.

 

root@Atlas [3] /boot/custom/bin>showdups

sh: line 1: syntax error near unexpected token `('

sh: line 1: `ls -l /*/*/*/*/*/(TV)'

 

Probably because there is a () in my file name.

That was cool, but it did not work on mine.

 

root@Atlas [3] /boot/custom/bin>showdups

sh: line 1: syntax error near unexpected token `('

sh: line 1: `ls -l /*/*/*/*/*/(TV)'

 

Probably because there is a () in my file name.

You are correct.  The updated version would need to deal with escaping any characters in the file name, but not in the directory path.  Let me think a moment...

 

OK, try this:

grep "duplicate object" /var/log/syslog | cut -d" " -f8- | sed -e "s/^\/\([^\/]*\)\/\([^\/]*\)\/\(.*\)/ls -l \/*\/*\/'\3'/" | sort -u | sh -

Not as intuitive... but close.   I use single quotes to eliminate shell expansion of the trailing part of the path and file name, and I use the string, but un-quoted for the first two directories in the hierarchy.

 

This should be reasonably efficient, since the top level directories will be in RAM, and the lower levelr ones are explicitly in the path.

As I'm certain you'll see, I only substitute "*" for the /mnt/disk and /mnt/user directories.  The lower level ones are unchanged by the regular expression.

 

Let me know if this works.  I'll upgrade the "dupe files" plug-in on unmenu if it does to make it even better.

 

Joe L.

(in the middle of user-acceptance tests, apparently)

 

 

edit: fixed "cut" to get full file/folder name if it has embedded spaces.

Doent quite work here:

 

 

root@TOWER:/var/log# grep "duplicate object" /var/log/syslog | cut -d" " -f8 | sed -e "s/^\/\([^\/]*\)\/\([^\/]*\)\/\(.*\)/ls -l \/*\/*\/'\3'/" | sort -u | sh -

ls: cannot access /*/*/Movies/MOVIES: No such file or directory

ls: cannot access /*/*/TV/TV: No such file or directory

ls: cannot access /*/*/To: No such file or directory

 

Can i also suggest a syslog size sanity check first. Running this on a huge syslog kills the box for a fair length of time.

 

Last suggestion:

 

The natural progression of this is automatic dealing with the results. A perfect solution would be to move all the dupe files to a new base folder that maintains the old full path of the dupes

 

e.g.

 

Dupe file detected folder.jpg

Moved /user/mnt/TV/TV - A/Andromeda/Season 1/folder.jpg to /user/mnt/Dupes/TV/TV - A/Andromeda/Season 1/folder.jpg

 

This way you maintain the path info should the dupe file not be a true dupe.

 

Doent quite work here:

 

 

root@TOWER:/var/log# grep "duplicate object" /var/log/syslog | cut -d" " -f8 | sed -e "s/^\/\([^\/]*\)\/\([^\/]*\)\/\(.*\)/ls -l \/*\/*\/'\3'/" | sort -u | sh -

ls: cannot access /*/*/Movies/MOVIES: No such file or directory

ls: cannot access /*/*/TV/TV: No such file or directory

ls: cannot access /*/*/To: No such file or directory

 

Can i also suggest a syslog size sanity check first. Running this on a huge syslog kills the box for a fair length of time.

 

Last suggestion:

 

The natural progression of this is automatic dealing with the results. A perfect solution would be to move all the dupe files to a new base folder that maintains the old full path of the dupes

 

e.g.

 

Dupe file detected folder.jpg

Moved /user/mnt/TV/TV - A/Andromeda/Season 1/folder.jpg to /user/mnt/Dupes/TV/TV - A/Andromeda/Season 1/folder.jpg

 

This way you maintain the path info should the dupe file not be a true dupe.

 

Good suggestions.

The error messages you gave had the wild-cards expanded by the shell as /mnt/disk1, /mnt/disk2, /mnt/user/ etc.  That worked as expected.

I'm guessing perhaps that you syslog has records of files and folders that no longer exist.  That is why you get the "file or folder does not exist" messages.

You probably did have a /mnt/user/Movies/MOVIES folder at one time and have since renamed it.

 

I like the automatic movement to a "Dupes" folder idea.  It would work if each "duplicate" only existed on one other disk.  How would you like it to work if three or more folders had the same file in their tree?

 

Joe L.

Joe that was the sum total of results i got. My syslog is jammed with dupe entry results though so something is up.

 

root@TOWER:/var/log# grep duplicate syslog | wc -l

637863

 

What would help for me to try?

 

 

Multiple same dupes is a tricky one. Don't think there is a right way to do it but i suppose the neatest is renaming to .1, .2 etc and moving to the dupes folder.

Joe that was the sum total of results i got. My syslog is jammed with dupe entry results though so something is up.

 

root@TOWER:/var/log# grep duplicate syslog | wc -l

637863

 

What would help for me to try?

 

 

Multiple same dupes is a tricky one. Don't think there is a right way to do it but i suppose the neatest is renaming to .1, .2 etc and moving to the dupes folder.

First things first... I did not think when about all the possibilities when I was grabbing the 8th whitespace delimited field from the matching syslog lines to use as file names.  If the file name, or any of the folders above have embedded spaces, the 8th field would not have been the complete name, but only the first "word" of the name.

 

To fix that, all you need is to change the "cut -f8" to "cut -f8-"

That will provide from the 8th whitespace delimited field onward to the end of the line.   You might still have files in the syslog that no longer exist, but that is a much smaller issue.

 

The second is a bit trickier.  (How to deal with a huge number of log entries)

What do you get as an output with (how many distinct "files" do we need to look for):

grep "duplicate object" /var/log/syslog | cut -d" " -f8- | sed -e "s/^\/\([^\/]*\)\/\([^\/]*\)\/\(.*\)/ls -l \/*\/*\/'\3'/" | sort -u | wc -l

 

here we go (a couple of other things posted for completeness):

 

root@TOWER:/var/log# grep "duplicate object" /var/log/syslog | cut -d" " -f8- | sed -e "s/^\/\([^\/]*\)\/\([^\/]*\)\/\(.*\)/ls -l \/*\/*\/'\3'/" | sort -u | wc -l

1137

 

root@TOWER:/var/log# ls -alh

total 155M

...

-rw-r--r--  1 root root 155M Sep 15 13:51 syslog

 

root@TOWER:/var/log# grep duplicate syslog | wc -l

1243919

 

root@TOWER:/var/log# wc -l syslog

1248493 syslog

 

Hey Guys, Should this cool duplicate file finder fixer be in a separate thread?

Hey Guys, Should this cool duplicate file finder fixer be in a separate thread?

Probably...

Good call.

 

Creating one now watch out for it

seems this thread has enough valuable info to be moved to the user customization thread, no?