X9SCM-F slow write speed, good read speed

After startup of all my plugins performance went down again to 4 Mb/s.

i always thought it is a 64 bit OS, i cannot understand we have low and high memory, should it not just be one pool, or is this because of legacy PC hardware support or whatever.

Another data point. System RAM is 16GB.

Boot into 5.0-rc8a; array is off-line

Write test to an unformatted non-array drive: speed is good (>100MB/s)

Start array in maintenance mode

Write test to the same unformatted non-array drive: speed drops to 850kB/s

So just starting the array has a profound effect on non-array drives. I am not sure why my system slows down with 16GB while others see an improvement when downgrading to 16GB. The tunables are set to 4096 / 1024 / 1280 (md_num_stripes / md_write_limit / md_sync_window).

I'm also doing some limited testing this morning, again on Version 8a

And I am running with a full 32GB of ram as well.

When a parity check is running, i get transfer speeds on the array (disk to disk) of about 15MB/s

When I stop the parity check, and copy the same file, it speeds up to about 33Mb/s

Helmonder, if it would help you to 'mirror' my setup/plugins.  I could zip up the contents of my flash drive for you to share.

Has anyone running with 16G tried 8G?

I'm also doing some limited testing this morning, again on Version 8a

 

And I am running with a full 32GB of ram as well.

 

When a parity check is running, i get transfer speeds on the array (disk to disk) of about 15MB/s

 

When I stop the parity check, and copy the same file, it speeds up to about 33Mb/s

 

 

Helmonder, if it would help you to 'mirror' my setup/plugins.  I could zip up the contents of my flash drive for you to share.

Question: How many drives are in your array ?

The speeds you are getting are the speeds that should be there in my opinion... Those are the speeds I get on the previous version..

Mmmmm.... I left my array running since yesterday, this morning speeds were slow, BUT, that was around the time SABnzbd was just finished downloading and processing some stuff.

I just came home and did a transfer, it went thru (in-array) at 24 MB/s.

That is new behavious as far as I can remember, speeds just to remain low after they had gotten worse.. I am wondering if the kernel setting I made yesterday now is making sure that memory is freed where it did not use to ?

Absolutely no idea if that could be right and one experience does not prove anything, I will try some stuff..

33 MBps is perfectly normal. 24 may be normal for the drives in the system. I get 40 using only ST3000DM001 and ST2000DM001.

Started a new SABnzbd download while copying a few large directories.

When I -started- the SABnzbd my speed dropped to 10 MB/s, a couple of seconds later it was back up to 24 MB/s Speeds are still holding.

Please type this command at the console or telnet session and report if it makes any difference in write transfer throughput:

sysctl vm.highmem_is_dirtyable=1

Just woke up and did a test transfer (did not use above command yet), speeds 17.96 MB/s

Gave the command and did the same transfer backwards: 30,09 MB/s... at the top, 28 MB/s on average.

Did the transfer the other way around: 18.6 MB/s

Back again: 40 MB/s on top, 36 MB on average.

So something is going on... I still have that other parameter running and need to go work now, when I get home I will perform a clean reboot and retest.

Mmm... just got hot home earlier then expected, a quick test (same files as this morning): transfer foes at 37.93 KB/s (EXTREMELY slow), I did not do anything since this morning (so no reboots or whatsoever). I was copying from a 37% full 3TB WD RED to a 97% full 2TB WD EAR.

Now tried it the other way around (from the full to the 37% drive), transfer started lightning fast at 60 MB/s and went down to 30 MB/s. Tried another file, still 25 MB/s.

For testing sake I now tried to copy the files from the 37% drive to the 97% drive again (the first test), this starts out at 500 b/s and is creeping up slowly, getting faster slowly. At the end of a 1gig file I am at 6MB/S. The file that was queued and copied now starts at that level and continuous to get faster still. At the end of the next  1 gig file I am at 10 MB/s. Next file starts at that level and continuous to get faster, sometimes dropping but then climbing again from the level where it started to drop. Last one gig file ended at 16MB/s.

I did another one from the full to the empty drive, started out with a burst of 55 MB/s and ended at 33MB/s. Copying that file back again from the empty to the full drive starts at 1MB/s and gets faster and faster again, but quicker now. Speed was around 7MB/s at half of the file and at 8 MB/s at the end of the file (1 gig file).

SO...

1) There is an enormous difference in speed dependent on the drive that gets written to

2) When writing to a "slow" drive the speed gets faster during the transfer

The 97% full drive was WDC_WD20EARS-60MVWB0_WD-WCAZA8452711 (sdl)

The 37% full drive was WDC_WD30EFRX-68AX9N0_WD-WMC1T0076253 (sda)

All my drives have no SMART errors.

The 97% full ("slow") drive is on my AOC-SASLP-MV8.

The 37% full ("fast") drive is on my motherboard

I tried some other transfers:

From the 2 TB "slow" drive I used before to a WDC_WD20EARX-00PASB0_WD-WCAZAD290617 gives the same lightning fast performance, this EARX drive is also on the AOC-SASLP-MV8. Writing that same file back starts out at an over the top 80MB/s and evens out around 30 MB/s.

After a reboot, same drives as in first test and after making only the setting Tom suggested:

from 2TB to 3TB: 35 MB/s, no slow increase of speed, stable

from 3TB to 2 TB: Same behaviour, starts really slow but speed increases during the transfer.

This actually looks pretty workable... Next phase I will put back in the 16 gigs so I will be at 32 gig again to see if it makes a difference..

After a reboot, same drives as in first test and after making only the setting Tom suggested:

 

from 2TB to 3TB: 35 MB/s, no slow increase of speed, stable

from 3TB to 2 TB: Same behaviour, starts really slow but speed increases during the transfer.

 

This actually looks pretty workable... Next phase I will put back in the 16 gigs so I will be at 32 gig again to see if it makes a difference..

Let's try and not mix these issues.  The original issue has to do with slow disk throughput on a particular cpu/motherboard platform.  There is a separate issue of write slowing down as a a disk fills up, particularly when it's very full.  This latter issue is a known quirk of the file system and in fact many file systems perform poorly as the remaining free spaces approaches zero.

Since today i have a total slowdown with my new mainbord (Supermicro X9SAE-V, E3-1265Lv2) as well. (copy to cache disk)

small files 2 Kbyte/sec, larger files 500 Kbyte/sec.

I put the command in Tom placed, and withouth the cache drive now, speeds up to 70 MByte/sec (copy to a 3 TB hdd)

I do not know what that is, but it seems to work.

After a reboot, same drives as in first test and after making only the setting Tom suggested:

 

from 2TB to 3TB: 35 MB/s, no slow increase of speed, stable

from 3TB to 2 TB: Same behaviour, starts really slow but speed increases during the transfer.

 

This actually looks pretty workable... Next phase I will put back in the 16 gigs so I will be at 32 gig again to see if it makes a difference..

 

Let's try and not mix these issues.  The original issue has to do with slow disk throughput on a particular cpu/motherboard platform.  There is a separate issue of write slowing down as a a disk fills up, particularly when it's very full.  This latter issue is a known quirk of the file system and in fact many file systems perform poorly as the remaining free spaces approaches zero.

I agree, thats why I stated I see this as beiing workable.. I have tried not to make assumptions based on my numbers, if it sounded like that, was not meant in that way !

In addition: I have a preclear running at 127 MB/s. Also looks ok.

When I copy a big file at the same time I still start out at 55 MB/s, after a coupe of seconds it starts to slow down a bit.

As far as I am concerned this is a-ok behaviour... I will need to wait for the preclear to finish before I can switch back in the 16gigs more but I do not expect that becomming an issue (removing it also did not make a difference in my case).

Please type this command at the console or telnet session and report if it makes any difference in write transfer throughput:

 

sysctl vm.highmem_is_dirtyable=1

This works, write speed is back to 50+ MB/s. Terrific - thanks Tom. Is it ok to put this into my go script?

Please type this command at the console or telnet session and report if it makes any difference in write transfer throughput:

 

sysctl vm.highmem_is_dirtyable=1

I type the above and get command not found?

recheck your typing, that should not be possible..

I found the issue.

I needed to be logged in as Root.  SSH'ing as another user did not work.

I'll try and test some speeds now and see what happens.

I found the issue.

 

I needed to be logged in as Root.  SSH'ing as another user did not work.

 

I'll try and test some speeds now and see what happens.

This is odd because root is the only user wit a login shell configured. No other user should be able to remote login unless you've modified some some linux user settings.

Just tried with a restarted system running the parameter Tom supplied in the GO file, I also have reinserted the 8gig I took out, now running on 16gig. Tried testtransfers and they are as quick as with the 8gig, if anything then quicker.

So I really think we can call this one licked, thanks Tom !!

I don't know how comfortable I am with calling this one fixed.  I'm not a kernel guru, but if you read the comments at the link that Tom provided in post #70, there seems to be a concern with "other impacts" with this command/parameter setting.  Like I said, I'm not knowledgeable enough to know if this has any other side effects or not.

Edit:  I tried this command last night with 16GB RAM and it did appear to increase write speed.  I need to do some further testing but have been working ~ 15 hours days this week...will likely have time to test further this weekend.

I found the issue.

 

I needed to be logged in as Root.  SSH'ing as another user did not work.

 

I'll try and test some speeds now and see what happens.

 

This is odd because root is the only user wit a login shell configured. No other user should be able to remote login unless you've modified some some linux user settings.

How did you install OpenSSH?  I used this plugin here versus the one found in UnMenu.    I then added a second user versus using root

http://lime-technology.com/forum/index.php?topic=20848.0