Read/Write Performance on New 5.0.6 Build


Recommended Posts

Just got my 5.0.6 system up and running and running a few tests.  I'm getting 104/108 MB/s read/write using CrystalDiskMark 3.0.4 x64 on Intel Pro/1000 EXPI9402PTBLK on HP 8 port GS108E gigabit Hub.  But when I actually copy multiple gigabytes of music files, I'm seeing 7MB/s.  This seems really slow.  Any ideas why it would be so slow?

Link to comment

When you copy a LOT (i.e. thousands) of SMALL files, EVERY file causes a directory update on the target disk ... i.e. the drive has to seek to the directory; update it; and then seek back to where the next file needs to be written.    This will be MUCH slower than copying a few LARGE files that occupy the same amount of space.

 

If you're copying to a drive that uses write caching, this will be much less of an issue, as the directory writes will be cached, and done in a group.  But when you're copying across a network, this is generally not the case.

 

Copy a few LARGE (multi-GB) files and see what kind of speeds you see with those.    THAT will tell you your real write speed.    [if you're copying TO the parity protected array, you'll likely see ~ 35MB/s or so.]

 

Link to comment

You should also get an improvement in the speed of creating lots of new small files if you have CacheDirs installed as the creation process has to check all disks in a share to see if the file already exists.  Having the directory entries already in RAM should speed this up.  I say SHOULD just in case you have a LOT of files and not enough RAM for CacheDirs to hold directory entries for all of them in RAM.

Link to comment

I'm working off the trial version at this point, so don't have a cache drive yet and can't test that.  Is there a way to schedule the mover to run in real-time or every few minutes?  I'll have many TB of data to move.  Perhaps it would make more sense to move the data over on a physical drive?  Attach locally on PC, move files, the move disk, then copy files from disk to array?

Link to comment

Writing to the protected array should be in the 35MB/s range.    Speeds in the 11-12MB/s range generally indicate that your network is only operating at 100Mb ... NOT the Gb speed you're expecting.

 

Change your Ethernet cables and see if that helps -- ANY discontinuity in one of the pairs in the cable will slow things down to 100Mb.    You could also have a bad switch or router port, but the cable is the most likely culprit.

 

Link to comment

Excellent suggestions.  I did the following.

 

A. Checked BIOS.  Virtualization is disabled.

http://my.jetscreenshot.com/12412/20150418-ah5m-144kb

 

B. Bought BRAND NEW Cat 6 cables.

 

C. Root Cause Diagnostics

> Stripped network down to only essentials - PC (GB Intel on Mobo), Laptop (GB Intel on Mobo) and UnRaid on known Cat 6 cables on single gigabit router (Netgear WNDR4000).

> Verified ALL interfaces reporting gigabyte connections.

http://my.jetscreenshot.com/12412/20150419-vxnz-135kb

 

D. Conducted Test Transfers to test network connections/xfer rates

1. Transfer video from PC to Laptop.  108MB/s transfer rate avg.

2. Transfer video from Laptop to PC.  104MB/s transfer rate avg.

3. Transfer video from PC to Unraid 5.0.6.  11MB/s transfer rate avg.

4. Transfer video from Laptop to Unraid 5.0.6.  12MB/s transfer rate avg.

5. Transfer video from Unraid to Laptop. 2.25MB/s transfer rate avg.

6. Transfer video from Unraid to PC.  2.3MB/s transfer rate avg.

 

Then I reconfigured my network back to its standard config and ran the same tests.  Identical results.

 

E. Verify components work in another OS (Windows 7 professional)

> Then I installed an SSD in another port on the Unraid server (my Apps drive, not yet setup in UnRaid) using the Syba PCI-E Sata Card (disable ALL the features of the BIOS – marvell, intel, jbd, USB 3.0, etc).

> Then I installed WIN7 Professional.  Installed latest Intel Ethernet (Intel® 82579) NIC Driver Issue (Version 17.3.0.0, which also required NVM Update Utility for Intel 82579V Gigabit Ethernet PHY Network Connection).

 

F. Conduct PC to PC transfer from UnRaid Hardware Configuration

> Test Transfers - Laptop/PC to/from Unraid using MOBO Intel NIC

1. Transfer video from Laptop to Unraid PC running WIN7.  118MB/s transfer rate.

2. Transfer video from Unraid PC running WIN7 to PC.  102MB/s transfer rate.

> Then I disabled the mobo Intel 82579 and installed the Intel Pro/1000 EXPI9402PTBLK adapter.  Evidently the Windows Driver supports the Intel Pro/1000 as the network came live (at 1GB/s) as soon as the PC booted.

> Test Transfers - Laptop/PC to/from Unraid using Intel Pro/1000 NIC

1. Transfer video from PC to Unraid PC running WIN7.  99MB/s transfer rate.

2. Transfer video from Unraid PC running WIN7 to Laptop.  105MB/s transfer rate.

 

G. Test Supermicro Adapter

I’m waiting on instructions how to install the AOC-SAS2LP-MV8 on Windows 7 to test that.  Will get the answer to that on Monday I suspect.

 

H. Next Steps/Conclusion?

Not sure what more i can do.  The hardware on this system appears to be solid and performing within the reasonable range of specs.

 

Have I missed something?  Other ideas to try to diagnose this?

 

Just FYI - image of Pro/1000 NIC

http://my.jetscreenshot.com/12412/20150419-y1kn-165kb

http://my.jetscreenshot.com/12412/20150419-bdf8-173kb

  • Haha 1
Link to comment

Excellent testing regimen.  You've clearly eliminated cabling, NIC, and router ports from the possibilities, since the identical hardware gives superb speeds when running Windows.

 

What's really interesting is that your READS from the UnRAID array are significantly slower than your writes !!

 

Next thing I'd do, since you clearly have a few spare drives to "play" with here, is set up a New Config in UnRAID with ONE device assigned as data -- no parity, no cache ==> and then see what kind of read and write speeds you get to that drive.    You could even use your spare SSD (the one you installed Windows on), unless you want to leave that Windows install on it.

 

This will let you confirm whether or not this is some strange UnRAID driver issue [Clearly you SHOULD get VERY good speeds for both reads and writes].    If in fact you DO get excellent speeds (or, for that matter, even if you don't); then shut down and change where you're connecting that single drive to see if that changes anything [i'd try it 3 times -- once on a motherboard port; once on the SYBA card; and once on the SAS2LP].

 

Link to comment

Update on my testing.  Over the course of today, I've steadily tested this environment trying to isolate what's been affecting both performance and server crashes (see http://lime-technology.com/forum/index.php?topic=39215.0 and http://lime-technology.com/forum/index.php?topic=39293.0).

 

I decided to really evaluate server performance gradually building confidence in my server configuration and function.  Thought I'd share the results...  Net I think I'm pretty close to saturating the ethernet given protocol overhead, etc.

 

I’ve been working on Unraid 6b15 regarding some crash issues, so that’s the server platform I have active at the moment (sorry, I know this discussion has be 5.0.6 so far).  So I ran the following tests.  Interesting results.  But the XFS results should be applicable for 5.0.6 too?

 

I ran two different type of performance tests.  A movie file (12 monkeys) cut and paste via Windows share to Server drive and back a couple times each (I took the last MB/s displayed by Windows 7 File Manager (yes I know it's not that accurate, but its a relative comparison).  And using CrystalDiskMark x64 v3.0.4.  Generally CrystalDiskMark showed better performance.

 

You'll notice this is with NO PARITY.  Just direct file access.  I think Parity might be the cause of my crashes.  We'll see, my next set of tests will be to start parity and run some more of these tests to see if it panics again.

 

Here's a link to a summary table (each column is Read/Write speeds).  Copy is simple copy using Windows 7 File Manager, CDM 50 is CrystalDiskMark 50MB and CDM4K is 4000MB:

http://my.jetscreenshot.com/12412/20150420-sxkh-43kb.jpg

 

1. No parity, 2TB Hitachi HDS5C3020ALA632/667gb platter/5900rpm/32gb Cache - (formatted XFS)

-- CrystalDiskMark (50mb) – http://my.jetscreenshot.com/12412/20150419-p1yu-46kb.jpg got only 37MB read, ran a second time and got much better - http://my.jetscreenshot.com/12412/20150419-znw5-44kb.jpg

-- CrystalDiskMark (4000mb) – http://my.jetscreenshot.com/12412/20150419-5o8y-46kb.jpg

-- Copy PC to UR6 - 80MB/s.  Still 20MB/s slower than OS to OS, but much better.  You (garycase) mentioned 35MB/s as the expected rate (which was close to the CrystalDiskMark).  Why is that so slow compared to PC to PC xfer?  And on the PC to UR6 why is the Read twice what the CrystalDiskMark showed on the same server?

-- 2nd Copy UR6 to PC - 90.3MB/s.

-- Copy UR6 to PC - 78.3MB/s.

-- 2nd Copy UR6 to PC - 78.3MB/s.

 

2. No parity, 2TB Hitachi HDS5C3020ALA632/Sata III/667gb platter/7200rpm/32gb Cache - (formatted BTRFS)

-- CrystalDiskMark (50mb) – http://my.jetscreenshot.com/12412/20150420-zina-47kb.jpg

-- CrystalDiskMark (4000mb) – http://my.jetscreenshot.com/12412/20150420-ktyb-47kb.jpg

-- Copy PC to UR6 – 114MB/s

-- 2nd Copy UR6 to PC - 113MB/s.

-- Copy PC to UR6 - 106MB/s.

-- 2nd Copy UR6 to PC - 102MB/s.

 

3. No parity, 1TB Seagate ST31000528AS/Sata II/1000gb platter/7200rpm/64gb Cache - (formatted XFS)

-- CrystalDiskMark (50mb) – http://my.jetscreenshot.com/12412/20150420-qv6g-45kb.jpg

-- CrystalDiskMark (4000mb) – http://my.jetscreenshot.com/12412/20150420-tilf-47kb.jpg

-- Copy PC to UR6 – 113MB/s

-- 2nd Copy UR6 to PC - 114MB/s.

-- Copy PC to UR6 - 107MB/s.

-- 2nd Copy UR6 to PC - 97MB/s.

 

4. No parity, 1TB Seagate ST31000528AS/Sata II/1000gb platter/7200rpm/64gb Cache - (formatted BTRFS)

-- CrystalDiskMark (50mb) – http://my.jetscreenshot.com/12412/20150420-ihbd-49kb.jpg

-- CrystalDiskMark (4000mb) – http://my.jetscreenshot.com/12412/20150420-s4bs-47kb.jpg

-- Copy PC to UR6 – 113MB/s

-- 2nd Copy UR6 to PC - 102MB/s.

-- Copy PC to UR6 - 97MB/s.

-- 2nd Copy UR6 to PC - 112MB/s.

 

5. No parity, 750GB Samsung EVO (BTRFS)

-- CrystalDiskMark (50mb) – http://my.jetscreenshot.com/12412/20150420-kskc-46kb.jpg

-- CrystalDiskMark (4000mb) – http://my.jetscreenshot.com/12412/20150420-ca8m-45kb.jpg

-- Copy PC to UR6 - 118MB/s, second run 114MB/s.

-- Copy UR6 to PC – 108MB/s, second run 107MB/s.

 

I’m assuming that the faster speed of the SSD will translate to when I have that SSD setup as a cache drive?  So essentially, once the cache drive is in place, all my transfer speeds (write to server) should match the SSD (PC to UR6) speed?

 

6. No parity, 750GB Samsung EVO (XFS)

-- CrystalDiskMark (50mb) – http://my.jetscreenshot.com/12412/20150420-ofkm-46kb.jpg

-- CrystalDiskMark (4000mb) – http://my.jetscreenshot.com/12412/20150420-xizd-47kb.jpg

-- Copy PC to UR6 - 114MB/s, second run 113MB/s.

-- Copy UR6 to PC – 92.4MB/s, second run 96.6MB/s.

 

 

Link to comment

These speeds all look fine => clearly UnRAID isn't having any networking-related issues; and your disks are performing fine.

 

Did you try this with the disk connected to each of the 2 different controller options?  [motherboard & SAS2LP)

 

r.e. your question:

...You (garycase) mentioned 35MB/s as the expected rate (which was close to the CrystalDiskMark).  Why is that so slow compared to PC to PC xfer?

 

35-40MB/s is a typical write speed for a PROTECTED array ... i.e. WITH parity, and without a cache drive involved.    Since you were writing without parity you should have had -- and DID -- much higher speeds;  and you'll get roughly the same results even with a parity drive as long as you're writing to cached shares.

 

Link to comment

Thanks Gary, parity is at 85%, so should be able to test that shortly.  And yes, next test is the HBA.

 

I saw someone mention they were getting 400MB/s, I assume their using Bonding of some sort?  And even if you do that, you're still limited on the client side - unless you have dual nics there as well...right?

 

Trurl - right, sorry forgot about that. Would you expect to see differences in performance between BTRFS and ReiserFS?  Worth the effort to reformat and test that?

Link to comment

Thanks Gary, parity is at 85%, so should be able to test that shortly.  And yes, next test is the HBA.

 

I saw someone mention they were getting 400MB/s, I assume their using Bonding of some sort?  And even if you do that, you're still limited on the client side - unless you have dual nics there as well...right?

 

Trurl - right, sorry forgot about that. Would you expect to see differences in performance between BTRFS and ReiserFS?  Worth the effort to reformat and test that?

I wouldn't bother since v6 is at least as stable as v5 and ReiserFS is not the way forward.
Link to comment

Parity has completed and running tests now

 

Config

Parity Drive: 4TB Seagate ST4000DM000 (5900 64mb cache Sata III)

Data Drive: 2Tb Hitachi HDS5C3020ALA632 (7200 32mb cache Sata III)

Cache Drive: 750G Samsung 840 Pro SSD

 

Using the same parameters before...video file

 

1. Copy to Data Drive w/Cache Active

-- CrystalDiskMark 50MB - http://my.jetscreenshot.com/12412/20150420-wiqd-39kb

-- CrystalDiskMark 4000MB - http://my.jetscreenshot.com/12412/20150420-duxb-47kb

-- PC to UR6: 113MB/s

-- URt to PC: 102MB/s

 

2. Copy to Data Drive w/Cache Turned Off

-- CrystalDiskMark 50MB - http://my.jetscreenshot.com/12412/20150420-qhsp-47kb

-- CrystalDiskMark 4000MB - http://my.jetscreenshot.com/12412/20150420-mm4g-48kb

-- PC to UR6: 114MB/s

-- URt to PC: 107MB/s

 

So now I'm confused.  I expected that with the cache turned off (I stopped the array, unassigned the cache drive and started the array again - http://my.jetscreenshot.com/12412/20150420-t2ao-65kb).  I was expecting to see the 35mb/s speed garycase mentioned earlier.  I must be missing something...?  Perhaps not turned on something?  Parity is supposed active (see screenshot).

 

Then just for fun...

 

Multiple Use test (w/out cache drive active - WHILE running a parity check).

Ran CrystalDiskMark at 4000MB while copying video.

-- PC to UR6: 97MB/s

-- UR6 to PC: 97MB/s

 

 

 

 

 

Link to comment

Parity has completed and running tests now

 

Config

Parity Drive: 4TB Seagate ST4000DM000 (5900 64mb cache Sata III)

Data Drive: 2Tb Hitachi HDS5C3020ALA632 (7200 32mb cache Sata III)

Cache Drive: 750G Samsung 840 Pro SSD

 

Using the same parameters before...video file

 

1. Copy to Data Drive w/Cache Active

-- CrystalDiskMark 50MB - http://my.jetscreenshot.com/12412/20150420-wiqd-39kb

-- CrystalDiskMark 4000MB - http://my.jetscreenshot.com/12412/20150420-duxb-47kb

-- PC to UR6: 113MB/s

-- URt to PC: 102MB/s

 

2. Copy to Data Drive w/Cache Turned Off

-- CrystalDiskMark 50MB - http://my.jetscreenshot.com/12412/20150420-qhsp-47kb

-- CrystalDiskMark 4000MB - http://my.jetscreenshot.com/12412/20150420-mm4g-48kb

-- PC to UR6: 114MB/s

-- URt to PC: 107MB/s

 

So now I'm confused.  I expected that with the cache turned off (I stopped the array, unassigned the cache drive and started the array again - http://my.jetscreenshot.com/12412/20150420-t2ao-65kb).  I was expecting to see the 35mb/s speed garycase mentioned earlier.  I must be missing something...?  Perhaps not turned on something?  Parity is supposed active (see screenshot).

 

Then just for fun...

 

Multiple Use test (w/out cache drive active - WHILE running a parity check).

Ran CrystalDiskMark at 4000MB while copying video.

-- PC to UR6: 97MB/s

-- UR6 to PC: 97MB/s

 

Just to explain this, when you have a 3 disk configuration with 1 cache, 1 parity, and 1 data, your array is essentially a RAID1.  This gives you a false sense of performance.  The only way to really see write-speed performance to the array is with 1 parity and 2 or more data.

Link to comment
  • 2 weeks later...

Just an update.  Running many more disks.  Speeds are about the same.

 

Config: http://my.jetscreenshot.com/12412/20150508-ym6n-146kb

 

FYI, a parity check was in progress on a new parity drive during these tests.  And after I was all done, I happened to look at Mover, and it was running. 

 

Using the same parameters before...video file and CrystalDiskMark

 

1. I've made tests both through the cache (via a share 'use cache' and directly to the disk).  Performance through the cache is of course faster because cache is a 750gb SSD.

2. Disk 1 is formatted BTRFS, Disk4 is formatted XFS.  Disks in the system: http://my.jetscreenshot.com/12412/20150508-cf1h-117kb

3. CrystalDiskMark is made against a drive letter created using Windows File Explorer Map Network Drive against the Drive under the unRAID server in Network.

http://my.jetscreenshot.com/12412/20150508-qdvz-49kb.jpg

 

Copies are made to the following drives:

> Disk1 - this is the windows share under the server name in Windows Network (not a share i created).

> Disk1(BTRFS-SHARE) - this is a Share I created in the WebGUI

> Disk4 - this is the windows share under the server name in Windows Network (not a share i created).

> Disk4(XFS-SHARE) - this is a Share I created in the WebGUI

 

PC1 - Copy 5 +-1GB movie files.

PC2 - Copy 4 +-1GB movie files.

 

Results

=====

1. Copy to Data Drive DISK1 w/o using share/Cache (direct to Disk)

-- CrystalDiskMark 100MB - http://my.jetscreenshot.com/12412/20150508-ofuz-72kb.jpg

-- PC to UR6: Start at 114MB/s fell to 103MB/s (Direct to Disk1 (not the Cache share))

-- UR6 to PC: Start at 113MB/s fel to 102MB/s (Direct to Disk1 (not the Cache share))

-- simultaneous file xfer to same Share from two wired PC's both at 1GB/s Ethernet.

--->  PC1 to UR6 - Started at 80mb/s then slowly fell to 9.8mb/smb/s by end of xfer (5GB)

--->  PC2 to UR6 - Started at 60mb/s then slowly fell to 7mb/s by end of xfer (4GB), the other finished first, the second increase quickly to about 10mb/s (had another 700mb to go).

Interesting that it started fast, then slowed to 1/10th of the speed over the course of the xfer.  Anyone have an opinion on why that is?

-- singular copy same file xfer to same share, one at a time.

--->  PC1 to UR6 - Started at 30mb/s then slowly fell to 14.8mb/smb/s by end of xfer (5GB)

Interesting that it was way slower than the first test (103mb/s).  I watched the cache disk, and no files showed up on it.

--->  UR6 to PC1 - Started at 120 and finished at 102MB/s

--->  PC1 to UR6 - Started at 60 and finished at 30mb/s

Strange how its very different from copy to copy.

 

2. Copy to Data Drive DISK4 (XFS) w/o using share/Cache (direct to Disk)

-- CrystalDiskMark 100MB - http://my.jetscreenshot.com/12412/20150508-vg3h-79kb.jpg

-- PC to UR6: Start at 114MB/s fell to 102MB/s (Direct to Disk4)

-- UR6 to PC: Start at 117MB/s fell to 102MB/s (Direct to Disk4)

-- simultaneous file xfer to same Share from two wired PC's.

--->  PC1 to UR6 - Started at 102mb/s then slowly rose to 104mb/smb/s by end of xfer (5GB)

--->  PC2 to UR6 - Started at 29mb/s then slowly fell to 17mb/s then rose to 53mb/s by end of xfer (4GB), the other finished first, the second had another 1.8gb to go).

Interesting speeds are so different with XFS

-- singular same file xfer to same share, one at a time.

--->  PC1 to UR6 - Started at 102mb/s then slowly fell to 40mb/smb/s by end of xfer (5GB)

 

3. Copy to Data Drive Disk1(BTRFS) using the share on the cache

-- CrystalDiskMark 100MB -

-- PC to UR6: Start at 110MB/s and fell to 53MB/s (Disk1 via share/cache)

-- UR6 to PC: Start at 112MB/s fell to 102MB/s (Disk1 via share/cache)

-- Second copy - PC to UR6: Start at 110MB/s and fell to 107MB/s (Disk1 via share/cache)

-- Second copy - UR6 to PC: Start at 122MB/s fell to 98.6MB/s (Disk1 via share/cache)

 

-- simultaneous file xfer to same Share from two wired PC's.

--->  PC1 to UR6 - Started at 117mb/s then slowly rose to 104mb/smb/s by end of xfer (5GB)

--->  PC2 to UR6 - Started at 49mb/s then slowly fell to 17mb/s then rose to 53mb/s by end of xfer (4GB), the other finished first, the second had another 2.4gb to go).

-- singular same file xfer to same share, one at a time.

--->  PC1 to UR6 - Started at 120mb/s then slowly fell to 111mb/smb/s by 60%, the slowed to 80MB/s by end of xfer (5GB)

 

====

Just for Kicks I copied 5GB files from Disk1 to Disk2 - both disks are BTRFS.  Started at 60GB/s then slowly fell to 23MB/s.  So this was a Server Disk to Server disk copy through the NIC on my PC.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.