Dismal performance when writing with parity - SOLVED

[ender]

Hi all - brand new unRAID convert here. Forums and wiki have been very helpful in helping me decide on unRAID as I migrate away from WHS (mostly to get better density on the drives). I have a brand new build, trying to squeeze all I can out of existing hardware from the WHS box. Trying to avoid building a brand new system. Here are some specs, along with more details in the syslog.

 

FIC K8-800T motherboard, Athlon 64 Wolfdale 3200+, 2GB RAM. RealTeak Gigabit ethernet attached to PCI.

1X Maxtor DiamondMax IDE 250GB - attached to Primary IDE master

4X Samsung SpinPoint F2 1TB - attached to RocketRaid 1740 via PCI (used to run RAID 5 on here, but now set in JBOD mode). These are SATA2 drives connected to a SATA2 card.

2X Samsung SpinPoint T 500GB - attached to Promise SATA card via PCI. These are SATA2 drives that have been jumpered down to SATA1 since the card is SATA1.

1X WD Caviar Black 2TB - attached directly to motherboard SATA port. This drive is jumpered down to SATA1 since MB doesn't support SATA2.

 

 

(Yes, yes... I realize the performance hit I'm taking using the PCI bus, but wanted to make sure this set up was what I needed before investing more money. Hopefully it's not the source of my problems).

 

Here's the deal. I'm still waiting for program keys to arrive, so running unRAID basic right now. I've done some test copies just to get a sense of throughput. All copies are using a test .iso file, approx. 3GB. Throughput readings are being reported by Windows 7. Here are results:

 

Following results are with NO PARITY:

Set Samsung 1TB as Disk 1 device: approx. 31MB/s. Network usage chart via Windows 7 shows steady network utilization at ~35%.

Set Samsung 500GB as Disk 1 device: approx. 35MB/S. Network utilization fluctuates wildly between 25-80%, centered around 45%.

Set Maxtor 250GB as Disk 1 device: approx. 50MB/s. Network utilization fluctuates wildly between 50-90%, cetered around 75%.

 

These numbers aren't stellar, but I can live with them, especially since I'm asking a six year old PC to shove all that data through the PCI bus. I'm a bit surprised that the IDE drive outperformance the others - but no biggie. I plan to set that one as Cache later.

 

However, things get interesting when I add in parity:

 

Set Caviar Black 2TB as parity.

Set Samsun 1TB as Disk 1 device.

Write speeds plummet to approx. 1.1MB/s. Network utilization barely registers.

 

I didn't test any of the other drives as Disk 1 device since none of them are zeroed.

 

This is awful! I guess my plan was to put the fastest drive as parity and have it on the native SATA bus. Have absolutely no idea why write speeds are so low now.

HELP!

 

Sidenote: I'm zeroing all the drives now, and doing a parity check. The 4X Samsung drives connected to the RocketRaid card seem to share bandwidth nicely, each is at approx. 6.7MB/s, current parity sync is at 6.6MB/s. The 2 Samsung drives connected to the Promise card (came with a Maxtor drive) aren't playing so nice. One is at 42MB/s and the other is at 9.2MB/s. The smallest drive connected to IDE is already done, averaged between 30-50MB/S. (See attached screenshot).

 

Since I now have all of my drives running, is this an indication of the type of performance I can expect going forward? I feel like I must be doing something wrong, since non-parity writes were between 50-30MB/s, and then parity dropped it down to 1MB/s. As I mentioned, I will most likely put the IDE drive as cache when my registration key comes in, so slowER parity writes in the 15-20MB/s range won't bother me. I knew that was a risk using such old hardware. BUT, at 1MB/s, it would take 13 hours to write a Blu-ray .iso file. At current speeds, it's taking 80+ hours to run a parity check on *1* data drive. What happens when I put all the drives online?

 

Thanks in advance for your help.

syslog.zip

[Joe]

I would suggest moving the drives around to different controllers and testing some more.  I got really bad performance with 3ware controllers and when moved to my motherboard worked great.  I think it was a Linux driver issue with the card as they were fine with Windows.

 

Joe

[Joe L.]

The bulk of your errors are affiliated with 1 drive:

Dec 10 01:28:17 Hurricane kernel: ata5: SATA link up 1.5 Gbps (SStatus 113 SControl 300)

Dec 10 01:28:17 Hurricane kernel: ata5.00: ATA-8: SAMSUNG HD501LJ, CR100-10, max UDMA7

Dec 10 01:28:17 Hurricane kernel: ata5.00: 976773168 sectors, multi 0: LBA48 NCQ (depth 0/32)

Dec 10 01:28:17 Hurricane kernel: ata5.00: configured for UDMA/133

Dec 10 01:28:17 Hurricane kernel: scsi 5:0:0:0: Direct-Access    ATA      SAMSUNG HD501LJ  CR10 PQ: 0 ANSI: 5

Dec 10 01:28:17 Hurricane kernel: sd 5:0:0:0: [sde] 976773168 512-byte logical blocks: (500 GB/465 GiB)

Dec 10 01:28:17 Hurricane kernel: sd 5:0:0:0: [sde] Write Protect is off

Dec 10 01:28:17 Hurricane kernel: sd 5:0:0:0: [sde] Mode Sense: 00 3a 00 00

Dec 10 01:28:17 Hurricane kernel: sd 5:0:0:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

Dec 10 01:28:17 Hurricane kernel:  sde: sde1

 

The slowness is probably not helped by the PIO mode choosen for your one disk on the IDE bus.

Dec 10 01:28:17 Hurricane kernel: Probing IDE interface ide0...

Dec 10 01:28:17 Hurricane kernel: hda: Maxtor 6B250R0, ATA DISK drive

Dec 10 01:28:17 Hurricane kernel: hda: host max PIO5 wanted PIO255(auto-tune) selected PIO4

Are you using a 80 conductor flat IDE cable to connect to that drive?  Round cables often have VERY terrible performance.  Older 40 conductor cables can't handle the higher speeds needed by modern disks.

 

The major errors seem to be with /dev/sde

Dec 10 01:35:10 Hurricane kernel: ata5.00: exception Emask 0x10 SAct 0x0 SErr 0x100 action 0x6

Dec 10 01:35:10 Hurricane kernel: ata5.00: port_status 0x20200000

Dec 10 01:35:10 Hurricane kernel: ata5: SError: { UnrecovData }

Dec 10 01:35:10 Hurricane kernel: ata5.00: failed command: READ DMA

Dec 10 01:35:10 Hurricane kernel: ata5.00: cmd c8/00:00:e0:01:02/00:00:00:00:00/e0 tag 0 dma 131072 in

Dec 10 01:35:10 Hurricane kernel:          res 51/0c:ed:f4:01:02/00:00:00:00:00/e0 Emask 0x10 (ATA bus error)

Dec 10 01:35:10 Hurricane kernel: ata5.00: status: { DRDY ERR }

Dec 10 01:35:10 Hurricane kernel: ata5.00: error: { ABRT }

Dec 10 01:35:10 Hurricane kernel: ata5: hard resetting link

Dec 10 01:35:11 Hurricane kernel: ata5: SATA link up 1.5 Gbps (SStatus 113 SControl 300)

Dec 10 01:35:11 Hurricane kernel: ata5.00: configured for UDMA/133

Dec 10 01:35:11 Hurricane kernel: ata5: EH complete

Dec 10 01:35:15 Hurricane kernel: ata5.00: exception Emask 0x10 SAct 0x0 SErr 0x100 action 0x6

Dec 10 01:35:15 Hurricane kernel: ata5.00: port_status 0x20200000

Dec 10 01:35:15 Hurricane kernel: ata5: SError: { UnrecovData }

Dec 10 01:35:15 Hurricane kernel: ata5.00: failed command: READ DMA

Dec 10 01:35:15 Hurricane kernel: ata5.00: cmd c8/00:00:e0:01:02/00:00:00:00:00/e0 tag 0 dma 131072 in

Dec 10 01:35:15 Hurricane kernel:          res 51/0c:ed:f4:01:02/00:00:00:00:00/e0 Emask 0x10 (ATA bus error)

Dec 10 01:35:15 Hurricane kernel: ata5.00: status: { DRDY ERR }

Dec 10 01:35:15 Hurricane kernel: ata5.00: error: { ABRT }

Dec 10 01:35:15 Hurricane kernel: ata5: hard resetting link

Dec 10 01:35:15 Hurricane kernel: ata5: SATA link up 1.5 Gbps (SStatus 113 SControl 300)

Dec 10 01:35:15 Hurricane kernel: ata5.00: configured for UDMA/133

Dec 10 01:35:15 Hurricane kernel: ata5: EH complete

Dec 10 01:35:19 Hurricane kernel: ata5.00: exception Emask 0x10 SAct 0x0 SErr 0x100 action 0x6

Dec 10 01:35:19 Hurricane kernel: ata5.00: port_status 0x20200000

Dec 10 01:35:19 Hurricane kernel: ata5: SError: { UnrecovData }

Dec 10 01:35:19 Hurricane kernel: ata5.00: failed command: READ DMA

Dec 10 01:35:19 Hurricane kernel: ata5.00: cmd c8/00:00:e0:01:02/00:00:00:00:00/e0 tag 0 dma 131072 in

Dec 10 01:35:19 Hurricane kernel:          res 51/0c:ed:f4:01:02/00:00:00:00:00/e0 Emask 0x10 (ATA bus error)

Dec 10 01:35:19 Hurricane kernel: ata5.00: status: { DRDY ERR }

Dec 10 01:35:19 Hurricane kernel: ata5.00: error: { ABRT }

Dec 10 01:35:19 Hurricane kernel: ata5: hard resetting link

Dec 10 01:35:19 Hurricane kernel: ata5: SATA link up 1.5 Gbps (SStatus 113 SControl 300)

Dec 10 01:35:19 Hurricane kernel: ata5.00: configured for UDMA/133

Dec 10 01:35:19 Hurricane kernel: ata5: EH complete

Dec 10 01:35:23 Hurricane kernel: ata5: limiting SATA link speed to 1.5 Gbps

Dec 10 01:35:23 Hurricane kernel: ata5.00: exception Emask 0x10 SAct 0x0 SErr 0x100 action 0x6

Dec 10 01:35:23 Hurricane kernel: ata5.00: port_status 0x20200000

Dec 10 01:35:23 Hurricane kernel: ata5: SError: { UnrecovData }

Dec 10 01:35:23 Hurricane kernel: ata5.00: failed command: READ DMA

Dec 10 01:35:23 Hurricane kernel: ata5.00: cmd c8/00:00:e0:01:02/00:00:00:00:00/e0 tag 0 dma 131072 in

Dec 10 01:35:23 Hurricane kernel:          res 51/0c:ed:f4:01:02/00:00:00:00:00/e0 Emask 0x10 (ATA bus error)

Dec 10 01:35:23 Hurricane kernel: ata5.00: status: { DRDY ERR }

Dec 10 01:35:23 Hurricane kernel: ata5.00: error: { ABRT }

Dec 10 01:35:23 Hurricane kernel: ata5: hard resetting link

Dec 10 01:35:23 Hurricane kernel: ata5: SATA link up 1.5 Gbps (SStatus 113 SControl 300)

Dec 10 01:35:23 Hurricane kernel: ata5.00: configured for UDMA/133

Dec 10 01:35:23 Hurricane kernel: ata5: EH complete

Dec 10 01:35:26 Hurricane kernel: ata5.00: exception Emask 0x10 SAct 0x0 SErr 0x100 action 0x6

Dec 10 01:35:26 Hurricane kernel: ata5.00: port_status 0x20200000

Dec 10 01:35:26 Hurricane kernel: ata5: SError: { UnrecovData }

Dec 10 01:35:26 Hurricane kernel: ata5.00: failed command: READ DMA

Dec 10 01:35:26 Hurricane kernel: ata5.00: cmd c8/00:00:e0:01:02/00:00:00:00:00/e0 tag 0 dma 131072 in

Dec 10 01:35:26 Hurricane kernel:          res 51/0c:ed:f4:01:02/00:00:00:00:00/e0 Emask 0x10 (ATA bus error)

Dec 10 01:35:26 Hurricane kernel: ata5.00: status: { DRDY ERR }

Dec 10 01:35:26 Hurricane kernel: ata5.00: error: { ABRT }

Dec 10 01:35:26 Hurricane kernel: ata5: hard resetting link

Dec 10 01:35:27 Hurricane kernel: ata5: SATA link up 1.5 Gbps (SStatus 113 SControl 300)

Dec 10 01:35:27 Hurricane kernel: ata5.00: configured for UDMA/133

Dec 10 01:35:27 Hurricane kernel: ata5: EH complete

Dec 10 01:35:30 Hurricane kernel: ata5.00: exception Emask 0x10 SAct 0x0 SErr 0x100 action 0x6

Dec 10 01:35:30 Hurricane kernel: ata5.00: port_status 0x20200000

Dec 10 01:35:30 Hurricane kernel: ata5: SError: { UnrecovData }

Dec 10 01:35:30 Hurricane kernel: ata5.00: failed command: READ DMA

Dec 10 01:35:30 Hurricane kernel: ata5.00: cmd c8/00:00:e0:01:02/00:00:00:00:00/e0 tag 0 dma 131072 in

Dec 10 01:35:30 Hurricane kernel:          res 51/0c:ed:f4:01:02/00:00:00:00:00/e0 Emask 0x10 (ATA bus error)

Dec 10 01:35:30 Hurricane kernel: ata5.00: status: { DRDY ERR }

Dec 10 01:35:30 Hurricane kernel: ata5.00: error: { ABRT }

Dec 10 01:35:30 Hurricane kernel: ata5: hard resetting link

Dec 10 01:35:30 Hurricane kernel: ata5: SATA link up 1.5 Gbps (SStatus 113 SControl 300)

Dec 10 01:35:30 Hurricane kernel: ata5.00: configured for UDMA/133

Dec 10 01:35:30 Hurricane kernel: sd 5:0:0:0: [sde] Result: hostbyte=0x00 driverbyte=0x08

Dec 10 01:35:30 Hurricane kernel: sd 5:0:0:0: [sde] Sense Key : 0xb [current] [descriptor]

Dec 10 01:35:30 Hurricane kernel: Descriptor sense data with sense descriptors (in hex):

Dec 10 01:35:30 Hurricane kernel:        72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00

Dec 10 01:35:30 Hurricane kernel:        00 02 01 f4

Dec 10 01:35:30 Hurricane kernel: sd 5:0:0:0: [sde] ASC=0x0 ASCQ=0x0

Dec 10 01:35:30 Hurricane kernel: sd 5:0:0:0: [sde] CDB: cdb[0]=0x28: 28 00 00 02 01 e0 00 01 00 00

Dec 10 01:35:30 Hurricane kernel: end_request: I/O error, dev sde, sector 131552

Dec 10 01:35:30 Hurricane kernel: Buffer I/O error on device sde, logical block 16444

Dec 10 01:35:30 Hurricane kernel: Buffer I/O error on device sde, logical block 16445

Dec 10 01:35:30 Hurricane kernel: Buffer I/O error on device sde, logical block 16446

Dec 10 01:35:30 Hurricane kernel: Buffer I/O error on device sde, logical block 16447

Dec 10 01:35:30 Hurricane kernel: Buffer I/O error on device sde, logical block 16448

Dec 10 01:35:30 Hurricane kernel: Buffer I/O error on device sde, logical block 16449

Dec 10 01:35:30 Hurricane kernel: Buffer I/O error on device sde, logical block 16450

Dec 10 01:35:30 Hurricane kernel: Buffer I/O error on device sde, logical block 16451

Dec 10 01:35:30 Hurricane kernel: Buffer I/O error on device sde, logical block 16452

Dec 10 01:35:30 Hurricane kernel: Buffer I/O error on device sde, logical block 16453

Dec 10 01:35:30 Hurricane kernel: ata5: EH complete

 

The poor linux kernel and device driver keep trying to reset it to allow it to be read.  You've got 8 disks in there.  Probably needs a power supply capable of at least 30 Amps on the 12 volt rail.  Is your power supply capable?  Is it a single rail supply?

 

Joe L.

[ender]

I would suggest moving the drives around to different controllers and testing some more.  I got really bad performance with 3ware controllers and when moved to my motherboard worked great.  I think it was a Linux driver issue with the card as they were fine with Windows.

 

Joe

 

Hey Joe - I don't have much choice, the MB only has 2 SATA connectors... that's why I'm cobbling together with PCI cards now.

[ender]

Thanks Joe. I've been around computers a ton, but a relative newbie when it comes to Linux and system outputs. Please forgive the dumb questions...

 

The bulk of your errors are affiliated with 1 drive:

Dec 10 01:28:17 Hurricane kernel: ata5: SATA link up 1.5 Gbps (SStatus 113 SControl 300)

Dec 10 01:28:17 Hurricane kernel: ata5.00: ATA-8: SAMSUNG HD501LJ, CR100-10, max UDMA7

Dec 10 01:28:17 Hurricane kernel: ata5.00: 976773168 sectors, multi 0: LBA48 NCQ (depth 0/32)

Dec 10 01:28:17 Hurricane kernel: ata5.00: configured for UDMA/133

Dec 10 01:28:17 Hurricane kernel: scsi 5:0:0:0: Direct-Access    ATA      SAMSUNG HD501LJ  CR10 PQ: 0 ANSI: 5

Dec 10 01:28:17 Hurricane kernel: sd 5:0:0:0: [sde] 976773168 512-byte logical blocks: (500 GB/465 GiB)

Dec 10 01:28:17 Hurricane kernel: sd 5:0:0:0: [sde] Write Protect is off

Dec 10 01:28:17 Hurricane kernel: sd 5:0:0:0: [sde] Mode Sense: 00 3a 00 00

Dec 10 01:28:17 Hurricane kernel: sd 5:0:0:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

Dec 10 01:28:17 Hurricane kernel:  sde: sde1

 

Can you tell errors from the above, or are you just identifying the drive for me? I see the errors later in the log, but don't know whether this is something here that tells you the drive is in trouble.

 

The slowness is probably not helped by the PIO mode choosen for your one disk on the IDE bus.

Dec 10 01:28:17 Hurricane kernel: Probing IDE interface ide0...

Dec 10 01:28:17 Hurricane kernel: hda: Maxtor 6B250R0, ATA DISK drive

Dec 10 01:28:17 Hurricane kernel: hda: host max PIO5 wanted PIO255(auto-tune) selected PIO4

Are you using a 80 conductor flat IDE cable to connect to that drive?  Round cables often have VERY terrible performance.  Older 40 conductor cables can't handle the higher speeds needed by modern disks.

 

Definitely not a round cable. I'll have to go home and check on whether it's 40 or 80 conductor. I'll also check to see that the positioning of the connector is correct (that it's not on the slave connector). When the guts of the computer was put together years ago, I didn't pay much attention to the cables, as I/O was definitely NOT the limiting factor.

 

In any case - this drive was still the fastest out of all three combinations in the single drive write test. Thoughts?!?

 

The major errors seem to be with /dev/sde

--snip--

 

The poor linux kernel and device driver keep trying to reset it to allow it to be read. 

 

Is this indicative of drive failure or incorrect setup? The drive was working fine before I pulled it out of the other box. But, it never did get much of an exercise in it's old home. This particular drive is on pre-clear. I guess the SMART output will be telling.

 

Also - at one point I did an initconfig before stopping the array. Don't recall if it was with this drive -- could that be the cause of the issues?

 

You've got 8 disks in there.  Probably needs a power supply capable of at least 30 Amps on the 12 volt rail.  Is your power supply capable?  Is it a single rail supply?

 

Joe L.

 

Yes - I'll have to double check the specs again when I get home, but one of the upgrades I did make was the power supply. 650W with a single 12V rail - I believe at 38 amps.

 

 

These items aside - the two drives with apparent issues were not the ones I used when I did the parity write test. It was one of the Samsung 1TB drives (/dev/sdd). No other drives were assigned when I did that test.

 

What else could be wrong?

[ender]

Ok - after much futzing around, I've found two things:

 

1. One of the hard drives was bad. Since removed.

2. The FIC K8-800T motherboard on-board SATA ports DO NOT WORK. Kept getting parity errors (1000+ every time, even when all the drives were pre-cleared with nothing on them). Moved parity drive to the promise card, and no parity errors since.

 

This is MB in question: http://www.fic.com.tw/product/motherboard/AMD/k8-800t.aspx

[ender]

Does someone know who to put this into the hardware compatibility wiki so others can be warned?

[prostuff1]

Does someone know who to put this into the hardware compatibility wiki so others can be warned?

 

You can create a free account in the wiki and add it yourself.