How to check assigned PCIe Lanes in OS?

I have 4 Adaptec 1430SA PCI Express (x4 PCIe lane connection) RAID cards in my unRAID system, each plugged into a x16 sized PCIe slot.  While the motherboard has 4 physical x16 slots, it only supports quad x8 lanes when all four slots are in use.  The motherboard is an ASUS M4A79T Deluxe, running the AMD 790FX chipset.

The problem I'm having is that HD's attached to the cards in slots 1 and 3 perform half as fast (50-65 MB/s) as the HD's attached to the cards in slots 2 and 4 (105-125 MB/s).  I've ruled out the HD's themselves by moving them around.  I've tinkered with the BIOS settings that control the allocation of lanes among the PCIe slot pairs (slots 1 & 2 are paired, as are 3 & 4).  None of my tinkering has affected performance. 

My hunch is that the BIOS is setting PCIe slots 1 and 3 to x1 (one lane only), and allowing slots 2 & 4 the full x4 lane allotment that the Adeptec 1430SA supports.  I would like to confirm this hunch, but don't know how to check in Linux.  Is there anyway I can query how many lanes are enabled on each PCIe slot? I've already installed the PCIUTILS 2.2, but it doesn't give lane allocation info.  I also checked the syslog, and found no lane assignment info there either.

Below is information I compiled during troubleshooting.  I've listed the disk performance for every drive (including the flash drive), grouped by the connection.  The oddest part to me is that the slow drives are connected to controllers in the blue PCIe slots, which the manual identifies as the primary, and the fast drives are in the black slots which the manual calls secondary.  When the BIOS allocates lanes, based upon the manual the blue primary slots should get equal or more lanes than the black secondary slots.  The motherboard is an enthusiast motherboard that supports quadfire (dual crossfire) graphics cards, and as such I doubt that the manufacturer optimizes it for use in an unRAID server running SATA cards in place of graphics cards.

If I can at least confirm lane misallocation, I can go back to ASUS and demand a fixed BIOS to properly allocate lanes.  Thanks for any help!  -Paul

USB:
/dev/sda:  24.96 MB/sec (Flash Drive)

Motherboard SATA (pci-0000:00:11.0):
/dev/sdo:  97.46 MB/sec (2.0 GB, 5400RPM) Parity Drive

Adaptec 1430SA (pci-0000:02:00.0) (IRQ 19) (in PCIe Slot 3 - Blue Primary):
/dev/sdk:  55.14 MB/sec (1.0 GB, 7200RPM)
/dev/sdl:  63.04 MB/sec (1.0 GB, 7200RPM)
/dev/sdm:  55.15 MB/sec (1.0 GB, 7200RPM)
/dev/sdn:  55.68 MB/sec (1.0 GB, 7200RPM)

Adaptec 1430SA (pci-0000:03:00.0) (IRQ 18) (in PCIe Slot 4 - Black Secondary):
/dev/sdg: 119.59 MB/sec (1.0 GB, 7200RPM)
/dev/sdh: 123.21 MB/sec (1.0 GB, 7200RPM)
/dev/sdi: 106.62 MB/sec (1.5 GB, 5400RPM)
/dev/sdj: 105.87 MB/sec (1.5 GB, 5400RPM)

Adaptec 1430SA (pci-0000:05:00.0) (IRQ 17) (in PCIe Slot 2 - Black Secondary):
/dev/sdc: 112.92 MB/sec (1.5 GB, 5400RPM)
/dev/sdd: 107.03 MB/sec (1.5 GB, 5400RPM)
/dev/sde: 106.99 MB/sec (1.5 GB, 5400RPM)
/dev/sdf: 107.47 MB/sec (1.5 GB, 5400RPM)

Adaptec 1430SA (pci-0000:06:00.0) (IRQ 18) (in PCIe Slot 1 - Blue Primary):
/dev/sdb:  50.65 MB/sec (1.5 GB, 5400RPM)

That's really odd. A x1 lane should support a drive at full speed and you list the one card with a single drive as being slow. You should not begin to see a slowdown until you try to access more than 2 drives at once.

Peter

Agreed.  Each PCIe lane should support 250 MB/s, I wouldn't expect a slowdown until 3 drives are accessed simultaneously on the same controller. 

Maybe I'm barking up the wrong tree here, but  I would still love a Linux command that would show PCIe lane width.  I just tried a bunch of parameters with the lspci command, but none gave that detail.

I'm also willing to load up another distro (or even Windows) on the hardware, if that is what it takes to extract the current pCIe lane widths. Open to suggestions.

According to the manual,

If you install two graphics cards to the blue slots and use other X4 PCIe cards simultaneously, the X4 PCIe cards will downgrade to x1 mode for optimum VGA performance, You may re-assign bus width in BIOS Settings, Refer to page 3.22 for details.

Even seeing the chart shows that wider cards would set the other cards to run at x1.

Judging from your experiments, certain slots are running at x1.. or even less.  Only question I have "is this for parallel access?"

I would think that a single drive access, would be at full speed.

I'm not sure how the Drive Speed Test using hdparm -tT works, does it test each drive one by one, or all drives simultaneously?

I find it interesting that I have two controllers giving 50% performance, one with only one drive attached, and one with four drives.  The speed test has remained consistent as the array has grown, meaning it won't get slower if I add three more drives to the last controller, just that they too will run at 50%.  If the Drive Speed Test only tests drives one by one, that could explain the consistency.

The reference to page 3-22 is referring to information that is actually on page 3-21 of the manual.  I have tried both values for the GFX Dual Slot Configuration.

GFX Dual Slot Configuration [Auto] is default:

    [Auto]    Switches the PCIe lane of the PCIEX16_1 and PCIEX16_2 slots automatically.

    [Enable]  Sets the PCIe lane of the PCIEX16_1 and PCIEX16_2 slots to x8/x8 or x16/x1 links.

[Auto] sounds like it would automatically figure out how many lanes are needed, and I would expect it would configure as x4/x4 or x8/x8.  [Enable] is confusing, as it doesn't tell you how it will set the mode to x8/x8 or x16/x1, so this might as well be [Auto] too.  What I need is a [Manual] or [x8/x8] setting, so I can set it correctly.  Regardless, I still need to determine what the end result of the values are.  It is quite possible that both [Auto] and [Enable] are resulting in x16/x1.  I just need a way to check and see that is what is happening.

If on the other hand I can establish that the lanes are being set to x8/x8 or x4/x4, I can start looking elsewhere for the issue.  That's why I'm looking for a tool/command to query what the lane widths are set to.

Another possibility is that during boot up, only two of the four Adaptec cards go through a BIOS routine.  I had to configure the BIOS of the cards two at a time, removing the other two from the system and then swapping them to configure the other two.  This is a know feature of the cards, and is supposed to be inconsequential, as the drivers are loaded for all cards during the OS boot.  Perhaps this BIOS routine is somehow affecting performance, even though I have it disabled.

I just double-checked, and during boot the two Adaptec cards that go through the BIOS routine are pci-0000:02:00.0 (a slow controller) and pci-0000:03:00.0 (a fast controller).  That kinda rules out the BIOS routine from impacting performance, since it doesn't directly correlate to the fast or slow controllers.

I'm not sure how the Drive Speed Test using hdparm -tT works, does it test each drive one by one, or all drives simultaneously?

One by one.

There is still something wrong.

What happens if you only have the adaptec cards in the "slow" slots?

What if you have 3 cards?

Even a x1 port will yeild 250MB/s and if you have a 1TB or a modern drive on an x1 you should get 70-110Mb/s depending on drive model.

For example, on a PCIe x1 - SIL3132 - I get 77-100Mb/s on two drives simultaneously.

at 50Mb/s it almost seems as if you are on some form of port multiplier.

With my port multipliers, I could not get faster then 60-70Mb/s no matter what model of drive I used.

have you tried two/three card setups?

Curious to see what happens with two cards in blue, black and also one in each. Also mb supports 3 way xfire so trying three cards is a possibility.

Have you tried Adaptec? Ive never had much success wth them but they might be able to help.

Can you change nb to sb lane allocation on that board, dropping to 8-8 or 4-4 might free some lanes?

Have you tried moving controllers?

I'm willing to try two/three card configurations, but not at the expense of my array.  Can I remove a controller/4 drives from my system that are part of the array, boot up in a non-functioning mode (being sure NOT to hit repair/restore/format), test the drive speed, then reinstall the card and boot normally?  I'm scared I could corrupt my array by testing with active cards and drives removed.

I haven't tried 2/3 card setups to this point, as I built the server from the beginning to support 20 drives (the last 4 drives are on the motherboard), and everything was installed from the get-go.  I didn't discover the speed issue until later, as I continued to expand the array with new drives.

I haven't tried contacting Adaptec yet.  My gut instinct is telling me this is a MB issue, so I would probably contact ASUS first.  That's the main reason I want a tool that tells me the lane width, so I can provide ASUS (or Adaptec) with some hard data.

I don't have any controls for NB to SB lane allocation.  On the AMD platform these communications travel over the HyperTransport link.

I have moved controllers around, back in the beginning when I first built the server.  Haven't tried that since I noticed the performance issue.

Yeah, got confused with ht lanes and pci-e lanes. Apologies.

I made progress in analyzing the performance issue, and thought I would share my troubleshooting steps and results - hopefully they can help others.

I wasn't able to find a method to determine PCIe lane allocation in the unRAID linux distribution.  I was also concerned about corrupting my array via testing with various cards removed.  To sidestep both issues, I installed Windows on a spare drive attached to the motherboard.  This allowed me to use Windows based tools to analyze the issue, and remove cards without fear of unRAID seeing the activity and invalidating the array.  I will return hardware back to the original configuration before allowing the system to boot off the unRAID flash drive again.

I then installed SiSoftware Sandra Lite, which is a fantastic suite of free software tools that can benchmark your system and tell you all kinds of nitty gritty detail.  Sure enough, the information on PCI lane allocation was easy to find.  Please note, I only browsed the hardware information.  I did not run any benchmarks or attempt to access the unRAID drives in any way, for fear of corrupting unRAID data.  I don't care about Windows performance anyway, only what PCIe lane width was being assigned to the RAID controller cards.

I then set about testing every combination of BIOS settings to see how they affected lane allocation.  No matter what I did, I got this:

Bus 2 - x4

Bus 3 - x1

Bus 4 - x1

Bus 5 - x4

The only discrepancy I found was that in Linux, buses 4 and 5 are actually reported as 5 and 6.  That small discrepancy aside, I was able to directly correlate the fast drives to the controllers on buses 2 and 5(aka 6), and the slow drives to the controllers on buses 3 and 4(aka 5).

I was also able to see that the Adaptec 1430SA was reporting x4 capability in spite of the motherboard/BIOS assigning only x1 to the card.

This testing does not illustrate why I am seeing performance degradation on an x1 link with only 1 drive attached, only that it is related to the issue.  Perhaps that is just an idiosyncrasy of the Adaptec 1430SA.  I am hopeful that simply establishing an x4 connection will resolve the performance gap.

I also tried pulling a controller card that was operating at x4, and this allowed the sister card that was operating at x1 to speed up to x4.

Since it seems the hardware is fully capable of running all cards at x4, the limiting factor is the motherboard BIOS.  I will be contacting ASUS for resolution.

Great job!  Let us know how you make out with ASUS.

As expected, contacting customer service and explaining the issue was challenging.  At this point my case is logged and is submitted to the testing team.  I hope to hear back this year.

The previous generation motherboard to my M4A79 Deluxe is the M3A79 Deluxe, and beside a slightly older chipset it has one notable difference: the BIOS allows you to manually allocate PCIe lanes.  I stumbled upon this feature during my research online today, and I pointed this out to ASUS in hope that they would incorporate those BIOS parameters into the newer board.

As luck would have it, I actually have the M3A79 Deluxe installed in my HTPC.  I booted up into the BIOS to double check the parameters, and found I had to hit the secret key (F4) to make them appear.  Since these two motherboards are basically the same, I plan to swap them this weekend.  The newer M4A79 is actually better suited to my HTPC anyway, so this seems a logical choice.