bungee91 Posted November 30, 2015 Share Posted November 30, 2015 Edit: So okay, I overlooked "PCH" on this workaround, and clearly see the proper entries for the processor root ports (in attachment of complete output of lspci -vvv).. However what's the workaround for the PCH ports doing? Is isolation not real for these ports/peer to peer DMA between devices from the PCH? This is likely directed more towards our guru here JonP, however if anyone else has the answer, well then fantastic! I see this message in my syslog of some new hardware I'm testing. I'm hoping to make this my new server components, however want to test it thoroughly while within my return window. i7-5930k and X99 Gigabyte SLI MB. As per the Vfio mailing list, and vfio blog, the Intel High End Desktop Processors have recently been found to have PCIe Access Control Services (ACS) support on the processor root ports (awesome!). I am seeing proper IOMMU isolation, however dmesg | grep ACS outputs ACPI: FACS 0x0000000036740F80 000040 pci 0000:00:1c.0: Intel PCH root port ACS workaround enabled pci 0000:00:1c.6: Intel PCH root port ACS workaround enabled pci 0000:00:1c.7: Intel PCH root port ACS workaround enabled I wouldn't think any "workaround" should be needed, isolation should be there and working however maybe not. Output of sudo lspci -vvv 00:1c.0 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #1 (rev d5) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Bus: primary=00, secondary=05, subordinate=05, sec-latency=0 I/O behind bridge: 0000f000-00000fff Memory behind bridge: fff00000-000fffff Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Root Port (Slot-), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #1, Speed 5GT/s, Width x2, ASPM L0s L1, Latency L0 <1us, L1 <4us ClockPM- Surprise- LLActRep+ BwNot+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+, OBFF Via WAKE# ARIFwd- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit- Address: fee00318 Data: 0000 Capabilities: [90] Subsystem: Gigabyte Technology Co., Ltd Device 7270 Capabilities: [a0] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Kernel driver in use: pcieport 00:1c.6 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #7 (rev d5) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Bus: primary=00, secondary=06, subordinate=06, sec-latency=0 I/O behind bridge: 0000b000-0000bfff Memory behind bridge: fb300000-fb3fffff Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #7, Speed 5GT/s, Width x1, ASPM unknown, Latency L0 <512ns, L1 <4us ClockPM- Surprise- LLActRep+ BwNot+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #6, PowerLimit 10.000W; Interlock- NoCompl+ SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+, OBFF Not Supported ARIFwd- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit- Address: fee00358 Data: 0000 Capabilities: [90] Subsystem: Gigabyte Technology Co., Ltd Device 7270 Capabilities: [a0] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Kernel driver in use: pcieport 00:1c.7 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #8 (rev d5) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Bus: primary=00, secondary=07, subordinate=07, sec-latency=0 I/O behind bridge: 0000a000-0000afff Memory behind bridge: fb200000-fb2fffff Prefetchable memory behind bridge: 00000000d0000000-00000000dfffffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #8, Speed 5GT/s, Width x1, ASPM unknown, Latency L0 <512ns, L1 <4us ClockPM- Surprise- LLActRep+ BwNot+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #7, PowerLimit 10.000W; Interlock- NoCompl+ SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet- LinkState- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+, OBFF Not Supported ARIFwd- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit- Address: fee00398 Data: 0000 Capabilities: [90] Subsystem: Gigabyte Technology Co., Ltd Device 7270 Capabilities: [a0] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Kernel driver in use: pcieport It looks as if I should see something like below, however I DO NOT see this (as can be seen above), so is the "workaround" disabling this proper isolation with ACS on root ports? Or is the MB not properly supporting it? Capabilities: [110 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- tower-syslog-20151130-1639.zip Quote Link to comment
bungee91 Posted November 30, 2015 Author Share Posted November 30, 2015 Maybe I am looking into this too much, as I see the correct ACS entries for the PCI bridges. So what's the "workaround" all about listed on the dmesg | grep ACS output for ACPI: FACS 0x0000000036740F80 000040 pci 0000:00:1c.0: Intel PCH root port ACS workaround enabled pci 0000:00:1c.6: Intel PCH root port ACS workaround enabled pci 0000:00:1c.7: Intel PCH root port ACS workaround enabled Attached is the full lspci -vvv output. lspci_-vvv.txt Quote Link to comment
bungee91 Posted November 30, 2015 Author Share Posted November 30, 2015 (As I research I will just keep talking..... plus I had bad assumptions in my 1st post as those are clearly PCH root ports, not processor) Ok, so 1.0, 2.0 & 3.0 are PCI bridges from the CPU and they support proper ACS on root ports. 1c.0, 1c.6, & 1c.7 are from the chipset/PCH and do NOT support ACS, however the workaround does something to make it all good (grouping or something like that)... Right? All of these 16X (physical, not necessarily wired) are wired to the CPU, the 1X slots, audio, etc,, all come from the PCH over DMI 2.0. Which is confusing to me for isolation and ACS being properly supported... Anyhow I loaded every slot I could (that's what she said! ) for testing, and all devices are properly in their own IOMMU group, and work well. If someone could take me to school on this/confirm my thoughts, it'd be appreciated. Quote Link to comment
bhinkle50 Posted December 2, 2015 Share Posted December 2, 2015 I'm tagging along as I am in process of my ASRock X99 build with an E5 Xeon. Still waiting for a few components to come in, but should have mine fired up next week. Curious to see what the answer is to your question. Quote Link to comment
bungee91 Posted December 3, 2015 Author Share Posted December 3, 2015 Received a response from the VFIO mailing list (JonP doesn't seem to like me anymore... :'( ) Anyhow everything is good to go, isolation is very much intact and functioning as it should... Now to sell my old equipment! Response from Alex Williamson: The kernel log indicates that ACS-equivalent isolation has been enabled for those PCH root ports: http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=d99321b63b1f2cd386cdf20aefe4d8d3d414cb5b For more on IOMMU groups, see: http://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html Quote Link to comment
jonp Posted December 4, 2015 Share Posted December 4, 2015 Sorry buddy, I've been crazy busy these last several weeks. Some pretty cool stuff happening very soon! Glad Alex got back to you with the proper info. Quote Link to comment
bungee91 Posted December 4, 2015 Author Share Posted December 4, 2015 Sorry buddy, I've been crazy busy these last several weeks. Some pretty cool stuff happening very soon! Glad Alex got back to you with the proper info. Stop teasing us , but I get it, and look forward to the release. Now to build my 4 headed monster (literally if GPU's are heads).. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.