PCIe ACS Issue, QEMU Config

I recently built a new system for unRAID so I could virtualize a couple systems and ran into an issue with using pass-through on certain pcie slots. I think I figured out a solution and thought someone else might find it useful.

 

CPU: E5-1650v3

Mobo: Supermicro X10SRH-CLN4F

GPU: EVGA GT 730

HBA: LSI 9211-8i (M1015)

 

unRAID migration from one system to another was simple enough after I figured out I needed to disable UEFI (leaving only legacy) boot on my  UnRAID flash drive in the BIOS.

The system ran perfect with the HBA in the lowest slot (slot 2, pcie 2.0x4in8).

 

Then I added my GPU, in the slot the HBA was in, and moved the HBA to the next upper slot (slot 4, pcie 3.0x8).

System boots fine and everything is listed under system devices correctly.

 

00:00.0 Host bridge: Intel Corporation Xeon E5 v3/Core i7 DMI2 (rev 02)
00:01.0 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 1 (rev 02)
00:02.0 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 2 (rev 02)
00:02.2 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 2 (rev 02)
00:03.0 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 02)
00:03.2 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 02)
00:03.3 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 02)
00:04.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DMA Channel 0 (rev 02)
00:04.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DMA Channel 1 (rev 02)
00:04.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DMA Channel 2 (rev 02)
00:04.3 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DMA Channel 3 (rev 02)
00:04.4 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DMA Channel 4 (rev 02)
00:04.5 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DMA Channel 5 (rev 02)
00:04.6 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DMA Channel 6 (rev 02)
00:04.7 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DMA Channel 7 (rev 02)
00:05.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management (rev 02)
00:05.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Hot Plug (rev 02)
00:05.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 RAS, Control Status and Global Errors (rev 02)
00:05.4 PIC: Intel Corporation Xeon E5 v3/Core i7 I/O APIC (rev 02)
00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 series chipset SPSR (rev 05)
00:11.4 SATA controller: Intel Corporation C610/X99 series chipset sSATA Controller [AHCI mode] (rev 05)
00:14.0 USB controller: Intel Corporation C610/X99 series chipset USB xHCI Host Controller (rev 05)
00:16.0 Communication controller: Intel Corporation C610/X99 series chipset MEI Controller #1 (rev 05)
00:16.1 Communication controller: Intel Corporation C610/X99 series chipset MEI Controller #2 (rev 05)
00:1a.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 (rev 05)
00:1c.0 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #1 (rev d5)
00:1c.6 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #7 (rev d5)
00:1d.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation C610/X99 series chipset LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)
00:1f.3 SMBus: Intel Corporation C610/X99 series chipset SMBus Controller (rev 05)
00:1f.6 Signal processing controller: Intel Corporation C610/X99 series chipset Thermal Subsystem (rev 05)
01:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
04:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
06:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
06:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
06:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
06:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
08:00.0 VGA compatible controller: NVIDIA Corporation GK208 [GeForce GT 730] (rev a1)
08:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1)
09:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)
0a:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)
ff:0b.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02)
ff:0b.1 Performance counters: Intel Corporation Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02)
ff:0b.2 Performance counters: Intel Corporation Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02)
ff:0c.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0c.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0c.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0c.3 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0c.4 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0c.5 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0f.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02)
ff:0f.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02)
ff:0f.4 System peripheral: Intel Corporation Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02)
ff:0f.5 System peripheral: Intel Corporation Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02)
ff:0f.6 System peripheral: Intel Corporation Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02)
ff:10.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 PCIe Ring Interface (rev 02)
ff:10.1 Performance counters: Intel Corporation Xeon E5 v3/Core i7 PCIe Ring Interface (rev 02)
ff:10.5 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02)
ff:10.6 Performance counters: Intel Corporation Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02)
ff:10.7 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02)
ff:12.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Home Agent 0 (rev 02)
ff:12.1 Performance counters: Intel Corporation Xeon E5 v3/Core i7 Home Agent 0 (rev 02)
ff:13.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers (rev 02)
ff:13.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers (rev 02)
ff:13.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02)
ff:13.3 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02)
ff:13.4 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02)
ff:13.5 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02)
ff:13.6 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO Channel 0/1 Broadcast (rev 02)
ff:13.7 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO Global Broadcast (rev 02)
ff:14.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 Thermal Control (rev 02)
ff:14.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 Thermal Control (rev 02)
ff:14.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 ERROR Registers (rev 02)
ff:14.3 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 ERROR Registers (rev 02)
ff:14.6 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)
ff:14.7 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)
ff:15.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 Thermal Control (rev 02)
ff:15.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 Thermal Control (rev 02)
ff:15.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 ERROR Registers (rev 02)
ff:15.3 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 ERROR Registers (rev 02)
ff:16.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers (rev 02)
ff:16.6 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO Channel 2/3 Broadcast (rev 02)
ff:16.7 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO Global Broadcast (rev 02)
ff:17.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 Thermal Control (rev 02)
ff:17.4 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)
ff:17.5 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)
ff:17.6 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)
ff:17.7 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)
ff:1e.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Power Control Unit (rev 02)
ff:1e.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Power Control Unit (rev 02)
ff:1e.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Power Control Unit (rev 02)
ff:1e.3 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Power Control Unit (rev 02)
ff:1e.4 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Power Control Unit (rev 02)
ff:1f.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 VCU (rev 02)
ff:1f.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 VCU (rev 02)

/sys/kernel/iommu_groups/0/devices/0000:ff:0b.0
/sys/kernel/iommu_groups/0/devices/0000:ff:0b.1
/sys/kernel/iommu_groups/0/devices/0000:ff:0b.2
/sys/kernel/iommu_groups/1/devices/0000:ff:0c.0
/sys/kernel/iommu_groups/1/devices/0000:ff:0c.1
/sys/kernel/iommu_groups/1/devices/0000:ff:0c.2
/sys/kernel/iommu_groups/1/devices/0000:ff:0c.3
/sys/kernel/iommu_groups/1/devices/0000:ff:0c.4
/sys/kernel/iommu_groups/1/devices/0000:ff:0c.5
/sys/kernel/iommu_groups/2/devices/0000:ff:0f.0
/sys/kernel/iommu_groups/2/devices/0000:ff:0f.1
/sys/kernel/iommu_groups/2/devices/0000:ff:0f.4
/sys/kernel/iommu_groups/2/devices/0000:ff:0f.5
/sys/kernel/iommu_groups/2/devices/0000:ff:0f.6
/sys/kernel/iommu_groups/3/devices/0000:ff:10.0
/sys/kernel/iommu_groups/3/devices/0000:ff:10.1
/sys/kernel/iommu_groups/3/devices/0000:ff:10.5
/sys/kernel/iommu_groups/3/devices/0000:ff:10.6
/sys/kernel/iommu_groups/3/devices/0000:ff:10.7
/sys/kernel/iommu_groups/4/devices/0000:ff:12.0
/sys/kernel/iommu_groups/4/devices/0000:ff:12.1
/sys/kernel/iommu_groups/5/devices/0000:ff:13.0
/sys/kernel/iommu_groups/5/devices/0000:ff:13.1
/sys/kernel/iommu_groups/5/devices/0000:ff:13.2
/sys/kernel/iommu_groups/5/devices/0000:ff:13.3
/sys/kernel/iommu_groups/5/devices/0000:ff:13.4
/sys/kernel/iommu_groups/5/devices/0000:ff:13.5
/sys/kernel/iommu_groups/5/devices/0000:ff:13.6
/sys/kernel/iommu_groups/5/devices/0000:ff:13.7
/sys/kernel/iommu_groups/6/devices/0000:ff:14.0
/sys/kernel/iommu_groups/6/devices/0000:ff:14.1
/sys/kernel/iommu_groups/6/devices/0000:ff:14.2
/sys/kernel/iommu_groups/6/devices/0000:ff:14.3
/sys/kernel/iommu_groups/6/devices/0000:ff:14.6
/sys/kernel/iommu_groups/6/devices/0000:ff:14.7
/sys/kernel/iommu_groups/7/devices/0000:ff:15.0
/sys/kernel/iommu_groups/7/devices/0000:ff:15.1
/sys/kernel/iommu_groups/7/devices/0000:ff:15.2
/sys/kernel/iommu_groups/7/devices/0000:ff:15.3
/sys/kernel/iommu_groups/8/devices/0000:ff:16.0
/sys/kernel/iommu_groups/8/devices/0000:ff:16.6
/sys/kernel/iommu_groups/8/devices/0000:ff:16.7
/sys/kernel/iommu_groups/9/devices/0000:ff:17.0
/sys/kernel/iommu_groups/9/devices/0000:ff:17.4
/sys/kernel/iommu_groups/9/devices/0000:ff:17.5
/sys/kernel/iommu_groups/9/devices/0000:ff:17.6
/sys/kernel/iommu_groups/9/devices/0000:ff:17.7
/sys/kernel/iommu_groups/10/devices/0000:ff:1e.0
/sys/kernel/iommu_groups/10/devices/0000:ff:1e.1
/sys/kernel/iommu_groups/10/devices/0000:ff:1e.2
/sys/kernel/iommu_groups/10/devices/0000:ff:1e.3
/sys/kernel/iommu_groups/10/devices/0000:ff:1e.4
/sys/kernel/iommu_groups/11/devices/0000:ff:1f.0
/sys/kernel/iommu_groups/11/devices/0000:ff:1f.2
/sys/kernel/iommu_groups/12/devices/0000:00:00.0
/sys/kernel/iommu_groups/13/devices/0000:00:01.0
/sys/kernel/iommu_groups/14/devices/0000:00:02.0
/sys/kernel/iommu_groups/15/devices/0000:00:02.2
/sys/kernel/iommu_groups/16/devices/0000:00:03.0
/sys/kernel/iommu_groups/17/devices/0000:00:03.2
/sys/kernel/iommu_groups/18/devices/0000:00:03.3
/sys/kernel/iommu_groups/19/devices/0000:00:04.0
/sys/kernel/iommu_groups/19/devices/0000:00:04.1
/sys/kernel/iommu_groups/19/devices/0000:00:04.2
/sys/kernel/iommu_groups/19/devices/0000:00:04.3
/sys/kernel/iommu_groups/19/devices/0000:00:04.4
/sys/kernel/iommu_groups/19/devices/0000:00:04.5
/sys/kernel/iommu_groups/19/devices/0000:00:04.6
/sys/kernel/iommu_groups/19/devices/0000:00:04.7
/sys/kernel/iommu_groups/20/devices/0000:00:05.0
/sys/kernel/iommu_groups/20/devices/0000:00:05.1
/sys/kernel/iommu_groups/20/devices/0000:00:05.2
/sys/kernel/iommu_groups/20/devices/0000:00:05.4
/sys/kernel/iommu_groups/21/devices/0000:00:11.0
/sys/kernel/iommu_groups/21/devices/0000:00:11.4
/sys/kernel/iommu_groups/22/devices/0000:00:14.0
/sys/kernel/iommu_groups/23/devices/0000:00:16.0
/sys/kernel/iommu_groups/23/devices/0000:00:16.1
/sys/kernel/iommu_groups/24/devices/0000:00:1a.0
/sys/kernel/iommu_groups/25/devices/0000:00:1c.0
/sys/kernel/iommu_groups/26/devices/0000:00:1c.6
/sys/kernel/iommu_groups/27/devices/0000:00:1d.0
/sys/kernel/iommu_groups/28/devices/0000:00:1f.0
/sys/kernel/iommu_groups/28/devices/0000:00:1f.2
/sys/kernel/iommu_groups/28/devices/0000:00:1f.3
/sys/kernel/iommu_groups/28/devices/0000:00:1f.6
/sys/kernel/iommu_groups/29/devices/0000:01:00.0
/sys/kernel/iommu_groups/30/devices/0000:04:00.0
/sys/kernel/iommu_groups/31/devices/0000:06:00.0
/sys/kernel/iommu_groups/32/devices/0000:06:00.1
/sys/kernel/iommu_groups/33/devices/0000:06:00.2
/sys/kernel/iommu_groups/34/devices/0000:06:00.3
/sys/kernel/iommu_groups/35/devices/0000:08:00.0
/sys/kernel/iommu_groups/35/devices/0000:08:00.1
/sys/kernel/iommu_groups/36/devices/0000:09:00.0
/sys/kernel/iommu_groups/36/devices/0000:0a:00.0

 

So the GPU and HBA are in groups by themselves which means passthrough should work.

But it didn't...

The error is the familiar one (internal error: early end of file from monitor: possible problem:...) solved by enabling PCIe Override in VM Settings.

 

This option made everything work but I wasn't sure why. More searching led to this forum post

So I looked at the file

...
cgroup_device_acl = [
    "/dev/null", "/dev/full", "/dev/zero",
    "/dev/random", "/dev/urandom",
    "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
    "/dev/rtc","/dev/hpet", "/dev/vfio/vfio",
    "/dev/vfio/1","/dev/vfio/2","/dev/vfio/3",
    "/dev/vfio/4","/dev/vfio/5","/dev/vfio/6",
    "/dev/vfio/7","/dev/vfio/8","/dev/vfio/9",
    "/dev/vfio/10","/dev/vfio/11","/dev/vfio/12",
    "/dev/vfio/13","/dev/vfio/14","/dev/vfio/15",
    "/dev/vfio/16","/dev/vfio/17","/dev/vfio/18",
    "/dev/vfio/19","/dev/vfio/20","/dev/vfio/21",
    "/dev/vfio/22","/dev/vfio/23","/dev/vfio/24",
    "/dev/vfio/25","/dev/vfio/26","/dev/vfio/27",
    "/dev/vfio/28","/dev/vfio/29","/dev/vfio/30",
    "/dev/vfio/31","/dev/vfio/32"
]
...

Some groups are missing, 1-32 are there but I have 36 IOMMU groups on my motherboard.

 

So I edited it to add 33-35 (36 is for IPMI).

cgroup_device_acl = [
    "/dev/null", "/dev/full", "/dev/zero",
    "/dev/random", "/dev/urandom",
    "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
    "/dev/rtc","/dev/hpet", "/dev/vfio/vfio",
    "/dev/vfio/1","/dev/vfio/2","/dev/vfio/3",
    "/dev/vfio/4","/dev/vfio/5","/dev/vfio/6",
    "/dev/vfio/7","/dev/vfio/8","/dev/vfio/9",
    "/dev/vfio/10","/dev/vfio/11","/dev/vfio/12",
    "/dev/vfio/13","/dev/vfio/14","/dev/vfio/15",
    "/dev/vfio/16","/dev/vfio/17","/dev/vfio/18",
    "/dev/vfio/19","/dev/vfio/20","/dev/vfio/21",
    "/dev/vfio/22","/dev/vfio/23","/dev/vfio/24",
    "/dev/vfio/25","/dev/vfio/26","/dev/vfio/27",
    "/dev/vfio/28","/dev/vfio/29","/dev/vfio/30",
    "/dev/vfio/31","/dev/vfio/32","/dev/vfio/33",
    "/dev/vfio/34","/dev/vfio/35"
]

 

Disabled PCIe override. Stop Array. Reboot. Start Array. Checked with cat to see if the change is persistent, it was.

Now pass-through works in slot 2 and everything else still works as it should.

 

I'm not sure what the root cause of this issue is (if it's a compile time decision, 2^5 seemed like enough?).

If there's a reason I shouldn't do this please let me know. I don't see any issues since the slots are in their own IOMMU groups.

 

There's no reason currently that required me to passthough a device in the lowest slot, I just wanted to so the GPU didn't block another expansion slot.

 

 

P.S. I had a major scare/issue when trying different pcie slots.

[*]set passthough for GPU in slot 2 to VM

[*]HBA in slot 4

[*]Attempt to start VM, ACS related error message results

[*]Shutdown

[*]GPU in slot 2, HBA in slot 4

[*]Power on, start array

[*]Start VM...ALL drives on HBA disappear from array

[*]Stop everythin, reboot, drive reappear

Result: VM address settings for passthough addresses persist after hardware changes is you don't update them. So when I started the VM it tryed to pass through a GPU in the slot my HBA was in which dropped all the drives connected to it from my array.

 

I'm not sure what the root cause of this issue is (if it's a compile time decision, 2^5 seemed like enough?).

If there's a reason I shouldn't do this please let me know.

 

Smart man...

 

You found the same bug we ran into recently with Linus from LinusTechTips when we were helping him set up this monster rig of his for two gaming systems at the same time.  It is quite literally just an oversight on our part to not put more VFIO groups in the qemu.conf file.  This will be patched in the next release.

 

P.S. I had a major scare/issue when trying different pcie slots.

[*]set passthough for GPU in slot 2 to VM

[*]HBA in slot 4

[*]Attempt to start VM, ACS related error message results

[*]Shutdown

[*]GPU in slot 2, HBA in slot 4

[*]Power on, start array

[*]Start VM...ALL drives on HBA disappear from array

[*]Stop everythin, reboot, drive reappear

Result: VM address settings for passthough addresses persist after hardware changes is you don't update them. So when I started the VM it tryed to pass through a GPU in the slot my HBA was in which dropped all the drives connected to it from my array.

 

Yowza, good catch.  Something we'll have to figure out how to handle.

I'm not sure what the root cause of this issue is (if it's a compile time decision, 2^5 seemed like enough?).

If there's a reason I shouldn't do this please let me know.

 

Smart man...

 

You found the same bug we ran into recently with Linus from LinusTechTips when we were helping him set up this monster rig of his for two gaming systems at the same time.  It is quite literally just an oversight on our part to not put more VFIO groups in the qemu.conf file.  This will be patched in the next release.

 

P.S. I had a major scare/issue when trying different pcie slots.

[*]set passthough for GPU in slot 2 to VM

[*]HBA in slot 4

[*]Attempt to start VM, ACS related error message results

[*]Shutdown

[*]GPU in slot 2, HBA in slot 4

[*]Power on, start array

[*]Start VM...ALL drives on HBA disappear from array

[*]Stop everythin, reboot, drive reappear

Result: VM address settings for passthough addresses persist after hardware changes is you don't update them. So when I started the VM it tryed to pass through a GPU in the slot my HBA was in which dropped all the drives connected to it from my array.

 

Yowza, good catch.  Something we'll have to figure out how to handle.

 

Seems it hasn't been patched.  Just ran into this on 6.1.6.

Linus from LinuxTechTips is running a 6.2 beta so perhaps it'll be fixed in that release.....

 

 

It is fixed in 6.2. VM Manager had a lot of under the hood upgrades for 6.2.

Jonp when can we expect 6.2 ?

Soon 

Soon 

 

Soon 

 

Patience  Soon haven't passed yet 

Soon 

 

Patience  Soon haven't passed yet 

Past history shows that soon may be as much as a year, give or take. If you don't believe me, look through the forums for the discussions when previous versions were pending.

Yeah I know.. 6.2 has some VM improvements (nebulous PM from JonP) which I am told I would like.. and since my VM has gone all crappy I could do with some positive news

Sorry for coming in late on this one. I am having some issues that I think are related and have seen in a few places that the 6.2 release will have some VM improvements.

 

@methanoid, your signature says you're on 6.2Beta... have you noticed any VM improvements?

 

My initial issue with the PCI pass through and ordering of GPUs was somewhat solved (wish I could rearrange them though since my best PCI bandwidth slot is taken by the unreal connection)... Anyway, my primary problem now is audio glitches. It was really bad, but then I bought a dedicated USB controller and passed it through directly to the VM. Then I made sure all my devices use the MSI IRQ protocol (win registry hack). That took care of 95% of it.

 

However there is still that 5%... that crystal clear audio then that slight skip, pop, or crack. Sometimes it goes minutes without noticing, but then it just comes back. I know this is something extremely knit-picky... and maybe the virtualization tech/drivers/OS support isn't there yet, but I'm hoping this is something that can be magically fixed in a qemu update, or a reworking of the groups.

I had issues with audio with my Xeon 8 core. These got cleared up largely by switching off Hyperthreading but left me a little disappointed. I then decided to revert to my 4790K system for unRAID (which didnt give me audio issues). But I suspect I still get the odd glitch. This is with HT on though. I only just switched to 6.2 a few days back. VM seems fine. Not used enough to say I can tell any difference though as I have been unwell and not on the PC much

 

I'd try disable HT first if I were you.  I will report if I spot any 6.2 improvements (although the Readme mentions VM improvements for WIn10 VMs which is what I have)