Some pointers on KVM, GPU Pass Through, and Syslinux.cfg


jonp

Recommended Posts

UPDATE:  THIS GUIDE IS OUTDATED!  AS OF BETA15, YOU CAN CREATE VMS WITH GPU PASS THROUGH AND KVM USING THE WEBGUI ITSELF!

 

Guys,

 

Just thought I'd take a minute to help clarify some things that may be confusing when working with KVM in the latest beta of unRAID 6.

 

Syslinux.cfg 101

As of beta 10a, you no longer have to include nearly as many additional "append" statements in the syslinux.cfg to make use of KVM or the advanced features of virtualization like IOMMU.  While we haven't tested this with AMD, for Intel-based chipsets, you do not need iommu=1 in your syslinux config.  You also do not need to use pciback to hide any PCI devices from unRAID.  unRAID doesn't bind drivers to PCI devices.

 

The only statement you WILL need in there to ensure clean operations with VFIO is this: 

pcie_acs_override=downstream

 

For the sake of simplicity, here's my complete syslinux.cfg that I use on my multiple test systems today:

 

default /syslinux/menu.c32

menu title Lime Technology

prompt 0

timeout 50

label unRAID OS

  menu default

  kernel /bzimage

  append pcie_acs_override=downstream initrd=/bzroot

label unRAID OS Safe Mode (no plugins)

  kernel /bzimage

  append initrd=/bzroot unraidsafemode

label Memtest86+

  kernel /memtest

label Xen/unRAID OS

  kernel /syslinux/mboot.c32

  append /xen dom0_mem=4096M,max:4096M --- /bzimage --- /bzroot

label Xen/unRAID OS Safe Mode (no plugins)

  kernel /syslinux/mboot.c32

  append /xen --- /bzimage --- /bzroot unraidsafemode

 

Notice the bolded part that you may not have been used to seeing previously...

 

We haven't gotten iGPU pass through to work yet

Not saying this is impossible, but we haven't been able to get this to work on any of our test systems as of yet.  For those wondering, iGPU means integrated graphics or "on-board" graphics as in the graphics device is not installed into a PCI/e slot on the server, but a part of the motherboard itself.

 

The challenges relating to iGPU pass through are not easy to solve, but some may be able to get this to work.  We can confirm that none of our test systems have had success with this yet.  We have tried numerous patches of various types into our kernel with no success.  Even Alex Williamson of Red Hat who wrote the patches even said there is limited success in using them at this time.

 

If anyone gets this working on their system, please share!

 

How do I get started with passing through devices?

Stay tuned for better guides and videos on this that we are working on.  Also check out the work of dmacias here on our forums.  He's built a wonderful set of plugins to help folks along who are eager to get started with KVM virtual machines.

 

Avoid passing through SATA/SAS/USB controllers

Just remember if you are trying to pass through your SATA/SAS/USB controllers to a VM (not a single device, but the whole controller), it can be hard to discern which controller is attached to your unRAID flash device.  If you pass through the controller used for your flash device, you should reboot immediately and you will see you have had an unclean shutdown.  Same goes if you pass through a SATA/SAS controller and you inadvertently lose a number of disks from your array.

 

You can pass through a SATA/SAS or USB device to a VM without passing through the entire controller and we expect this capability to become easier to use over time.

 

GPU Pass Through...The Basics

Once you have a syslinux.cfg set as I have indicated above, the two things you need to get moving are binding your GPU to the vfio-pci driver and adding the appropriate lines of code to your XML file for your VM when getting started.  To bind a device to vfio-pci, type the following from command line:

 

vfio-bind 0000:0#:00.0 0000:0#:00.1

 

Replace # with the PCI slot you have the device installed into.  So for the first slot, use 1, for the second, use 2, and so on.

 

Now in your XML for your virtual machine:

 

    <os>
	<type arch='x86_64' machine='q35'>hvm</type>
	<boot dev='hd'/>
	<bootmenu enable='no'/>
    </os>

 

Some folks may not use the machine= statement here and still have success.  We prefer to use the q35 chipset for emulation by QEMU because it's a more modern virtual chipset that we've had better success with in our test hardware.

 

	<cpu mode='host-passthrough'>
	<topology sockets='2' cores='2' threads='1'/>
</cpu>

 

Now between the </devices> and </domain> part of your XML (near the bottom), you will put this:

 

<qemu:commandline>

<qemu:arg value='-device'/>

<qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>

<qemu:arg value='-device'/>

<qemu:arg value='vfio-pci,host=0#:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>

<qemu:arg value='-device'/>

<qemu:arg value='vfio-pci,host=0#:00.1,bus=pcie.0'/>/>

</qemu:commandline>

 

Hopefully this is helpful for those that are experimenting with KVM right now, but know that we are making this a lot easier.

 

Edit:  someone here made me aware of a typo in this original post. The original typo had the syntax partially incorrect for the vfio-bind command. It has since been rectified. Thanks!!

 

Link to comment
  • Replies 103
  • Created
  • Last Reply

Top Posters In This Topic

<qemu:commandline>

<qemu:arg value='-device'/>

<qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>

<qemu:arg value='-device'/>

<qemu:arg value='vfio-pci,host=0#:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>

<qemu:arg value='-device'/>

<qemu:arg value='vfio-pci,host=0#:00.1,bus=pcie.0'/>/>

</qemu:commandline>

 

Thanks for some input on this, I think it will be helpful to a lot of people.

 

A couple of questions that I have.

 

[*]When passing through 1 device that is not a GPU/video card, do you need to include this line in the <qemu:commandline> section, and what exactly does it do?  "<qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>

<qemu:arg value='-device'/>"

In my case I was experimenting passing through a tv tuner/capture card, tried both ways, I believe both were hit or miss.

 

[*]It seems that when passing through a device in Windows (8.1 for my example) I can get it to install the drivers and function, however after shutting down the VM, or restarting I get stuck at the boot screen. If I remove the pass-through it starts right up. Do you HAVE to eject any pass through device prior to shutdown/reset? I have automated this with Devmanview, however I still have issues and it is inconsistent

[*]What is the correct syntax (an example would be nice) to add to the go script for vfio-bind at bootup? I've attempted /usr/local/sbin/vfio-bind 0000:0#:00.0 0000:0#.00.1 (with my proper addresses) and I get an error of some sort on the console.

[*]When do we get to "play" with your special compiled XBMC (Kodi...Grrrr) build that you've been working on?... I'm certainly looking forward to it, and even if it is somewhat Beta it would be nice to get a chance to try it out.

 

Link to comment

The QEMU statements in the XML code seem to be the most reliable way to success on the multiple test systems we've used, but they have always been with more modern chipsets.  That said, updating the motherboard BIOS can also have dramatic impacts to the success of pass through.

 

As for your blue screening issues, can I ask you to share your xml?  It will help me understand your setup better.

 

With respect to OpenELEC, we just updated it to 4.2.1 and are going through some final testing but it is working solid.  I am hoping to release that soon.

 

Link to comment

The QEMU statements in the XML code seem to be the most reliable way to success on the multiple test systems we've used, but they have always been with more modern chipsets.  That said, updating the motherboard BIOS can also have dramatic impacts to the success of pass through.

I have a Z97 chipset, so I'm current/modern however I just don't know what it does?

As for your blue screening issues, can I ask you to share your xml?  It will help me understand your setup better.

I just get stuck at boot, no blue screen, no halt, just a spinning little wheel.

I'd be more than happy to share, if needed.

With respect to OpenELEC, we just updated it to 4.2.1 and are going through some final testing but it is working solid.  I am hoping to release that soon.

I think a lot of people are looking forward to it..

 

Now about Windows and restarting/ejecting?... Input would be very helpful, as hard locking up the entire server due to a restart got old REAL quick!..

Link to comment

Check your motherboard bios on the manufacturers website and see if there is a BIOS update available.  Let me know.  We've seen issues like you described but updates to the bios resolved it 99% of the time...

No updates since June, so I'm as recent as I can be.

 

So when you're messing with you VM and GPU pass-through, no eject or anything is needed when rebooting/shutting down?

Link to comment

Take this out please:

 

<input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes' listen='192.168.1.161' keymap='en-us'> <listen type='address' address='192.168.1.161'/> </graphics> <video> <model type='cirrus' vram='9216' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </video>

 

If you want remote control, enable that from within the VM.

Link to comment

Take this out please:

 

<input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes' listen='192.168.1.161' keymap='en-us'> <listen type='address' address='192.168.1.161'/> </graphics> <video> <model type='cirrus' vram='9216' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </video>

 

If you want remote control, enable that from within the VM.

 

Will do, but question (worry).

Without it if it doesn't successfully boot (without passing through a GPU) how am I to know where/what it may be doing?.. I'm basically blind if the remote connection just fails?

I assume the best answer here is to pass a GPU so that I can see? Or just hope all is well?...  :P

Link to comment

I don't follow.  If you're passing through a GPU, you will know if pass through is working right away. You should see a SeaBIOS loading immediately. I'll be posting videos of what you should expect to see.  This will help.

 

No I get it, but thanks...

Again the question is WITHOUT passing through a GPU, I'm playing with TV tuner/capture card because GPU pass through was locking up the entire server on reboot/shutdown of the VM. If this is no longer the case, I have a GPU just waiting to be passed.

 

Will do, but question (worry).

Without it if it doesn't successfully boot (without passing through a GPU)

Link to comment

vfio-pci 0000:0#:00.0 0000:0#.00.1

Did you change the name from vfio-bind to vfio-pci?  Or is this a typo?  Also, per a previous answer from you, if you want the settings to work after a reboot, this line would need to go into your "go" script, correct?

Good catch. No we didn't. Sorry about that. Just fixed the typo in the OP.

 

Oh and correct.  In your go script, put /usr/local/sbin/vfio-bind to reference the full path.

 

Link to comment

So when you're messing with you VM and GPU pass-through, no eject or anything is needed when rebooting/shutting down?

Any input on this?

 

I'd love to start playing with this again, but complete lock-ups & the parity check afterwards was annoying.

 

With Windows 8.1 and multiple GPUs on multiple motherboards now, yes, we have had no issues.

 

Try without the VNC component and with GPU pass through and see how reboots work.

 

If possible, use a test vm (create a new one) just to test it out.

Link to comment

When I use Webvirtmrg to create an instance it chooses:

 

<type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type>

 

Is there some way to change this. Changing this line in the vm.xml file creates errors. I also have virt-manager on another machine but see no way to change this entry.

 

Reed

Link to comment

When I use Webvirtmrg to create an instance it chooses:

 

<type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type>

 

Is there some way to change this. Changing this line in the vm.xml file creates errors. I also have virt-manager on another machine but see no way to change this entry.

 

Reed

 

Copy the XML from the domain you created and make the change, then create a "new instance" and paste the XML in there with the edit.  It will work.

Link to comment

Using web virt manager I copy the the vm.xlm contents to the clipboard, then destroy the instance. Using web virt manager new instance from xlm. I paste from the clipboard and edit line:

 

  <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type>

to:

  <type arch='x86_64' machine='q35'>hvm</type>

 

Then create and I get:

 

internal error: PCI bus is not compatible with the device at 0000:00:1e.0. Device requires a PCI Express slot, which is not provided by bus 0000:00

 

also tried with same error:

    <type arch='x86_64' machine='q35'>hvm</type>

    <loader>/usr/share/qemu/bios-256k.bin</loader>

 

Reed

 

Link to comment

This is from virtmanager (latest,supporting Q35 machine) from my arch VM

Its untouched, so you need to tweak network,VNC, and more to fit your needs....... but its a original q35 machine.

 

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit rhel7.0
or other application using the libvirt API.
-->

<domain type='kvm'>
  <name>rhel7.0</name>
  <uuid>87eb0395-c085-444a-a2c5-67310e94e3cf</uuid>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-2.1'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>SandyBridge</model>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/mnt/vm_disk/KVM/testarch.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </disk>
    <disk type='block' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x2'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
    <controller type='ide' index='0'/>
    <interface type='network'>
      <mac address='52:54:00:50:b0:e4'/>
      <source network='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='vmvga' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.