jonp Posted October 11, 2014 Share Posted October 11, 2014 UPDATE: THIS GUIDE IS OUTDATED! AS OF BETA15, YOU CAN CREATE VMS WITH GPU PASS THROUGH AND KVM USING THE WEBGUI ITSELF! Guys, Just thought I'd take a minute to help clarify some things that may be confusing when working with KVM in the latest beta of unRAID 6. Syslinux.cfg 101 As of beta 10a, you no longer have to include nearly as many additional "append" statements in the syslinux.cfg to make use of KVM or the advanced features of virtualization like IOMMU. While we haven't tested this with AMD, for Intel-based chipsets, you do not need iommu=1 in your syslinux config. You also do not need to use pciback to hide any PCI devices from unRAID. unRAID doesn't bind drivers to PCI devices. The only statement you WILL need in there to ensure clean operations with VFIO is this: pcie_acs_override=downstream For the sake of simplicity, here's my complete syslinux.cfg that I use on my multiple test systems today: default /syslinux/menu.c32 menu title Lime Technology prompt 0 timeout 50 label unRAID OS menu default kernel /bzimage append pcie_acs_override=downstream initrd=/bzroot label unRAID OS Safe Mode (no plugins) kernel /bzimage append initrd=/bzroot unraidsafemode label Memtest86+ kernel /memtest label Xen/unRAID OS kernel /syslinux/mboot.c32 append /xen dom0_mem=4096M,max:4096M --- /bzimage --- /bzroot label Xen/unRAID OS Safe Mode (no plugins) kernel /syslinux/mboot.c32 append /xen --- /bzimage --- /bzroot unraidsafemode Notice the bolded part that you may not have been used to seeing previously... We haven't gotten iGPU pass through to work yet Not saying this is impossible, but we haven't been able to get this to work on any of our test systems as of yet. For those wondering, iGPU means integrated graphics or "on-board" graphics as in the graphics device is not installed into a PCI/e slot on the server, but a part of the motherboard itself. The challenges relating to iGPU pass through are not easy to solve, but some may be able to get this to work. We can confirm that none of our test systems have had success with this yet. We have tried numerous patches of various types into our kernel with no success. Even Alex Williamson of Red Hat who wrote the patches even said there is limited success in using them at this time. If anyone gets this working on their system, please share! How do I get started with passing through devices? Stay tuned for better guides and videos on this that we are working on. Also check out the work of dmacias here on our forums. He's built a wonderful set of plugins to help folks along who are eager to get started with KVM virtual machines. Avoid passing through SATA/SAS/USB controllers Just remember if you are trying to pass through your SATA/SAS/USB controllers to a VM (not a single device, but the whole controller), it can be hard to discern which controller is attached to your unRAID flash device. If you pass through the controller used for your flash device, you should reboot immediately and you will see you have had an unclean shutdown. Same goes if you pass through a SATA/SAS controller and you inadvertently lose a number of disks from your array. You can pass through a SATA/SAS or USB device to a VM without passing through the entire controller and we expect this capability to become easier to use over time. GPU Pass Through...The Basics Once you have a syslinux.cfg set as I have indicated above, the two things you need to get moving are binding your GPU to the vfio-pci driver and adding the appropriate lines of code to your XML file for your VM when getting started. To bind a device to vfio-pci, type the following from command line: vfio-bind 0000:0#:00.0 0000:0#:00.1 Replace # with the PCI slot you have the device installed into. So for the first slot, use 1, for the second, use 2, and so on. Now in your XML for your virtual machine: <os> <type arch='x86_64' machine='q35'>hvm</type> <boot dev='hd'/> <bootmenu enable='no'/> </os> Some folks may not use the machine= statement here and still have success. We prefer to use the q35 chipset for emulation by QEMU because it's a more modern virtual chipset that we've had better success with in our test hardware. <cpu mode='host-passthrough'> <topology sockets='2' cores='2' threads='1'/> </cpu> Now between the </devices> and </domain> part of your XML (near the bottom), you will put this: <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=0#:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=0#:00.1,bus=pcie.0'/>/> </qemu:commandline> Hopefully this is helpful for those that are experimenting with KVM right now, but know that we are making this a lot easier. Edit: someone here made me aware of a typo in this original post. The original typo had the syntax partially incorrect for the vfio-bind command. It has since been rectified. Thanks!! Quote Link to comment
bungee91 Posted October 11, 2014 Share Posted October 11, 2014 <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=0#:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=0#:00.1,bus=pcie.0'/>/> </qemu:commandline> Thanks for some input on this, I think it will be helpful to a lot of people. A couple of questions that I have. [*]When passing through 1 device that is not a GPU/video card, do you need to include this line in the <qemu:commandline> section, and what exactly does it do? "<qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/>" In my case I was experimenting passing through a tv tuner/capture card, tried both ways, I believe both were hit or miss. [*]It seems that when passing through a device in Windows (8.1 for my example) I can get it to install the drivers and function, however after shutting down the VM, or restarting I get stuck at the boot screen. If I remove the pass-through it starts right up. Do you HAVE to eject any pass through device prior to shutdown/reset? I have automated this with Devmanview, however I still have issues and it is inconsistent [*]What is the correct syntax (an example would be nice) to add to the go script for vfio-bind at bootup? I've attempted /usr/local/sbin/vfio-bind 0000:0#:00.0 0000:0#.00.1 (with my proper addresses) and I get an error of some sort on the console. [*]When do we get to "play" with your special compiled XBMC (Kodi...Grrrr) build that you've been working on?... I'm certainly looking forward to it, and even if it is somewhat Beta it would be nice to get a chance to try it out. Quote Link to comment
jonp Posted October 11, 2014 Author Share Posted October 11, 2014 The QEMU statements in the XML code seem to be the most reliable way to success on the multiple test systems we've used, but they have always been with more modern chipsets. That said, updating the motherboard BIOS can also have dramatic impacts to the success of pass through. As for your blue screening issues, can I ask you to share your xml? It will help me understand your setup better. With respect to OpenELEC, we just updated it to 4.2.1 and are going through some final testing but it is working solid. I am hoping to release that soon. Quote Link to comment
jonp Posted October 11, 2014 Author Share Posted October 11, 2014 On and with respect to VFIO binding at boot, I have what you have, so can you share the error your getting? Quote Link to comment
bungee91 Posted October 11, 2014 Share Posted October 11, 2014 The QEMU statements in the XML code seem to be the most reliable way to success on the multiple test systems we've used, but they have always been with more modern chipsets. That said, updating the motherboard BIOS can also have dramatic impacts to the success of pass through. I have a Z97 chipset, so I'm current/modern however I just don't know what it does? As for your blue screening issues, can I ask you to share your xml? It will help me understand your setup better. I just get stuck at boot, no blue screen, no halt, just a spinning little wheel. I'd be more than happy to share, if needed. With respect to OpenELEC, we just updated it to 4.2.1 and are going through some final testing but it is working solid. I am hoping to release that soon. I think a lot of people are looking forward to it.. Now about Windows and restarting/ejecting?... Input would be very helpful, as hard locking up the entire server due to a restart got old REAL quick!.. Quote Link to comment
jonp Posted October 11, 2014 Author Share Posted October 11, 2014 Check your motherboard bios on the manufacturers website and see if there is a BIOS update available. Let me know. We've seen issues like you described but updates to the bios resolved it 99% of the time... Quote Link to comment
bungee91 Posted October 11, 2014 Share Posted October 11, 2014 Check your motherboard bios on the manufacturers website and see if there is a BIOS update available. Let me know. We've seen issues like you described but updates to the bios resolved it 99% of the time... No updates since June, so I'm as recent as I can be. So when you're messing with you VM and GPU pass-through, no eject or anything is needed when rebooting/shutting down? Quote Link to comment
bungee91 Posted October 11, 2014 Share Posted October 11, 2014 As for your blue screening issues, can I ask you to share your xml? It will help me understand your setup better. XML attached (had to change extension to .txt in order to attach), also lspci -k output Sorry to derail the thread all about me! Win8WebVirtMgr.txt lspci_output.txt Quote Link to comment
jonp Posted October 11, 2014 Author Share Posted October 11, 2014 Take this out please: <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes' listen='192.168.1.161' keymap='en-us'> <listen type='address' address='192.168.1.161'/> </graphics> <video> <model type='cirrus' vram='9216' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </video> If you want remote control, enable that from within the VM. Quote Link to comment
bungee91 Posted October 11, 2014 Share Posted October 11, 2014 Take this out please: <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes' listen='192.168.1.161' keymap='en-us'> <listen type='address' address='192.168.1.161'/> </graphics> <video> <model type='cirrus' vram='9216' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </video> If you want remote control, enable that from within the VM. Will do, but question (worry). Without it if it doesn't successfully boot (without passing through a GPU) how am I to know where/what it may be doing?.. I'm basically blind if the remote connection just fails? I assume the best answer here is to pass a GPU so that I can see? Or just hope all is well?... Quote Link to comment
jonp Posted October 11, 2014 Author Share Posted October 11, 2014 I don't follow. If you're passing through a GPU, you will know if pass through is working right away. You should see a SeaBIOS loading immediately. I'll be posting videos of what you should expect to see. This will help. Quote Link to comment
bungee91 Posted October 11, 2014 Share Posted October 11, 2014 I don't follow. If you're passing through a GPU, you will know if pass through is working right away. You should see a SeaBIOS loading immediately. I'll be posting videos of what you should expect to see. This will help. No I get it, but thanks... Again the question is WITHOUT passing through a GPU, I'm playing with TV tuner/capture card because GPU pass through was locking up the entire server on reboot/shutdown of the VM. If this is no longer the case, I have a GPU just waiting to be passed. Will do, but question (worry). Without it if it doesn't successfully boot (without passing through a GPU) Quote Link to comment
bungee91 Posted October 11, 2014 Share Posted October 11, 2014 On and with respect to VFIO binding at boot, I have what you have, so can you share the error your getting? I edited this again, I must have had something wrong as it works great now! Quote Link to comment
stottle Posted October 12, 2014 Share Posted October 12, 2014 vfio-pci 0000:0#:00.0 0000:0#.00.1 Did you change the name from vfio-bind to vfio-pci? Or is this a typo? Also, per a previous answer from you, if you want the settings to work after a reboot, this line would need to go into your "go" script, correct? Quote Link to comment
jonp Posted October 12, 2014 Author Share Posted October 12, 2014 vfio-pci 0000:0#:00.0 0000:0#.00.1 Did you change the name from vfio-bind to vfio-pci? Or is this a typo? Also, per a previous answer from you, if you want the settings to work after a reboot, this line would need to go into your "go" script, correct? Good catch. No we didn't. Sorry about that. Just fixed the typo in the OP. Oh and correct. In your go script, put /usr/local/sbin/vfio-bind to reference the full path. Quote Link to comment
bungee91 Posted October 12, 2014 Share Posted October 12, 2014 So when you're messing with you VM and GPU pass-through, no eject or anything is needed when rebooting/shutting down? Any input on this? I'd love to start playing with this again, but complete lock-ups & the parity check afterwards was annoying. Quote Link to comment
jonp Posted October 13, 2014 Author Share Posted October 13, 2014 So when you're messing with you VM and GPU pass-through, no eject or anything is needed when rebooting/shutting down? Any input on this? I'd love to start playing with this again, but complete lock-ups & the parity check afterwards was annoying. With Windows 8.1 and multiple GPUs on multiple motherboards now, yes, we have had no issues. Try without the VNC component and with GPU pass through and see how reboots work. If possible, use a test vm (create a new one) just to test it out. Quote Link to comment
rd_blair Posted October 13, 2014 Share Posted October 13, 2014 When I use Webvirtmrg to create an instance it chooses: <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type> Is there some way to change this. Changing this line in the vm.xml file creates errors. I also have virt-manager on another machine but see no way to change this entry. Reed Quote Link to comment
jonp Posted October 13, 2014 Author Share Posted October 13, 2014 When I use Webvirtmrg to create an instance it chooses: <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type> Is there some way to change this. Changing this line in the vm.xml file creates errors. I also have virt-manager on another machine but see no way to change this entry. Reed Copy the XML from the domain you created and make the change, then create a "new instance" and paste the XML in there with the edit. It will work. Quote Link to comment
rd_blair Posted October 13, 2014 Share Posted October 13, 2014 @jonp Tried that several times before did not work. Reed Quote Link to comment
jonp Posted October 13, 2014 Author Share Posted October 13, 2014 @jonp Tried that several times before did not work. Reed Are you using web virt manager to do this? What error do you get? Quote Link to comment
rd_blair Posted October 13, 2014 Share Posted October 13, 2014 Using web virt manager I copy the the vm.xlm contents to the clipboard, then destroy the instance. Using web virt manager new instance from xlm. I paste from the clipboard and edit line: <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type> to: <type arch='x86_64' machine='q35'>hvm</type> Then create and I get: internal error: PCI bus is not compatible with the device at 0000:00:1e.0. Device requires a PCI Express slot, which is not provided by bus 0000:00 also tried with same error: <type arch='x86_64' machine='q35'>hvm</type> <loader>/usr/share/qemu/bios-256k.bin</loader> Reed Quote Link to comment
bungee91 Posted October 13, 2014 Share Posted October 13, 2014 WebVirt for me after copy/paste into a new XML looks like this, in case it helps any. <os> <type arch='x86_64' machine='pc-q35-2.1'>hvm</type> <boot dev='hd'/> <bootmenu enable='no'/> </os> Quote Link to comment
peter_sm Posted October 13, 2014 Share Posted October 13, 2014 This is from virtmanager (latest,supporting Q35 machine) from my arch VM Its untouched, so you need to tweak network,VNC, and more to fit your needs....... but its a original q35 machine. <!-- WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE OVERWRITTEN AND LOST. Changes to this xml configuration should be made using: virsh edit rhel7.0 or other application using the libvirt API. --> <domain type='kvm'> <name>rhel7.0</name> <uuid>87eb0395-c085-444a-a2c5-67310e94e3cf</uuid> <memory unit='KiB'>1048576</memory> <currentMemory unit='KiB'>1048576</currentMemory> <vcpu placement='static'>1</vcpu> <os> <type arch='x86_64' machine='pc-q35-2.1'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>SandyBridge</model> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/mnt/vm_disk/KVM/testarch.qcow2'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/> </disk> <disk type='block' device='cdrom'> <driver name='qemu' type='raw'/> <target dev='hda' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x2'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> <controller type='ide' index='0'/> <interface type='network'> <mac address='52:54:00:50:b0:e4'/> <source network='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='vmvga' vram='9216' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </video> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/> </memballoon> </devices> </domain> Quote Link to comment
bungee91 Posted October 13, 2014 Share Posted October 13, 2014 Since we are emulating a Q35 chipset, is it recommended to install chipset drivers for this in Windows? Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.