jonp

***GUIDE*** Passing Through Network Controllers to unRAID 6 Virtual Machines

132 posts in this topic Last Reply

Recommended Posts

In a recent reply to a post by another forum member (archedraft), I provided this guide to help him assign one of his NIC devices to a virtual machine, leaving the other for host networking (unRAID OS).  I didn't see much point in this because with KVM and VirtIO, we can create virtual network interfaces that offer little to no overhead over a physical NIC, but after testing with pfSense, archedraft confirmed for me that he saw a dramatic performance increase.  The reason?  In this particular instance, pfSense was acting as a firewall and is based on FreeBSD.  The FreeBSD kernel used by pfSense, while having support for VirtIO, appears to be out of date and was not allowing full 1gbps LAN throughput as it does with Linux or Windows VMs.  Passing through a physical Ethernet controller to his pfSense VM in this instance resolved his issue.

 

So we have found at least one use case thus far to consider such a method, but in the future, we may find more.  And since the question comes up from time to time, I thought it prudent to post this here as an advanced guide for those that want to try it.

 

WARNING:

If you do not have multiple NICs in your system, doing this will result in your server losing all network connectivity.

 

IMPORTANT:  Regarding VM to Host Networking Performance

When VMs utilize VirtIO, their is another distinct advantage in that networking between the host and guest can take place without traversing the copper wire.  This allows for much faster throughput than the physical NIC hardware even supports at the port level. As an example, in mounting an SMB share to my SSD-based cache pool from inside my Windows VM, I was able to see IO throughput to the share exceed 250MB/s (that's megabytes, not bits).

 

When a VM is assigned a physical network controller, this advantage disappears as the VM will communicate with the host as if it was a separate physical machine, going out the one NIC, down to your router/switching infrastructure, and then back in.  This will limit your network throughput to that of the physical hardware.  In my previous Windows VM / SMB example, I would be limited to 1gbps or 125MB/s.

 

Guide

1 - Login to your server via ssh.

2 - Type the following command:

 

lspci

 

You will get a list like this:

 

00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06)
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06)
00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06)
00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 04)
00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-V (rev 04)
00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 04)
00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d4)
00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d4)
00:1f.0 ISA bridge: Intel Corporation Z87 Express LPC Controller (rev 04)
00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 04)
00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 04)
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series]
02:00.0 VGA compatible controller: NVIDIA Corporation GK110 [GeForce GTX 780] (rev a1)
02:00.1 Audio device: NVIDIA Corporation GK110 HDMI Audio (rev a1)
04:00.0 Multimedia video controller: Device 1a0a:6202 (rev 01)

 

Identify the Ethernet controller you wish to assign.  Note the PCI address for the device (from my list, it would be 00:19.0).  From my list, I only have one network card, so I shouldn't do this, but if you have multiple, either one SHOULD be fine to select.

 

3 - Type the following command:

 

lspci -n

 

00:00.0 0600: 8086:0c00 (rev 06)
00:01.0 0604: 8086:0c01 (rev 06)
00:01.1 0604: 8086:0c05 (rev 06)
00:02.0 0300: 8086:0412 (rev 06)
00:03.0 0403: 8086:0c0c (rev 06)
00:14.0 0c03: 8086:8c31 (rev 04)
00:16.0 0780: 8086:8c3a (rev 04)
00:19.0 0200: 8086:153b (rev 04)
00:1b.0 0403: 8086:8c20 (rev 04)
00:1c.0 0604: 8086:8c10 (rev d4)
00:1c.3 0604: 8086:8c16 (rev d4)
00:1f.0 0601: 8086:8c44 (rev 04)
00:1f.2 0106: 8086:8c02 (rev 04)
00:1f.3 0c05: 8086:8c22 (rev 04)
01:00.0 0300: 1002:68f9
01:00.1 0403: 1002:aa68
02:00.0 0300: 10de:1004 (rev a1)
02:00.1 0403: 10de:0e1a (rev a1)
04:00.0 0400: 1a0a:6202 (rev 01)

 

4 - Identify your network card by PCI address (first column of results).

5 - Obtain the vendor/product ID for that device from the last column.  00:19.0 from my example is 8086:153b.

6 - Edit your syslinux.cfg file and add the following after the append but before initrd=/bzroot.

pci-stub.ids=8086:153b

REPLACE THE VENDOR/PRODUCT ID FROM MY EXAMPLE ABOVE WITH THE ONE YOU OBTAINED IN STEP 5.

 

7 - Reboot your system.

8 - Edit your VM using the XML editor mode.

9 - Add the following between the <devices> and </devices> tags.

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
      </source>
    </hostdev>

 

Modify the address line entering in the two digit bus, slot, and function from your ID.  So 00:19.0 translates to what I have above.

 

Save the XML and start your VM.  All should be right as rain!

 

NOTE:  If you get an error, it could be because your NIC is in an IOMMU group with another in-use PCI device (either assigned to the host or to another VM).  In this instance, you can attempt to use the PCIE ACS Override option under the VM Manager settings page, but use of this toggle is considered experimental.

Share this post


Link to post

A easier method of discovering the right card to passthrough is running:

 

#!/bin/bash
for ETH in $(find /sys/devices/ -type d -iname "eth[0-9]"); do
  port=$(basename $ETH);
  export $(cat $ETH/device/uevent);
  link_status=$(cat $ETH/operstate);
  desc=$(lspci -d $PCI_ID| cut -d ':' -f3)
  pci_bus=$(echo $PCI_SLOT_NAME|sed -e 's#0000:##')
  echo -e "Found NIC: $port\n  Name:   $desc\n  PCI id:  $PCI_ID\n  PCI bus: $pci_bus\n  Status:  $link_status\n"
done

  • Like 1

Share this post


Link to post

A easier method of discovering the right card to passthrough is running:

 

#!/bin/bash
for ETH in $(find /sys/devices/ -type d -iname "eth[0-9]"); do
  port=$(basename $ETH);
  export $(cat $ETH/device/uevent);
  link_status=$(cat $ETH/operstate);
  desc=$(lspci -d $PCI_ID| cut -d ':' -f3)
  pci_bus=$(echo $PCI_SLOT_NAME|sed -e 's#0000:##')
  echo -e "Found NIC: $port\n  Name:   $desc\n  PCI id:  $PCI_ID\n  PCI bus: $pci_bus\n  Status:  $link_status\n"
done

If you use my method, it should not matter which card you use. It will hide the device from being usable by unraid and therefore by default the other device will become unRAID's NIC.

Share this post


Link to post

I know that. This code only shows you which NIC is eth(n) and the PCI id. It's designed to gather info for your method, not to replace it. If, for an example, I wish to passthrough the unconnected NIC of my board, I can easily do so.

 

It makes easier to pin a NIC to the physical port.

Share this post


Link to post

jonp, I was thinking, this method of yours, the two step passthrough, is much safer than the simple passthrough without the pci-stub step. This could make passthrough of USB/SATA/Ethernet pretty much doable via the VM Manager plugin. Is Eric working on something like this on the present moment?

 

PS: little code to find in which pci bus is unRAID's usb drive:

 

udevadm info -q path -n /dev/disk/by-label/UNRAID |grep -Po '0000:\K\w{2}:\w{2}\.\w{1}'

Share this post


Link to post

jonp, I was thinking, this method of yours, the two step passthrough, is much safer than the simple passthrough without the pci-stub step. This could make passthrough of USB/SATA/Ethernet pretty much doable via the VM Manager plugin. Is Eric working on something like this on the present moment?

 

PS: little code to find in which pci bus is unRAID's usb drive:

 

udevadm info -q path -n /dev/disk/by-label/UNRAID |grep -Po '0000:\K\w{2}:\w{2}\.\w{1}'

The logic work is partly done (which is why I have this method documented).  We have a few concepts as to how we would want to implement this, but the first question was, what's the use case? If its just pfSense, we could implement a VM template that let's the user specify this just in that template. If there are others, we may wish to implement a management tool for PCI Devices in general, but some PCI devices should never be stubbed such as DRAM controllers, root ports, and other general purpose devices. This would then take time for discovery and filtering.

 

Graphics and sound were logical first choices as unRAID OS didn't need them to operate...ever.  But as for other PCI devices, what would be the point?  "Just to do it" doesn't put this as a high priority feature. Archedraft found a good use case for NIC assignment with pfsense because of the fact that it doesn't have good support for VirtIO in the guest kernel. But beyond that, are there other viable use cases?

 

For USB, I could see this being valuable, but we are still testing and validating.  Need a way to determine which USB controller is in use by the boot device and prevent it from being stubbed. Otherwise you could get in a nasty situation where the boot process would keep crashing until you ejected the flash and manually edited your syslinux.cfg from another device to remove the stub (not ok to let that happen).  EDIT:  will test with your code provided to see if that gives us consistent results on various setups. Thanks!!

 

For storage, we also would need to understand the use cases for passing through an entire controller to a VM and what impact to other host storage performance that could have. Especially when another method could be to pass through just the block storage devices you want without passing through the entire controller.

 

So I guess my point here is that there are bigger fish to fry to get us to rc1 right now and more thought needs to go into deciding how to deal with the pass through of other PCI devices before we make it a priority to put in the time to code it. That said, if you have ideas you'd like to share, I think its pretty clear we love your input!

 

Share this post


Link to post

THANK YOU SOOOO MUCH FOR THIS!

 

I decided to migrate from Xen this weekend and started to think i wasn't going to find a thread on how to passthrough my NIC (like i did with Xen).

 

Perfect timing and works perfectly for me  :D

Share this post


Link to post

THANK YOU SOOOO MUCH FOR THIS!

 

I decided to migrate from Xen this weekend and started to think i wasn't going to find a thread on how to passthrough my NIC (like i did with Xen).

 

Perfect timing and works perfectly for me  :D

Your welcome!  Can I ask your use case?  Just curious if its pfsense or something else.

Share this post


Link to post

Sure,

 

Its partly to do with having VPN host software installed on the VM which my Windows 7 and Xen combo didn't like (BSD), so just stuck with it for KVM as well. Also i have the VM going into a different VLAN to unRAID, which wasn't the initial reason for the passthrough, but has subsequently allowed me to make some helpful network changes.

Share this post


Link to post

Hi,

 

i just tried to do those changes and facing now some issues after i amended the syslinux.cfg. My unRAID was not reachable after the reboot at all over the network until i removed the changes again.

my both NICs have the same vendor ID. the mainboard which i'm using its the ASRock - C2750D4I

 

 

Matt

Share this post


Link to post

I'm not sure if this is just me, or a fault, but i've just upgraded from beta15 to RC3 and passthrough no longer works.

 

I decided to rebuild my VM at the same time (after migrating primary functions to Docker) and when it comes to editing the XML i now get the following error...

 

Error creating VM: (domain_definition):98: attributes construct error

        <address domain='0x0000' bus='0x05’ slot='0x00’ function='0x0'/>

----------------------------------------------------^

 

The pic-stub entry is still in my syslinux.cfg and is correct. I have rebooted the server and VM and tried again, but the same error still pops up. The VM, other than this, is working perfectly though.

 

Any thoughts?

 

Thank you,

 

Rich

Share this post


Link to post

Anyone got any ideas on this please? :(

Share this post


Link to post

Anyone got any ideas on this please? :(

Please create a new post and include your xml for the VM.

Share this post


Link to post

Is this still a "thing"? We were told to not try to manually edit any config files or xml files so real world results can be shown and then fixed.

Share this post


Link to post

Is this still a "thing"? We were told to not try to manually edit any config files or xml files so real world results can be shown and then fixed.

 

Assigning physical NICs to VMs is still not officially supported and won't be for unRAID 6.0.  This guide is for advanced users that want to do it anyway, at their own risk.  There are multiple reasons why which I have spoken on here before about.  This pertains to SATA and USB controllers as well.

Share this post


Link to post

I am trying to edit my xml file by adding

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
      </source>
    </hostdev>

 

and I get.

 

error creating vm: can not creat file '/ect/libvirt/qemu/
windows.xml.new': Read-only File system

 

I am trying to add a pcie TV tuner to my Windows 8.1 VM

Share this post


Link to post

I am trying to edit my xml file by adding

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
      </source>
    </hostdev>

 

and I get.

 

error creating vm: can not creat file '/ect/libvirt/qemu/
windows.xml.new': Read-only File system

 

I am trying to add a pcie TV tuner to my Windows 8.1 VM

Can you show me the line from Tools --> system devices --> PCI devices for the tuner?

Share this post


Link to post

0a:00.0 Multimedia controller: Philips Semiconductors SAA7164 (rev 81)

 

Here is my XML file I am trying to edit

 

<domain type='kvm'>
  <name>Windows</name>
  <uuid>79bf8325-47e9-8d99-4794-5149debe8f0c</uuid>
  <metadata>
    <vmtemplate name="Custom" icon="windows.png" os="windows"/>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <nosharepages/>
    <locked/>
  </memoryBacking>
  <vcpu placement='static'>1</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.3'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='1' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/cache/vdisks/Windows 8.1/win81.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:02:79:19'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/Windows.org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='vmvga' vram='16384' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
      </source>
    </hostdev>
  </devices>
</domain>

 

My problem is that I can not save the file.  What I don't understand is that I was able to edit it in the past.

Share this post


Link to post

Of course it has nothing to do with the device you are trying to pass through.. Just me being blind.

Are you trying to edit the XML file through the webgui or command line?

It looks like you might have a problem with the domain.img file that contains the VM settings.

You can try to copy the XML you need and then rename the domain.img file so a new one is created.

If I remember correctly you have to deactivate libvirt in settings before you rename domain.img.

Share this post


Link to post

I was using the Web gui, but like I said I have made other changes to the xml recently without issue.  What is the domain.img?

Share this post


Link to post

I was using the Web gui, but like I said I have made other changes to the xml recently without issue.  What is the domain.img?

But no more changes after you got the error message?

What happens if you try to create a new VM?

Share this post


Link to post

I can't make any changes by editing the xml after I got the error.  I'll try editing the VM with the Web gui by changing one of the drop down opinions and see if that will change the xml.

 

I have not tried creating a new VM yet.

Share this post


Link to post

I just tried creating a new VM and get the same error.  Also I tried editing the current VM using the gui VM setup wizard and same error.  I suspect something happened with my flash drive.  Should I try copying over a fresh copy of unraid or is there another option? 

Share this post


Link to post

Shutdown your system and then plug your flash device into a PC and run a check disk, repairing any errors found.

Share this post


Link to post

Hello, you mention VirtIO is faster than directly passing through NIC's in most instances. I have a system with dual Intel NIC's on the mobo. I was intending on setting one for unraid, and passing the second to a Windows VM which hosts Plex, and numerous other things. So it is advisable, and faster, to turn bridge on and set my VM to it? (br0).

 

I had tried this, but wasn't given access to the net on my VM, however it did assign me the proper IP(based on a static IP to the MAC I set, router handles it)

 

I believe I read I need to turn teaming on, is any teaming more beneficial than another? I have like 6 options...

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


Copyright © 2005-2018 Lime Technology, Inc.
unRAID® is a registered trademark of Lime Technology, Inc.