Problem passing through Intel PRO/1000 Dual NIC


Recommended Posts

To start with I have successfully passed through GPU's / USB controllers so I know that my server is able. Also, my unRAID server is using the onboard LAN port.

 

So, I installed the NIC card and turn on the server and everything (including VM's) works just fine. I then add 0000:04:00.0 & 0000:04:00.1 to my vfio-bind command in my go file and reboot the server. The server and VM's start fine; however, I no longer can access unRAID through the web or SSH. The weird thing is my Windows VM can access the internet just fine. Why am I losing connectivity with unRAID when I pass through a PCIe NIC card??

Link to comment

SO I have noticed what I think is strange behavior. When I start my server without the Dual NIC installed my server starts and has a MAC address of AA:AA:AA:AA:AA:AA:AA. When I put install my Dual NIC card and start unRAID, the MAC address changes significantly to BB:BB:BB:BB:BB:BB:BB. The Ethernet cord is still plugged into the same onboard LAN port so why is would its MAC address change?

 

Also, I have found that if I

virsh nodedev-detach pci_0000:04:00.0

I can then pass through the device to my VM; however, if I add that code to vifo-bind in my GO file then I loose access to unRAID...

 

I'm so confused!

Link to comment

I noticed something similar when I tried a dual-nic card on my UnRAID machine.  Didn't really mess around with anything and all three Ethernet cables were plugged in and I would find that although I have a reserved address in my router for the onboard NIC sometimes Unraid was using one of the dual NICs as it's primary Ethernet adapter and therefore my MAC and IP address of my Unraid box would change. 

 

I didn't get any further investigating the issue as all I really wanted to know was that the card was working.  I do intend at some point to run a VM with PFSense to experiment with, but it'll be a good while before I get to trying that.

 

I know that hasn't helped solve your problem at all, but sometimes it's nice to know you're not the only one!

 

My observation led me to believe that Unraid randomly selects a NIC on boot to use, my plan was to try and bind a specific controller at boot time, but I don't yet know how to go about it.

 

Good Luck though!

Link to comment

Thanks for the reply! It is nice to know that I am not the only one. The thing I find weird is currently I only have 1 Ethernet cable plugged into my server and the dual NIC card ports are both unplugged, so why would it change its MAC address if it only has access to the onboard port? I wonder if there is a way to bind unRAID to only use a certain network port?

Link to comment

Well I think that a MAC address is hardware bound and not dependent on whether an IP address is allocated or not.  So Unraid boots, sees three NICs, randomly chooses one as the primary Ethernet whether it has an IP address or not, hence you're seeing a change in the MAC address you expect and not able to access the server as it's not on the LAN and doesn't have a IP address.

Link to comment

UPDATE: I removed the br0 setup and modified my GO file to create a out0 and in0 bridge. pfSense 2.2 is up and running.

 

I have noticed one issue so far:

The virtIO network drivers need some work (my upload speed is 0.03 Mbps ... SO BAD) I changed the network model type to e1000 and that fixes the upload issue. I'll have to wait until the pfSense team has time to fix the virtIO driver before I switch back.

 

Other than that it seems to be stable.

 

Link to comment
  • 3 weeks later...

Got around to putting my dual NIC in today and found the exact same behaviour. The MAC address changed, I think k I got a bit confused when I read your post but now I get what you mean. 

 

Did you ever find the reason for this?

Nope, but I have been running pfSense as a VM for the last couple of weeks. Haven't had any issues with it so that has been nice. I also changed the the e1000 drivers back to VirtIO and my upload is working as it should... Not sure why it's working now though but I am not complaining.

Link to comment

Nope, but I have been running pfSense as a VM for the last couple of weeks. Haven't had any issues with it so that has been nice. I also changed the the e1000 drivers back to VirtIO and my upload is working as it should... Not sure why it's working now though but I am not complaining.

 

Cool, mind posting that bit of your go file, I'd be interested to see how you did that.

Link to comment

archedraft - I'm hoping to start setting up a PfSense VM tonight using bridges.

 

I plan to do basically the same thing you've done. Have you seen any problems with allowing your VM's to use eth2 br0? Is this port connected to a switch for other machines to use as the connection to pfsense? Does each VM get an IP address from PfSense?

 

Thanks

 

 

Link to comment

archedraft - I'm hoping to start setting up a PfSense VM tonight using bridges.

 

I plan to do basically the same thing you've done. Have you seen any problems with allowing your VM's to use eth2 br0?

With mine I have my modem connected to out0 and then all my VM's use in0. The VM's haven't had any issues that I have noticed.

 

Is this port connected to a switch for other machines to use as the connection to pfsense?

My in0 port's ethernet connects to a switch and then that switch is used for other computers (including unRAID as eth0). All the machines connected to the switch and VM's are controlled by the pfSense VM.

 

Does each VM get an IP address from PfSense?

Yup

Link to comment

archedraft - I'm hoping to start setting up a PfSense VM tonight using bridges.

 

I plan to do basically the same thing you've done. Have you seen any problems with allowing your VM's to use eth2 br0?

With mine I have my modem connected to out0 and then all my VM's use in0. The VM's haven't had any issues that I have noticed.

 

Is this port connected to a switch for other machines to use as the connection to pfsense?

My in0 port's ethernet connects to a switch and then that switch is used for other computers (including unRAID as eth0). All the machines connected to the switch and VM's are controlled by the pfSense VM.

 

Does each VM get an IP address from PfSense?

Yup

 

Easy enough, I have PfSense up and running using bridges. Tomorrow I add my new switch and access point. Thanks for the info!

Link to comment

archedraft - I'm hoping to start setting up a PfSense VM tonight using bridges.

 

I plan to do basically the same thing you've done. Have you seen any problems with allowing your VM's to use eth2 br0?

With mine I have my modem connected to out0 and then all my VM's use in0. The VM's haven't had any issues that I have noticed.

 

Is this port connected to a switch for other machines to use as the connection to pfsense?

My in0 port's ethernet connects to a switch and then that switch is used for other computers (including unRAID as eth0). All the machines connected to the switch and VM's are controlled by the pfSense VM.

 

Does each VM get an IP address from PfSense?

Yup

 

Easy enough, I have PfSense up and running using bridges. Tomorrow I add my new switch and access point. Thanks for the info!

 

I'm confused now I'm starting my setup, so you guys are no longer passing through the NIC but instead setting up bridges in Unraid to be used in the VMs?

Link to comment

I'm confused now I'm starting my setup, so you guys are no longer passing through the NIC but instead setting up bridges in Unraid to be used in the VMs?

 

Thats exactly what I'm doing, my board doesn't support VT-D so I am using bridges instead. Its only been a couple of days, but so far it seems stable enough to replace my router, for home use at least.

 

Link to comment
  • 2 weeks later...

Yeah, I could not successfully pass through the NIC because the unRAID web GUI would fail to load.

 

Right, not sure whether to start a new thread for this but got around to trying it at last.

 

Got my bridges all set up as your config.  Confirmed they're both working by using them in Windows VMs

 

When I come to install PFsense it can't find the WAN interface.  Tried to load the drivers but it couldn't find them either.

 

Running

 

brctl show

in0             8000.0015176fd75a       yes             eth1
                                                        vnet0
                                                        vnet2
out0            8000.0015176fd75b       yes             eth2
                                                        vnet1

 

Tried entering both out0 and vnet1 as WAN names but no luck and it's not automatically discovering it either.

 

The WAN interface is connected to an unmanaged switch which then goes to my modem/router.

 

Any ideas where I'm going wrong?

 

EDIT: Here's a copy of my xml entry for completeness.

 

    <interface type='bridge'>
      <mac address='52:54:00:a0:18:6e'/>
      <source bridge='out0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:06:43:3a'/>
      <source bridge='in0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/>
    </interface>

Link to comment

When I come to install PFsense it can't find the WAN interface.  Tried to load the drivers but it couldn't find them either.

So when you try to assign devices on pfsense it does not show 2 available? pfsense should give you the option to assign vtnet0 and vtnet1 as your 2 ethernet ports.

 

Or do you mean that pfsense is not giving you an external ip address after you assign the 2 ethernet ports?

Link to comment

I'm not given any opportunity to assign vtnet0 or vtnet1,

 

Boot the iso to a launcher.  Checked all the option, tried to use multi boot and to install it by pressing i.

 

Clearly I'm missing something.

 

EDIT: Just checked, during the install it states no interfaces found, do I need to load any drivers?

 

 

Link to comment

OK, getting somewhere now, my problem was with my xml.

 

Once archedraft made me realise that I should have the option to assign vtnet0 & 1 and I saw the no interfaces found message I did a bit more experimenting,

 

Changing my xml to

 

    <interface type='bridge'>
      <mac address='52:54:00:a0:18:6e'/>
      <source bridge='out0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

 

and only using out0 pfsense was able to see the interface and I could assign it.

Now I'm left with the problem of how to define in0

 

I tried

 

    <interface type='bridge'>
      <mac address='52:54:00:06:43:3a'/>
      <source bridge='in0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/>
    </interface>

 

But that gives me an error and won't save.

 

The bridges are bound to two ports on a dual NIC, and I know I'm just getting confused conceptually.  I just can't see where.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.