[SOLVED] running unRAID on a KVM virtual machine with virtio net+block drivers

I am on an experiment at the moment, where I am about to run unRAID inside a KVM virtual machine.

The disks need to be passed through via the VIRTIO block device driver.

 

I have everything up on unRAID 5.0b7 with a custom kernel.

The Flash is passed through OK, eth0 via virtio-net is also running fine.

Now I have passed through three disks via VIRTIO block driver.

The unRAID VM kernel can see them fine, however emhttp is not

presenting them in the UI.

 

The disks are listed in /dev/disk/by-path but not in /dev/disk/by-id.

The naming vor VIRTIO block devices is vdX instead of sdX for scsi block devices.

 

How can I trick unRAID/emhttp to see them?

 

Here's the content of /dev/disk/by-path:

drwxr-xr-x 2 root root 240 2011-06-12 20:22 ./
drwxr-xr-x 6 root root 120 2011-06-12 20:22 ../
lrwxrwxrwx 1 root root   9 2011-06-12 20:22 pci-0000:00:01.1-scsi-0:0:0:0 -> ../../sda
lrwxrwxrwx 1 root root  10 2011-06-12 20:22 pci-0000:00:01.1-scsi-0:0:0:0-part1 -> ../../sda1
lrwxrwxrwx 1 root root   9 2011-06-12 20:22 pci-0000:00:01.2-usb-0:2.1:1.0-scsi-0:0:0:0 -> ../../sdb
lrwxrwxrwx 1 root root  10 2011-06-12 20:22 pci-0000:00:01.2-usb-0:2.1:1.0-scsi-0:0:0:0-part1 -> ../../sdb1
lrwxrwxrwx 1 root root   9 2011-06-12 20:22 pci-0000:00:04.0-virtio-pci-virtio1 -> ../../vda
lrwxrwxrwx 1 root root  10 2011-06-12 20:22 pci-0000:00:04.0-virtio-pci-virtio1-part1 -> ../../vda1
lrwxrwxrwx 1 root root   9 2011-06-12 20:22 pci-0000:00:05.0-virtio-pci-virtio2 -> ../../vdb
lrwxrwxrwx 1 root root  10 2011-06-12 20:22 pci-0000:00:05.0-virtio-pci-virtio2-part1 -> ../../vdb1
lrwxrwxrwx 1 root root   9 2011-06-12 20:22 pci-0000:00:06.0-virtio-pci-virtio3 -> ../../vdc
lrwxrwxrwx 1 root root  10 2011-06-12 20:22 pci-0000:00:06.0-virtio-pci-virtio3-part1 -> ../../vdc1

 

...and /dev/disk/by-id:

drwxr-xr-x 2 root root 160 2011-06-12 20:22 ./
drwxr-xr-x 6 root root 120 2011-06-12 20:22 ../
lrwxrwxrwx 1 root root   9 2011-06-12 20:22 ata-QEMU_HARDDISK_QM00001 -> ../../sda
lrwxrwxrwx 1 root root  10 2011-06-12 20:22 ata-QEMU_HARDDISK_QM00001-part1 -> ../../sda1
lrwxrwxrwx 1 root root   9 2011-06-12 20:22 scsi-SATA_QEMU_HARDDISK_QM00001 -> ../../sda
lrwxrwxrwx 1 root root  10 2011-06-12 20:22 scsi-SATA_QEMU_HARDDISK_QM00001-part1 -> ../../sda1
lrwxrwxrwx 1 root root   9 2011-06-12 20:22 usb-Lexar_JD_FireFly_AAV1SOQ525O5N61S-0:0 -> ../../sdb
lrwxrwxrwx 1 root root  10 2011-06-12 20:22 usb-Lexar_JD_FireFly_AAV1SOQ525O5N61S-0:0-part1 -> ../../sdb1

 

...and the content of /proc/partitions:

major minor  #blocks  name

254        0 1953514584 vda
254        1 1953514552 vda1
254       16  976762584 vdb
254       17  976762552 vdb1
254       32  732574584 vdc
254       33  732574552 vdc1
  8        0     128000 sda
  8        1     127984 sda1
  8       16    7831552 sdb
  8       17    7830428 sdb1

 

Any help is very much appreciated.

 

TIA,

 

Ford

...OK, answering my own post.

I currently have a minimal 5.0b7 system running parity-rebuild at approx 54MB/sec on a KVM host  

 

Here's the short info:

 

- emhttp in its current version needs to be patched in order to recognize the vdX devices.

- the virtio-blk driver and current kvm/qemu system cannot generate a serial-id for the virtual disks.

 This results in an incomplete presentation in the /dev/disks/by-* directories upon boot.

 In order to fix this, you need a patched version of the udev-ruleset in /etc/udev/rules.d

-> both modifications result in a modified, custom bzroot file for your unRAID flash.

-> besides that, you'll need a custom kernel as stated in the OP.

 

If the forum is interested and if the mods approve (because patching emhttp is....well,

considered bad manners, right? ), I can prepare a write-up on how-to achieve things.

 

I will approach limetech to include the modifications needed in future versions.

I, for one, would very much like to see your method documented and I think the knowledge base here could only be enhanced.

- emhttp in its current version needs to be patched in order to recognize the vdX devices.

- the virtio-blk driver and current kvm/qemu system cannot generate a serial-id for the virtual disks.

 This results in an incomplete presentation in the /dev/disks/by-* directories upon boot.

Far easier is for you to link the vDX devices to alternate devices named as sdX devices.  Then you do not need to patch udev rules or emhttp.

 

I''ve not tried it recently, but I did something similar years ago to get a USB drive recognized.

Far easier is for you to link the vDX devices to alternate devices named as sdX devices.   Then you do not need to patch udev rules or emhttp.

 

I''ve not tried it recently, but I did something similar years ago to get a USB drive recognized.

 

Yes, I thought into that direction as well, but it did not work out.

Ultimately, I think, a udev rule is the most flexible part because it will populate the disks correctly (i.e. when unRAID is creating partitions on them).

This is my current /dev/disks/by-id layout:

root@Tower:~# ls -l /dev/disk/by-id/
total 0
lrwxrwxrwx 1 root root  9 2011-07-12 18:11 ata-QEMU_HARDDISK_QM00001 -> ../../hda
lrwxrwxrwx 1 root root  9 2011-07-12 18:11 scsi-pci-0000:00:04.0-virtio-pci-virtio1 -> ../../vda
lrwxrwxrwx 1 root root 10 2011-07-12 18:11 scsi-pci-0000:00:04.0-virtio-pci-virtio1-part1 -> ../../vda1
lrwxrwxrwx 1 root root  9 2011-07-12 18:11 scsi-pci-0000:00:05.0-virtio-pci-virtio2 -> ../../vdb
lrwxrwxrwx 1 root root 10 2011-07-12 18:11 scsi-pci-0000:00:05.0-virtio-pci-virtio2-part1 -> ../../vdb1
lrwxrwxrwx 1 root root  9 2011-07-12 18:11 usb-Lexar_JD_FireFly_AAV1SOQ525O5N61S-0:0 -> ../../sda
lrwxrwxrwx 1 root root 10 2011-07-12 18:11 usb-Lexar_JD_FireFly_AAV1SOQ525O5N61S-0:0-part1 -> ../../sda1

When I create /dev/sdX symlinks to the /dev/vdX devices and appropriate links of pattern "scsi-*" in /dev/disks/by-id/,

a vanilla emhttp would not see them.

Since I would never need /dev/hdX devices in my unRAID context, I decided to patch emhttp to see "vdX" instead of "hdX" devices.

 

I, for one, would very much like to see your method documented and I think the knowledge base here could only be enhanced.

 

...OK...it is summer vacation season and I probably don't have better

things to do when the kids are asleep 

...stay tuned.

Ford Prefect, you can contact Tom about patching the emhttp binary, if those modifications are simple, I doubt that he refuses to include them.

Ford Prefect, you can contact Tom about patching the emhttp binary, if those modifications are simple, I doubt that he refuses to include them.

 

Yes, thanks for the heads up.

I will approach Tom as soon as I have everything sorted out.

All things needed, except emhttp features, can be provides as a customization / from the community.

I just want to make sure I have my arguments right.

Hi, did you get anywhere with this?

 

I've been struggling with Xen for weeks now, but I'm willing to give KVM a try if it works.

 

The main problem I've been facing is that I can't get the USB stick correctly passed through to the Unraid VM, so I can't use my Pro license.

 

My plan was to pass through all my SATA controllers via PCI-passthrough. I tried the same thing for the usb controller but since I'm booting my Xen/dom0 from another USB stick, the system didn't like that at all and just hung.

 

I've ordered a seperate PCI USB card to see if that helps at all.

 

Failing that, I'm going to look into using unraid itself as a dom0.

 

Please let me know your progress! Thanks.

Hi,

 

not sure if you are talking about KVM or XEN here...since this thread is about unRAID on KVM, I'll assume that this is the case.

The main requirement, as I see it, is to employ a KVM based hypervisor solution instead of the ESXi one...

Before I respond, a few more words:

 

I stopped working on that route short after making the OP.

My first reason was ,that using Raw-Device maps instead of using PCIPassthrough for a controller would always be limiting the solution....

The second and main reason was, that my requirements changed....I needed an enterprise level encryption solution for my data....so I moved away from unRAID for that reason.

 

..now back to the OT...can it be done...sure!

 

I strongly recommend using ProxmoxVE as KVM hypervisor solution.

 

...using disks/drives:

PVE now has PCIPasstrough capabilities, as displayed here: http://forum.proxmox.com/threads/7725-Once-again-PCI-passthrough-problems

Use it to passthrough your HBA card, just like in an ESXi Build (note: you need a vt-d capable mobo and CPU for that)

 

...using/booting the unRAID flash:

This indeed is tricky, as PVE/KVM has not a good reputation for USB support.

At least it can pass on USB devices with USB1.1 speed...this should be OK for most usecases

I have not tested the PVE2.0beta, but with PVE1.9 I did not succeed to employ an external Boot-manager, like Plop

The best way I found to enable the unRAID flash was to convert the flash to a virtual disk (vmdk, but different volume name than the flash!!) and boot from that.

If you enable your flash via USB in KVM, the vmdk will re-mount the unRAID volume, which will be your flash at that time.

The how-to produce the vmdk from your flash is in this post: http://lime-technology.com/forum/index.php?topic=7914.0

 

...good luck!