A brief synopsis of KVM, HVM, and VFIO


jonp

Recommended Posts

Generally speaking, there isn't a ton of easy-to-digest information on the web about the KVM hypervisor, VFIO, and IOMMU, so I thought I'd write up a brief post on these technologies to educate those that are interested.  The goal of this post is to write as little as possible while conveying as much as possible, so yes, I'm going to skip over a lot of the boring details and jump into the most important facts.

 

What is a Hypervisor?

KVM is a component in the Linux kernel that allows it to act as a hypervisor.  Simply put, hypervisors are responsible for monitoring and managing the resources allocated to virtual machines.  Virtual machines are emulated computers that run on top of a physical computer.  Ever wanted to run three operating systems on one computer all at the same time?  Hypervisors make this possible.  Examples of other hypervisors include Xen, VMWare (ESX/ESXi), VirtualBox, and Microsoft Hyper-V. 

 

So what's unique about KVM?

Unlike other hypervisors, KVM is the only one that is built directly into and supported by the Linux kernel itself.  All other type-1 hypervisors out there will load before Linux does, and then Linux runs in an underprivileged state to that hypervisor.  By leveraging a hypervisor that is part of the Linux kernel itself, it means better support, less complexity, and more room for optimization improvements.

 

What is QEMU?

KVM is just the component in the kernel that manages / monitors resources allocated to virtual machines.  The other half of that puzzle is in the emulation of hardware components such as a motherboard, CPU, and various controllers that make up a virtual machine.  That's where QEMU comes in (short for quick emulator).  KVM can't work without QEMU, so you'll often times see KVM referred to as QEMU/KVM or KVM/QEMU.  While QEMU is required to make use of KVM, other hypervisors also use QEMU including Xen and VirtualBox.

 

What's HVM?

When virtual machine technology was first starting to grow in adoption, it wasn't directly supported by the chipset manufacturers directly.  As such, there was a significant amount of overhead associated with virtual machines due to software emulation.  Later, Intel and AMD built support for virtualization directly into their hardware (Intel VT-x and AMD-v), reducing overhead for emulation, monitoring, and security.  These technologies allow for the creation of hardware assisted virtual machines (referred to as HVMs).  While Xen offers a way to eliminate the overhead associated with traditional emulation without the need for VT-x, this paravirtualization method only works with Linux-based guests, and for that, Docker Containers are a better solution anyway.  As such, HVMs are best suited for virtual machines where more than just basic Linux applications are needed.

 

What is VFIO?

Simply put, virtual function IO allows us to assign a physical device, such as a graphics card, directly to a virtual machine that in turn will provide driver support for the device directly.  But that's not all.  We can also prevent the device from accessing spaces in memory that are outside of that VM.  This means that if something goes wrong with the device or it's driver, the impact of such an event is limited to the virtual machine and not the host.  This shields unRAID OS from being exposed to unnecessary risk due to faulty hardware or misbehaving device drivers.  VFIO usage requires IOMMU capable hardware (your CPU must have Intel VT-d or AMD-Vi support).

 

Questions?

Like I said at the beginning, I wanted to keep this post as short and direct as I could to get the most important points across without making folks go crosseyed while reading.  If you have any questions, please reply here with them!

  • Like 6
  • Thanks 1
  • Upvote 1
Link to comment

Thanks for that jonp, I kind of had a handle on these things but that has definitely helped clarify everything for me.  I do believe that this thread should be stickied as it's a good introduction to what is in some ways a more advanced facet of using Unraid.

Link to comment

What's needed is a step by step walkthrough of building a vm using the unraid tools. I tried ... abd failed when it came to xml entries. I've no vackground in linux and it seems that therefore i an unable to participate in this new foubd enlightenment that is unraid virtualisation.

There's a plugin that should help with plain unRAID (non xen)

http://lime-technology.com/forum/index.php?topic=35858.0

Link to comment

What's needed is a step by step walkthrough of building a vm using the unraid tools. I tried ... abd failed when it came to xml entries. I've no vackground in linux and it seems that therefore i an unable to participate in this new foubd enlightenment that is unraid virtualisation.

Stay tuned. I am working on one

Link to comment

This post feels like useful information.    However rather than making it a stickied post here would it not be more useful in the  unRAID wiki (possibly linked to from here).  That would make it easier to maintain, and also to provide linked information about other aspects of virtualisation on unRAID.  It could provide part of the introduction to a more complete User Guide to virtualisation on unRAID.

Link to comment

What's needed is a step by step walkthrough of building a vm using the unraid tools. I tried ... abd failed when it came to xml entries. I've no vackground in linux and it seems that therefore i an unable to participate in this new foubd enlightenment that is unraid virtualisation.

There's a plugin that should help with plain unRAID (non xen)

http://lime-technology.com/forum/index.php?topic=35858.0

Doesnt seem to help me with what is needed as far as xml entries go? Even a hands-on ... 'what is xml and why is it necessary' would help. I've no idea what us needed here and just merrily 'lifted' other peoples files in ignorance. Needless to say that didnt work and i gave up.

Link to comment

What's needed is a step by step walkthrough of building a vm using the unraid tools. I tried ... abd failed when it came to xml entries. I've no vackground in linux and it seems that therefore i an unable to participate in this new foubd enlightenment that is unraid virtualisation.

There's a plugin that should help with plain unRAID (non xen)

http://lime-technology.com/forum/index.php?topic=35858.0

Doesnt seem to help me with what is needed as far as xml entries go? Even a hands-on ... 'what is xml and why is it necessary' would help. I've no idea what us needed here and just merrily 'lifted' other peoples files in ignorance. Needless to say that didnt work and i gave up.

 

This post from a while back was an intro to the basics and provided sample XML:

 

http://lime-technology.com/forum/index.php?topic=35732.msg332898#msg332898

 

I just stickied it because it probably should have been all along.

 

That said, wouldn't it be better if you don't need to do ANYTHING with XML ever again?  Maybe something like this ;-)

 

createvm-e1426086502956.png

 

Coming very soon ;-)

Link to comment

What's needed is a step by step walkthrough of building a vm using the unraid tools. I tried ... abd failed when it came to xml entries. I've no vackground in linux and it seems that therefore i an unable to participate in this new foubd enlightenment that is unraid virtualisation.

There's a plugin that should help with plain unRAID (non xen)

http://lime-technology.com/forum/index.php?topic=35858.0

Doesnt seem to help me with what is needed as far as xml entries go? Even a hands-on ... 'what is xml and why is it necessary' would help. I've no idea what us needed here and just merrily 'lifted' other peoples files in ignorance. Needless to say that didnt work and i gave up.

 

This post from a while back was an intro to the basics and provided sample XML:

 

http://lime-technology.com/forum/index.php?topic=35732.msg332898#msg332898

 

I just stickied it because it probably should have been all along.

 

That said, wouldn't it be better if you don't need to do ANYTHING with XML ever again?  Maybe something like this ;-)

 

createvm-e1426086502956.png

 

Coming very soon ;-)

 

You tease! Looks great though can't wait...

Link to comment

What's needed is a step by step walkthrough of building a vm using the unraid tools. I tried ... abd failed when it came to xml entries. I've no vackground in linux and it seems that therefore i an unable to participate in this new foubd enlightenment that is unraid virtualisation.

There's a plugin that should help with plain unRAID (non xen)

http://lime-technology.com/forum/index.php?topic=35858.0

Doesnt seem to help me with what is needed as far as xml entries go? Even a hands-on ... 'what is xml and why is it necessary' would help. I've no idea what us needed here and just merrily 'lifted' other peoples files in ignorance. Needless to say that didnt work and i gave up.

 

This post from a while back was an intro to the basics and provided sample XML:

 

http://lime-technology.com/forum/index.php?topic=35732.msg332898#msg332898

 

I just stickied it because it probably should have been all along.

 

That said, wouldn't it be better if you don't need to do ANYTHING with XML ever again?  Maybe something like this ;-)

 

createvm-e1426086502956.png

 

Coming very soon ;-)

 

 

the "template" section looks interesting, what kind of options will there be ?

 

Link to comment

I assume that the screenshots are from an enhanced version of the VM Manager plugin (the one currently stickied at the top of this forum area) that is going to be included as part of the base unRAID release? 

 

Only mentioning that as you can already avoid having to manually manipulate XML in most cases using the current plugin.  Using the current plugin might have been enough to get superloopy1 going without the need to get involved in XML.

Link to comment

What's needed is a step by step walkthrough of building a vm using the unraid tools. I tried ... abd failed when it came to xml entries. I've no vackground in linux and it seems that therefore i an unable to participate in this new foubd enlightenment that is unraid virtualisation.

There's a plugin that should help with plain unRAID (non xen)

http://lime-technology.com/forum/index.php?topic=35858.0

Doesnt seem to help me with what is needed as far as xml entries go? Even a hands-on ... 'what is xml and why is it necessary' would help. I've no idea what us needed here and just merrily 'lifted' other peoples files in ignorance. Needless to say that didnt work and i gave up.

 

This post from a while back was an intro to the basics and provided sample XML:

 

http://lime-technology.com/forum/index.php?topic=35732.msg332898#msg332898

 

I just stickied it because it probably should have been all along.

 

That said, wouldn't it be better if you don't need to do ANYTHING with XML ever again?  Maybe something like this ;-)

 

createvm-e1426086502956.png

 

Coming very soon ;-)

Looking good.

Link to comment

the "template" section looks interesting, what kind of options will there be ?

 

Initially there will be only three template types:  Windows, OpenELEC, and Other.  The entire look/presentation of the template controls will change based on the template selected.  OpenELEC will be a very short list of configurable options and a download button to pull the VM down from our host on Amazon automatically.  You can probably guess at some of the others that we would want to add to this list...

Link to comment

Purely subjectively speaking, whilst the list could stand a few more entries I feel you have to strike the balance right between choice and overwhelming less techie types.

 

The "Other" option will be a generic template where the techies can build a VM from scratch using an ISO to install an OS.  A lot of tunable settings will be made available under an advanced view here to make the more savvy users who want to roll their own OS very happy ;-).  The Windows template is similar to Other, but is tuned specially for Windows and has smart controls so if you assign an NVIDIA GPU, the hyper-v settings will automatically be disabled, solving the Code 43 issue (although you can manually turn them back on if you intend to use the 340.52 drivers).

Link to comment

Sounds good, catering to the users that will want passthrough etc but really don't want to get their hands dirty or don't care how it works etc... then for the more adventerous the other tab lets you have some free reign to play with config etc ... , whilst for the ultra techy there's still the roll your own xml option.

 

 

 

 

Link to comment

Sounds good, catering to the users that will want passthrough etc but really don't want to get their hands dirty or don't care how it works etc... then for the more adventerous the other tab lets you have some free reign to play with config etc ... , whilst for the ultra techy there's still the roll your own xml option.

 

That's what we're going for here.  Make it simple for those that want simple, but tunable for those that want to tweak settings.

 

We even updated the XML editor to have autocomplete for commonly used tags ;-)

Link to comment

This looks really good! I love where you are heading with this. Smart and simple VM creation with abilities for advanced uses too. Are you planning on adding CPU host passthrough, cpuset and pinning? I think those options and with what you have shown in the picture will cover 95% of the manual xml creation.

 

Hot plugging USB devices would be great too. That would kill the need for USB bus passthrough.

 

Gary

Link to comment

This looks really good! I love where you are heading with this. Smart and simple VM creation with abilities for advanced uses too.

 

Thanks!  Glad you like it!

 

Are you planning on adding CPU host passthrough, cpuset and pinning? I think those options and with what you have shown in the picture will cover 95% of the manual xml creation.

 

Actually it's a default setting now when you create VMs with this to work that way.  There's really no reason to use any other setting that host-passthrough unless you're trying to do high-availability failover which isn't a feature we're even focused on right now.  So when you set 2 vcpus in our config, it will pin them automatically.  We need to still add some intelligence / controls for customizing those settings within the webUI, but at least the defaults are easy to tweak in XML.

 

Hot plugging USB devices would be great too. That would kill the need for USB bus passthrough.

 

Gary

 

Two things:  USB 3 virtual chipset support (model=nec-xhci) and hotplugging USB is something on the roadmap, but won't make it in the first iteration of this new manager.

Link to comment

First off, many thanks to Jonp for this synopsis and for the sneak peek at the vm management utility to come.  As someone with no prior experience with vms this kind of overview is extremely helpful and I expect I'll be relying on that utility a great deal.

 

That said, can anyone help me lock in on the best approach for what I have in mind?  I just want to run a full Ubuntu desktop as the guest vm, to use mostly as HTPC (Plex Home Theater) but to occassionally remote in to run Handbrake and some other video programs, and I also want to run Plex server.  The Plex server part by itself is easy since I know I can just use Docker for that.  But for an Ubuntu desktop should I looking to KVM or Xen?  The graphics card I want to pass through is an HD 6450 (until IGP passthrough is possible then I'd just use the intel HD graphics w/my i7-4790).  Then whether it's Xen or KVM, should I stick with using Docker for Plex server or would there by any advantage to running it in the guest vm instead, since I'm setting that up anyway?

 

BTW I love the idea of pre-built vms for Openelec and Windows and I wonder if you might want to consider Ubuntu as another?  I think a lot of folks might find it useful to have a pre-built Windows alternative that offers more functionality than Openelec.  Just a thought. 

Link to comment

First off, many thanks to Jonp for this synopsis and for the sneak peek at the vm management utility to come.  As someone with no prior experience with vms this kind of overview is extremely helpful and I expect I'll be relying on that utility a great deal.

 

Glad to hear it and I am working on a lot more content that we'll be able to release once the manager is done.  I really want this to be stupid simple for folks so anywhere we can eliminate telling someone to "go to google and learn," we will.

 

That said, can anyone help me lock in on the best approach for what I have in mind?  I just want to run a full Ubuntu desktop as the guest vm, to use mostly as HTPC (Plex Home Theater) but to occassionally remote in to run Handbrake and some other video programs, and I also want to run Plex server.  The Plex server part by itself is easy since I know I can just use Docker for that.  But for an Ubuntu desktop should I looking to KVM or Xen?  The graphics card I want to pass through is an HD 6450 (until IGP passthrough is possible then I'd just use the intel HD graphics w/my i7-4790).  Then whether it's Xen or KVM, should I stick with using Docker for Plex server or would there by any advantage to running it in the guest vm instead, since I'm setting that up anyway?

 

For the Ubuntu desktop, I would probably go with KVM, but if your GPU doesn't work, you could always give Xen a whirl.  You should stick with Docker for all Linux headless / server applications and reserve virtual machines for non-Linux guests (e.g. Windows) and desktop applications.

 

BTW I love the idea of pre-built vms for Openelec and Windows and I wonder if you might want to consider Ubuntu as another?  I think a lot of folks might find it useful to have a pre-built Windows alternative that offers more functionality than Openelec.  Just a thought.

 

To be clear, OpenELEC is the only VM for which we plan to offer in a pre-built state.  The Windows template we have created isn't a pre-built VM, just a pre-built configuration so that it's optimized for Windows as opposed to Linux (there are differences in the XML used to generate the VM through libvirt, so we optimize the XML for the guest operating system to be loaded).  As far as pre-building / making other VMs available, stay tuned...

Link to comment

For the Ubuntu desktop, I would probably go with KVM, but if your GPU doesn't work, you could always give Xen a whirl.  You should stick with Docker for all Linux headless / server applications and reserve virtual machines for non-Linux guests (e.g. Windows) and desktop applications.

 

Thanks Jon.  That's what I figured for Plex server but I appreciate the confirmation.  I'll also try to KVM to begin with for the Ubuntu guest vm, though it's nice to know Xen might work too if I run into any problems.  (I actually bought the HD 6450 thinking that Xen was going to be the better option for me and seeing other users report VGA passthrough in Xen working with that card.  I also have a 750 ti which I assumed wouldn't work with either but if Nvidia passthrough is going to work with KVM I may choose to start with that instead.)

 

To be clear, OpenELEC is the only VM for which we plan to offer in a pre-built state.  The Windows template we have created isn't a pre-built VM, just a pre-built configuration so that it's optimized for Windows as opposed to Linux (there are differences in the XML used to generate the VM through libvirt, so we optimize the XML for the guest operating system to be loaded).  As far as pre-building / making other VMs available, stay tuned...

 

Thanks for the clarification and can't wait to see what's in the pipeline.  This is really exciting stuff. 

Link to comment
  • 1 month later...

Are you guys aware of any resource or list for vt-d compatible hardware?

 

I can easily find out whether the processors support it but haven't had luck about motherboards.

 

Very confusing info out there, I read about how some mobos have hardware with vt-d capability but their firmware doesn't support it. So frustrating.

 

Thanks

Link to comment

Are you guys aware of any resource or list for vt-d compatible hardware?

 

I can easily find out whether the processors support it but haven't had luck about motherboards.

 

Very confusing info out there, I read about how some mobos have hardware with vt-d capability but their firmware doesn't support it. So frustrating.

 

Thanks

This is your best bet for GPU passthrough:

 

https://docs.google.com/spreadsheet/lv?key=0Aryg5nO-kBebdFozaW9tUWdVd2VHM0lvck95TUlpMlE&usp=drive_web#gid=0

 

Its not the best thing in the world, but its better than nothing. Its a spreadsheet being maintained with those who have succeeded passing through various GPUs on various hardware configs. Has motherboard, processor, and GPU info.

 

FYI, if GPU pass through works, you can count on any non GPU PCI devices to work.

Link to comment
  • itimpi unpinned this topic

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.