Starting VMs independent of the [data disk] array

I started a short discussion in the 6.2.0-beta20 thread after asking:

 

Still in the hospital recovering from anaesthetic (stupid wisdom teeth) so have nothing to do but mess on iPad and read forums. Anyway, context is there as to why I can't just logon to my 6.2 test server to check this out myself:

 

Starting VMs independent of the array is a feature that has been requested allot and one I'd love to facilitate running pfsense. Did this "feature" make it into 6.2?

 

There were a few very quick responses so I quickly decided to move it over here in respect to the BETA release thread.

 

In essence I would like to ask about the possibility of including this configuration in a 6.2 BETA release of unRAID for evaluation / testing? I call it configuration and not feature as I "think" we are just talking about the starting and stopping of an existing feature.

 

I am sure this will generate some traffic as I believe it is something many users want.

 

In reply to my question, 2 suggestions as to how this could be implemented were:

 

No it has not (so far at least).

 

I already run my VMs from a disk external to the array, so would be quite happy it if there was a way to merely get unRAID to not close down VMs when the array stops.  I would be quite happy in the short term to handle this myself from the command line or the 'go' and 'stop' scripts run at system startup and shutdown.

 

Perhaps it can be considered by LT to have independent start/stop control of the cache drive/pool?

 

This would be more inline with the devices available in a plain unRAID configuration.

 

I thought both comments / suggestions were reasonable.

 

I guess what I am looking for here is if someone from LT has the time to respond as to the viability of what I am asking AND to generate further discussion from the wider community of course as we all know:

 

...

 

Hint: the more users can benefit from your feature, the more chance there is of it being accepted!

 

+ 1

 

my pfsense / clear OS router needs this ability.

 

Perhaps it can be considered by LT to have independent start/stop control of the cache drive/pool?

 

This would be more inline with the devices available in a plain unRAID configuration.

This particular suggestion may have some unintended consequences. How do you handle user shares when only part of the share exists on the cache pool? Let me elaborate. My VM's live in a cache only share, so the mover doesn't touch them. HOWEVER... I manually moved my little used VM images to an array disk in the share, so they appear in the same file structure, but live out of the pool. If the cache pool was started but not the array, half of my VM's would be missing their resources.

 

Granted, this is self inflicted, and could be easily solved on my end, but I suspect there are plenty of people that have VM's utilizing the cache pool for their boot media and the array for data. Do you create 2 classes of VM's so only those with no array references are left running? 

The changes that might satisfy the need are:

• make it possible for libvirt to be started at system start and not array start
  
• stop libvirt as part of system shutdown rather than array stop unless user explicitly stops it from the GUI
  
• add a new 'auto-shutdown' option for VMs to stop them on array stop.    This would default to 'Yes' to mimic current behaviour.  Only VMs with this set to Yes are stopped when the array stops.
  

I agree that the above does solve all possible issues, but at least users would have to wilfully change the default settings to get into any trouble, and those who know what they are doing can make far more use of the VM capability by running suitably set up VMs independently of the array behaviour. This leaves room for future enhancements (such as leaving cache mounted) and does not seem to compromise current capability.

Perhaps it can be considered by LT to have independent start/stop control of the cache drive/pool?

 

This would be more inline with the devices available in a plain unRAID configuration.

This particular suggestion may have some unintended consequences. How do you handle user shares when only part of the share exists on the cache pool? Let me elaborate. My VM's live in a cache only share, so the mover doesn't touch them. HOWEVER... I manually moved my little used VM images to an array disk in the share, so they appear in the same file structure, but live out of the pool. If the cache pool was started but not the array, half of my VM's would be missing their resources.

 

Granted, this is self inflicted, and could be easily solved on my end, but I suspect there are plenty of people that have VM's utilizing the cache pool for their boot media and the array for data. Do you create 2 classes of VM's so only those with no array references are left running?

 

See below for a solution.

 

The changes that might satisfy the need are:

• make it possible for libvirt to be started at system start and not array start
  
• stop libvirt as part of system shutdown rather than array stop unless user explicitly stops it from the GUI
  
• add a new 'auto-shutdown' option for VMs to stop them on array stop.    This would default to 'Yes' to mimic current behaviour.  Only VMs with this set to Yes are stopped when the array stops.
  

I agree that the above does solve all possible issues, but at least users would have to wilfully change the default settings to get into any trouble, and those who know what they are doing can make far more use of the VM capability by running suitably set up VMs independently of the array behaviour. This leaves room for future enhancements (such as leaving cache mounted) and does not seem to compromise current capability.

 

I like your suggestions and I agree as I feel they are constant with current behaviour but would like to suggest another one in addition (perhaps even to deal with the issue outlined by jonathanm).

 

- Implement ideas outlined by itimpi.

- Add 2 new options to each VM line that says "start with system" OR "start with array". There could be a quick rudimentary check that determines if this is possible or not (e.g. vidisk is on a cache only share). If it the checks are not successful you throw the user an error.

 

Yes there is still the issue of what happens when the user "manually" moves things around and this could "break" things BUT that is true today.

That may solve the VM resource issue, but what about user shares? Do you still export the user shares that exist on the cache drive, solely or cached? Do you allow writes to those shares as normal? How do you handle a file in a user share that exists only on the array if a file with the same name is written to the user share while the array is down? When the array comes back online, there will be a naming collision with the pre-existing file.

 

I suppose you could just keep the share export the same as it is now, where when the array is stopped, all shares except the flash are offline, but instead of unmounting the cache pool, leave it mounted.

That may solve the VM resource issue, but what about user shares? Do you still export the user shares that exist on the cache drive, solely or cached? Do you allow writes to those shares as normal? How do you handle a file in a user share that exists only on the array if a file with the same name is written to the user share while the array is down? When the array comes back online, there will be a naming collision with the pre-existing file.

 

You are right, I had missed that. I think what I envisaged when I thought about this was to NOT mess with the share settings at all. If the Array is down operation remains the same as today (from a share point of view).

 

I suppose you could just keep the share export the same as it is now, where when the array is stopped, all shares except the flash are offline, but instead of unmounting the cache pool, leave it mounted.

 

That sounds like a good idea. In addition I think we have a better chance of this getting implemented if it is maximum benefit minimal change AND your suggestion meets that criteria.

 

So at the moment we have:

 

- make it possible for libvirt to be started at system start and not array start

- make it possible to mount cache disk/pool on system start BUT all shares are disabled UNTIL Array starts as things stand today

- add 2 new options to each VM line that says "start with system" OR "start with array" => including perform a quick rudimentary check that determines if this is possible or not (e.g. vidisk is on a cache only share). If it the checks are not successful you throw the user an error.

- stop libvirt as part of system shutdown rather than array stop unless user explicitly stops it from the GUI

add a new 'auto-shutdown' option for VMs to stop them on array stop.    This would default to 'Yes' to mimic current behaviour.  Only VMs with this set to Yes are stopped when the array stops.

 

These "make it possible" statements could mean an enabled / disabled button via the gui or the cli I guess?

OK. So we haven't had very good initial freedback from LT re this feature

 

Starting VMs independent of the array is a feature that has been requested allot and one I'd love to facilitate running pfsense. Did this "feature" make it into 6.2?

This is not going to happen anytime soon, if ever.  It interferes with features we have planned for the future.  "Array Start/Stop" is really a misnomer.  In this context "array" refers to the entire set of devices attached to the server, not just the ones that are assigned to the parity-protected devices.

So do you have a solution in mind for those who wish to host a pfsense or other end point firewall appliance as a VM on unraid?

 

What is the issue exactly?

 

http://lime-technology.com/forum/index.php?topic=47875.msg460512#msg460512

 

Without being too combative, has anyone tried to start libvirt manually when the array is stopped and start a vm?

I was just going to start talking about this (the manual part of this equation).

 

Since this is the topic to discuss such a feature, I think this is a LARGE miss by LT for not wanting to implement this feature sooner.

I strongly feel that without this feature we're losing out on making this statement truly useful "This enables users to leverage the same hardware providing NAS services to the home as a workstation, where they can do work, play media/games, and be creative with a high-performance computing platform." https://lime-technology.com/unraid-6-press-release-2/

This statement is still true, and not specifically misleading, however it glorifies the situation.

 

Now no one said to me "Jeff, you should sell that other PC you have and virtualize everything into UnRAID, that will be the "cat's meow!"".

However I and many others did just that, and the whole "one box to rule them all" concept is very good, and we're very close to being there.

I wouldn't want to go back either, as I like everything in one computer, and I truly feel I'm using the processing power that I have (which was not very often prior as I don't game much, and an i5 for surfing the web is complete overkill).

 

I do not run Pfsense or other things that are very much important to others here (even though I like the idea and may play in the future), however it is obnoxious to have to shutdown my primary PC in order to do maintenance to the array. Leading me to always having a netbook near, or use my phone (which works, but is not the best for completing tasks/typing commands).

The quick and dirty solution was to add some form of GUI/X environment into 6.2 (which I have not used yet), and then open up the security risk of Firefox with admin rights (as I understand it) directly on the host. This solution still requires you to have an IGD for it to use, or sacrifice a video card for this output.

 

This is where the "UnRAID as a guest" is very intriguing, however it is also not the path followed by many here (or they're not very vocal).

However this solution still has customers buying UnRAID, so it is not exactly losing LT money.

 

So, with all of that said.

For a "primary" VM that we don't want to have shutdown unless the computer is actually rebooted or shutdown, what can we do to have another copy of libvirt/vfio/QEMU?

I say "primary" as I have no issue with my other VM's being managed in the way they are now, however I wouldn't be against this being universally an option.

 

Are the needed KVM related things in the kernel still accessible in this condition?

(This is not my area of expertise) Could we place a secondary copy of bzimage to use if not, or use Fedora/Arch/whatever as a base image to do what we want here?

Thinking out loud, but I think there are a decent amount of people who would benefit/appreciate this option.

We should call this "Unassigned VM's".. 

 

 

 

 

Just quoting myself elsewhere -

I like the idea of a separate drive or pool, call it 'Apps' for want of something better, with the primary distinction that it starts and stops with the system, unlike the Cache drive or pool which goes up and down with the array.  It makes it easier to manage always-on apps.  It also should be easy to implement, because it just uses a stripped down version of the Cache drive/pool code base, minus the Share stuff and 'Cache: Only' stuff.  An officially supported Apps drive/pool makes things like pfsense easier.  It also makes management more intuitive, both the Cache and Apps drives/pools are optional and work almost the same, but one starts and stops with the array, the other doesn't.

 

I still think it's the best framework for both users and developers, intuitive and straightforward.  Instead of 'Apps' drive/pool, could call it the 'System' drive/pool.

 

From Tom's remarks though, it will be harder to accomplish than I thought.  Most modules and drivers will have to be tagged as 'system' or 'array', that is, do they start and stop with the system or with the array.

 

I do think this is probably coming at some point, once Tom is convinced of its necessity.  The pfsense-type applications are one area, but I also think there will be more users wanting to run their primary desktop as a VM, and don't want to have to take it down for an array restart.  Or want to run a media playing app for the kids or the evening movie watchers, and don't want it interrupted at all.  (To be honest, it's hard to see that as too big an inconvenience, either make the unRAID maintenance wait an hour or so, or briefly interrupt the movie, or play it from an external device instead.)

 

... I think this is a LARGE miss by LT for not wanting to implement this feature sooner...

I do agree with the rest of what you said, good points, but I couldn't help feeling this was a little unfair.  They have been working very hard on both NAS and VM features for quite awhile, and the VM side is still very new, not quite unfinished, not completely stable.  If you feel it was a mistake not doing this sooner, then what would you have dropped that they HAVE added?  For awhile it seemed to some that they were concentrating too much on VM improvements and PR, so I think many are glad to see time put into the NAS side recently in adding dual parity.  Give them time, and as you have, make your voice heard, in order to help them decide feature priorities for the future.

Just to chime in quickly here as we did reply on this before and the truth is that there is a lot more to making this work than meets the eye.  It requires a lot of edge-case handling to support features that run outside of the array being available.  While this is a neat feature, it's not a mainline feature IMHO.  It's a niche for those that wish to run VMs like pfsense which I wouldn't necessarily advocate myself.  I'm all for consolidation, but I still feel that your router/firewall should live outside the server for now.  We're not quite ready for that, though maybe at some point in the future.

 

The bottom line is that I don't see us adding this any time soon.  Not even for 6.3.  As you can see in 6.2, we've moved the default storage location for the libvirt loopback image off the USB flash entirely and into the new system share that we create in the cache pool.

 

I'm not saying we would never ever do this, but it's a request that requires far more work than it's worth compared to other features we have planned for more near-term inclusion.  That said, we will, as always, keep monitoring the feature request board and posts like this for new user activity.  Active discussion among a handful of folks is great, but hardly the level of support I would say qualifies for prioritizing the feature in the roadmap.

... I think this is a LARGE miss by LT for not wanting to implement this feature sooner...

I do agree with the rest of what you said, good points, but I couldn't help feeling this was a little unfair.  They have been working very hard on both NAS and VM features for quite awhile, and the VM side is still very new, not quite unfinished, not completely stable.  If you feel it was a mistake not doing this sooner, then what would you have dropped that they HAVE added?  For awhile it seemed to some that they were concentrating too much on VM improvements and PR, so I think many are glad to see time put into the NAS side recently in adding dual parity.  Give them time, and as you have, make your voice heard, in order to help them decide feature priorities for the future.

 

Very true, and the improvements to the VM capabilities have really been terrific the last year.

I think the GUI boot was supposed to be a good middle ground, and I do kind of like it (now that I tried it), however I may need to post my experiences with "stealing" it's GPU from it (it doesn't seem to like that, but the console doesn't mind). Honestly if we could figure out a way to (somehow) reinitialize the GUI boot back to the primary display once a VM is shutdown, then I think my use case would be pretty covered (Example: need to stop array, shutting down all VM's, primary VM shuts down, local GUI boot pops back up on that GPU. Restart array, VM restarts, GPU is re-assigned to VM.. Me? a happy camper!  ).

 

I think that personally it has been a good balance of server "stuff" and VM "stuff" as of late.

The inclusion of dual-parity is enormous, however the changes to the VM manager have also been extensive. I'm certainly getting my $$'s worth (and now my Pro license is unlimited, booyay!).

I was just going to start talking about this (the manual part of this equation).

 

Since this is the topic to discuss such a feature, I think this is a LARGE miss by LT for not wanting to implement this feature sooner.

I strongly feel that without this feature we're losing out on making this statement truly useful "This enables users to leverage the same hardware providing NAS services to the home as a workstation, where they can do work, play media/games, and be creative with a high-performance computing platform." https://lime-technology.com/unraid-6-press-release-2/

This statement is still true, and not specifically misleading, however it glorifies the situation.

 

Now no one said to me "Jeff, you should sell that other PC you have and virtualize everything into UnRAID, that will be the "cat's meow!"".

However I and many others did just that, and the whole "one box to rule them all" concept is very good, and we're very close to being there.

I wouldn't want to go back either, as I like everything in one computer, and I truly feel I'm using the processing power that I have (which was not very often prior as I don't game much, and an i5 for surfing the web is complete overkill).

 

I do not run Pfsense or other things that are very much important to others here (even though I like the idea and may play in the future), however it is obnoxious to have to shutdown my primary PC in order to do maintenance to the array. Leading me to always having a netbook near, or use my phone (which works, but is not the best for completing tasks/typing commands).

The quick and dirty solution was to add some form of GUI/X environment into 6.2 (which I have not used yet), and then open up the security risk of Firefox with admin rights (as I understand it) directly on the host. This solution still requires you to have an IGD for it to use, or sacrifice a video card for this output.

 

This is where the "UnRAID as a guest" is very intriguing, however it is also not the path followed by many here (or they're not very vocal).

However this solution still has customers buying UnRAID, so it is not exactly losing LT money.

 

So, with all of that said.

For a "primary" VM that we don't want to have shutdown unless the computer is actually rebooted or shutdown, what can we do to have another copy of libvirt/vfio/QEMU?

I say "primary" as I have no issue with my other VM's being managed in the way they are now, however I wouldn't be against this being universally an option.

 

Are the needed KVM related things in the kernel still accessible in this condition?

(This is not my area of expertise) Could we place a secondary copy of bzimage to use if not, or use Fedora/Arch/whatever as a base image to do what we want here?

Thinking out loud, but I think there are a decent amount of people who would benefit/appreciate this option.

We should call this "Unassigned VM's".. 

 

Yep I'm on-board with most of that. In my case I'm looking to use a VM as a primary workstation but any shutdown of the array means I need to keep a laptop handy.

Missed this thread, but we need a way of keeping/running VMs with the array stopped.

 

E.g. Disk failed whilst your away? Tough luck!

 

It is abit silly that I can’t seem to have VMs running with the array stopped.

Edited February 15, 2018 by Interstellar

For me, it would be sufficiant when the startup/shutdown of VMs and Docker Containers including loading/unloading of libvirtd, mounting/unmounting libvirt config would be done by an external script instead of the monolotic array process. 

Either the control program could check, whether a script on the flash drive exists and execute this script instead of the default libvirt shutdown

or (preferred) there could be a default script responsible for starting/stopping libvirt and returning the result by exit code and this script would be called by the control program.

The same could be done for docker.

 

I think, this would be a minimalistic approach to solve the problem as long as there is no complete solution.

This way, we can modify the script to ignore some VMs, leaving libvirt loaded, and the "normal" user still has the same functionality as before.

Perhaps there is already some kind of script embedded in the control program, so this would be easy to implement.