[GUIDE] Virtualizing unRAID in Xen / KVM on Arch Linux

Post your compiled packaged on a google drive or something and post a link. That way people don't have to even to deal with compiling it.

Now for a real noob question, at the end of the compiling process, yaourt "asked" if I wanted to install it and I answered Y. Now I don't know where the compiled package is. It's not in my "builds" folder.

 

Have a look at /etc/makepkg.conf and tell it to compile with all your CPUs, tell it to do it in memory (/tmp/makepkg is the default but commented out) and tell it save the package on your system somewhere (the default that is commented out is /home/packages).

 

Otherwise it stores the package in your /tmp directory. If you reset your PC... It is gone and you will need to recompile.

Here's a link to my built package:

https://drive.google.com/folderview?id=0B3KPKAtShvQmeGR3VEJWTkQ1QmM&usp=sharing  I have a md5sum text file included.

I'll make corrections to my /mnt/makepkg.conf; thanks for the guidance!

 ABSROOT=. abs community/libvirt
==> Starting ABS sync...
rsync: failed to connect to rsync.archlinux.org (89.238.67.251): Connection timed out (110)
rsync: failed to connect to rsync.archlinux.org (2a00:1828:2000:547::2): Network is unreachable (101)
rsync error: error in socket IO (code 10) at clientserver.c(128) [Receiver=3.1.0]

did some looking, saw mention of the port allowed all to server, same error...

edit:

went back, saw this:

%

arch_1 ~ # abs
==> Downloading tarballs...
    ==> core...
    ==> extra...
    ==> community...
    ==> multilib...

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now

ran it again

# abs
==> Starting ABS sync...
rsync: failed to connect to rsync.archlinux.org (89.238.67.251): Connection timed out (110)
rsync: failed to connect to rsync.archlinux.org (2a00:1828:2000:547::2): Network is unreachable (101)
rsync error: error in socket IO (code 10) at clientserver.c(128) [Receiver=3.1.0]

ran it again

 

# abs
==> Starting ABS sync...
rsync: failed to connect to rsync.archlinux.org (89.238.67.251): Connection timed out (110)
rsync: failed to connect to rsync.archlinux.org (2a00:1828:2000:547::2): Network is unreachable (101)
rsync error: error in socket IO (code 10) at clientserver.c(128) [Receiver=3.1.0]

pacman -Syyu

Reboot after it installs all the updates and then...

abs

That worked.  Should add it to the guide...

One thing that happened,  on reboot my monitor showed out of range again and Putty does not connect.  Had to hard reset it,  but this time it booted Ok without making changes.  So not sure why it is doing that.

That worked.  Should add it to the guide...

It will. I am waiting for Arch to post the 01-2014 ISO and then I post will the guide. That way the grub error is fixed (the lvm possibly) and any other changes.

How is upgrading?

How is upgrading?

You tell me. When you did "pacman -Syu" (upgrade your system) did you find it difficult?

Arch is a Rolling Release Distro. There are no "versions" or "releases". Technically you could install Arch from 2009 and do "pacman -Syu" and it will upgrade it to what Arch is today.

I have used images from a year and a half ago and had no problems updating them and it being stable / working. Things like the switch from netcfg to netctl you would have to manually create the network interface is an example. However, all these type of things are published on the very front page of the Arch website with what you need to do.

This evening I also worked through the sections as (I think) smakovits, "Enabling libvirt with Xen support in Arch Linux" and "Configure libvirt to work via SSH or TCP," and got through them without much uncertainty. This showed up at the end of building/installing libvirt:

(1/1) reinstalling libvirt                         [######################] 100%
>>> You may need to run 'rm -rf ~/.libvirt'
>>> libvirt runs qemu from nobody:nobody by default
>>> change it in /etc/libvirt/qemu.conf

I'm going to assume by doing the configuration work that followed the above warning can be ignored. Anyway I rebooted and the system still looks pretty solid.

I don't quite understand how to create domU hard drive and guests yet. Oh well, tomorrow in another day. Cheers!

I'm going to assume by doing the configuration work that followed the above warning can be ignored. Anyway I rebooted and the system still looks pretty solid.

It should work. If not, that is an easy fix to solve.

Like I said in my previous post, when the new Arch ISO is released I will have my GF (Who is computer illiterate and no clue what Linux even is) follow my Arch Install Guide (I haven't posted yet) and the everything else in the first page of this thread and fix any mistakes / steps I might have left out / assumed you would know.

I don't quite understand how to create domU hard drive and guests yet. Oh well, tomorrow in another day. Cheers!

You finally reached the point where you will be using WebGUIs (Webmin) and Virt-Manager (Linux GUI that runs on your Windows Machine) to Manage your Server / LVMs / VMs now.

How is upgrading?

 

You tell me. When you did "pacman -Syu" (upgrade your system) did you find it difficult?

 

Arch is a Rolling Release Distro. There are no "versions" or "releases". Technically you could install Arch from 2009 and do "pacman -Syu" and it will upgrade it to what Arch is today.

 

I have used images from a year and a half ago and had no problems updating them and it being stable / working. Things like the switch from netcfg to netctl you would have to manually create the network interface is an example. However, all these type of things are published on the very front page of the Arch website with what you need to do.

When you put it like that,  pretty easy.  I was thinking about Linux mint,  they have a 6 month cycle.  I have v12 and do my apt-get update/upgrade,  but it is still v12 even though they are really on 16.  While my packages are up to date,  I have no clue what it means for the system other than online they usually say it is Ok.

Knowing I can do pacman - syu and be in the same place as the latest iso is a good thing. Makes me like arch more even if it has been a pain in my Butt

so here is a good one.

gcc: internal compiler error: Killed (program cc1)
Please submit a full bug report,
with preprocessed source if appropriate.
See <https://bugs.archlinux.org/> for instructions.
Makefile:6799: recipe for target 'conf/libvirt_conf_la-domain_conf.lo' failed
make[3]: *** [conf/libvirt_conf_la-domain_conf.lo] Error 1
make[3]: *** Waiting for unfinished jobs....
make[3]: Leaving directory '/tmp/makepkg/libvirt/src/libvirt-1.2.0/src'
Makefile:4742: recipe for target 'all' failed
make[2]: *** [all] Error 2
make[2]: Leaving directory '/tmp/makepkg/libvirt/src/libvirt-1.2.0/src'
Makefile:1942: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/tmp/makepkg/libvirt/src/libvirt-1.2.0'
Makefile:1844: recipe for target 'all' failed
make: *** [all] Error 2

I started some searches and saw some people reporting this as a result of running out of memory.  Quite possible I guess, but how do I know for sure?  I did what the xen guide told me to do and set dom-0 to 512 M and then I have no swap , so I guess the question is if I just assume it is an issue of running out of memory and how do I confirm?  Or, do i just change that setting back, re-build grub and try again when I have all 8G memory available to dom-0?

so here is a good one.

I started some searches and saw some people reporting this as a result of running out of memory.  Quite possible I guess, but how do I know for sure?  I did what the xen guide told me to do and set dom-0 to 512 M and then I have no swap

Don't set / limit the memory of your server until you get everything working the way you want and see how much memory it needs.

Or, do i just change that setting back, re-build grub and try again when I have all 8G memory available to dom-0?

That is exactly what I would do.

grumpy,

I took one of your past suggestions and took the issue to arch forum, over there someone told me to run dmesg

[46899.085077] Out of memory: Kill process 24124 (cc1) score 92 or sacrifice child
[46899.085347] Killed process 24124 (cc1) total-vm:80236kB, anon-rss:41672kB, file-rss:0kB

So, I set things to 2G and then everything completed successfully.

This then raised a question.  For dom-0 do we really gain and need that much performance boost, compiling in memory as opposed to on disk?  I only ask because how does one know how much memory dom-0 will actually need going forward? 

This then leads me to a second question.  Are these hard numbers?  Meaning, if my system has 8G memory and I set dom-0 to 2G and add 3 vm's at 2G, then I am at 8G total.  But, what if each system is only sipping 512M or even 1G, then technically I have free memory for another dom-u if I need it.  Not saying that I will, but still, I must ask since I thought dom-0 was going to sip 512M and it obviously wanted more here, so to be safe I should give it the more it wants so as to not have errors in the future, but does doing so limit me for what is available to my actual VM's?

You can hard code the amount of memory and vCPU on the server / VMs or shave it all amongst both.

All of my VMs run with one vCPU and 1gb of memory. I play XBMC fine with those and more memory or vCPUs isn't going to make it work any faster / better.

A Linux distro VM with sickbeard, couchpotato, sabnzb, etc. will probably take up 512k of memory and giving it more isn't going to do anything.

Same with an unRAID VM... without plugins its not using but 300mb of ram. giving it 4 vCPUs and 8 GB of memory isn't going to make it stream faster, run parity checks faster or copy / write faster. If you cache your drives give it more... otherwise don't bother.

Now if you are running VMs like Usenet indexer or Windows... You will probably want to give it more.

Tweak and fine tune everything once you get it the way you want. Not to mention, memory is cheap. You could add another 8GB if needed and allocate it wherever. Your VMs or the server do not care or have to be reloaded.

I compile as much as I can in memory and on a tmp filesystem - to ease the useage of my ssd until everything is all sexed up and I leave the base Dom0 alone

Myk

grumpy,

 

I took one of your past suggestions and took the issue to arch forum, over there someone told me to run dmesg

 

[46899.085077] Out of memory: Kill process 24124 (cc1) score 92 or sacrifice child
[46899.085347] Killed process 24124 (cc1) total-vm:80236kB, anon-rss:41672kB, file-rss:0kB

 

So, I set things to 2G and then everything completed successfully.

 

This then raised a question.  For dom-0 do we really gain and need that much performance boost, compiling in memory as opposed to on disk?  I only ask because how does one know how much memory dom-0 will actually need going forward? 

 

This then leads me to a second question.  Are these hard numbers?  Meaning, if my system has 8G memory and I set dom-0 to 2G and add 3 vm's at 2G, then I am at 8G total.  But, what if each system is only sipping 512M or even 1G, then technically I have free memory for another dom-u if I need it.  Not saying that I will, but still, I must ask since I thought dom-0 was going to sip 512M and it obviously wanted more here, so to be safe I should give it the more it wants so as to not have errors in the future, but does doing so limit me for what is available to my actual VM's?

The reviewing of one's syslog is a time-honored tradition with unRAID. Systemctl is great, I can type in, "systemctl --failed" and see that everything's OK. I'm wondering if in Arch there's the same sort of way to review system logs. I don't see anything in WebMin and in Arch Wiki I don't see much I understand.

So, is there a great web-app to query that queries the server?

TIA!

The reviewing of one's syslog is a time-honored tradition with unRAID. Systemctl is great, I can type in, "systemctl --failed" and see that everything's OK. I'm wondering if in Arch there's the same sort of way to review system logs. I don't see anything in WebMin and in Arch Wiki I don't see much I understand.

Arch uses systemd journal which is a replacement for syslogs. Most Linux Distros will be moving too it also. Fedora, openSUSE, etc. already have.

Systemd-Journal

Show all messages from this boot:

journalctl -b

However, often one is interested in messages not from the current, but from the previous boot (e.g. if an unrecoverable system crash happened). This is possible through optional offset parameter of the -b flag: journalctl -b -0 shows messages from the current boot, journalctl -b -1 from the previous boot, journalctl -b -2 from the second previous and so on. See man 1 journalctl for full description, the semantics is much more powerful.

Show all messages by a specific unit:

journalctl -u netcfg

or

journalctl -u samba

journalctl --fail --irish curse

journalctl --fail --irish curse

Don't feel bad.

I'm polish... So nothing works on mine and there is no chance I can fix it.

So if I am to the end of the guide with a seemingly clean install,  ready for vms,  what's the best method to back it up for easy recovery to start over if something goes crazy.  Lvm snapshot or some sort of backup?

You can hard code the amount of memory and vCPU on the server / VMs or shave it all amongst both.

 

All of my VMs run with one vCPU and 1gb of memory. I play XBMC fine with those and more memory or vCPUs isn't going to make it work any faster / better.

 

A Linux distro VM with sickbeard, couchpotato, sabnzb, etc. will probably take up 512k of memory and giving it more isn't going to do anything.

 

Same with an unRAID VM... without plugins its not using but 300mb of ram. giving it 4 vCPUs and 8 GB of memory isn't going to make it stream faster, run parity checks faster or copy / write faster. If you cache your drives give it more... otherwise don't bother.

 

Now if you are running VMs like Usenet indexer or Windows... You will probably want to give it more.

 

Tweak and fine tune everything once you get it the way you want. Not to mention, memory is cheap. You could add another 8GB if needed and allocate it wherever. Your VMs or the server do not care or have to be reloaded.

What distros are you running for xbmc or sickbeard /sabnzbd,  arch or something else?

Depends. Mostly Arch cause I like it.

If you know Ubuntu and like it... Use it. Just go to the Ubuntu Xen wiki and follow tjr guide. Use the Xen-Installer they provide a link too. Most Linux Distros have this.

OK, 2 things really.  The first is an issue I have seen since installing xen, which is when I boot the system, the monitor shows out of range and the system is not available.  I need to hold the power button to power off since nothing else works.  I thought it was only when grub is first created, but I just saw it occur randomly...I am attaching a log from last boot with hopes that someone more into logs can tell me something. 

Second, my xterm appears to fails and I cannot quite tell why:

xterm
Warning: This program is an suid-root program or is being run by the root user.
The full text of the error or warning message cannot be safely formatted
in this environment. You may get a more descriptive message by running the
program as a non-root user or by removing the suid bit on the executable.
xterm: Xt error: Can't open display: %s
xterm: DISPLAY is not set

so close I can taste it

log.txt

Second, my xterm appears to fails and I cannot quite tell why:

xterm
Warning: This program is an suid-root program or is being run by the root user.
The full text of the error or warning message cannot be safely formatted
in this environment. You may get a more descriptive message by running the
program as a non-root user or by removing the suid bit on the executable.
xterm: Xt error: Can't open display: %s
xterm: DISPLAY is not set

 

I state in my guide that it is based on you doing things as root.

Run xterm as root and with xming loaded and see what happens.

I did.

Xming on my windows box right?