Building an unRAID Development Environment


Recommended Posts

We all agree that unRAID is a great product, and a number of people have made their unRAID servers even better by running additional software on theirs.  There are a few ways to add additional functionality to the stock unRAID:

  • Precompiled Slackware Packages/Scripts - A number of great add-on packages and scripts already exist for Slackware (the base for unRAID), and many of them are plug-and-play with unRAID.  Occasionally, you might have to "redirect" a config file or two, but a lot of additional functionality can be added this way.  Other scripts have been specifically developed for unRAID (such as Joe L.'s unMENU) and can typically be found in the forums or in the wiki.
  • Custom unRAID distro - BubbaQ's bubbaRAID works with unRAID and provides some of the features most requested here in the forums.
  • Install unRAID onto hard drive - Probably the most limitless option of adding additional functionality to unRAID is to migrate unRAID to a full Slackware hard-drive install.  This way, you can include a ton of things that just aren't practical on a flash-drive install.

 

Although these are all great options, sometimes, they still aren't enough.  There may not be a precompiled Slackware package, and running unRAID from a hard drive might not be practical for your setup.  This was the situation for me when I wanted to get VMware Server running on my unRAID.  In these cases, you may be able to compile a package yourself that will be able to run on your normal unRAID install.  For this, you need a development system that mirrors unRAID but that has all the additional software development resources needed so that you can 1) install the desired software on the development system and 2) package that installation in a way that it can be run on the unRAID system.

 

These are the steps that I went through to build a development environment that mirrors unRAID Server 4.4 which is based on Slackware 12.1 and kernel updates to 2.6.27.7.  You development environment can be a physical or a virtual machine.  I built mine as a virtual machine (VMware Server 2.0 on a Windows XP host).  The steps I followed are basically copied from josetann's page at http://www.thetechguide.com/howto/unraid-on-hard-drive.html with only a few modifications and updates.  Hopefully, he won't get too mad at me reprinting them here.

 

---------------

 

These steps are specifically for unRAID Server using the 2.6.27.7 kernel.  At the time of this posting, that includes 4.4 and 4.5beta1.  The steps can easily be updated for future kernel versions though more care will need to be taken with Step #6 as the attached config file may not work with future kernel versions.

 

1.  Install Slackware 12.1.  I simply accepted all the installation defaults.  If you want a slimmer development system, you can be more selective in what's included in the install.  Josetann gives more detail as to what is needed and what is extra.  Since this was a development (not a production) environment, and because I was building it as a vm, I didn't care too much about a little extra bloat.  Additionally, I really like have X on here because it just makes somethings a lot easier.

 

2.  Get the updated kernel source, headers, and modules:

cd /usr/src

wget http://mirrors.easynews.com/linux/slackware/slackware-current/slackware/k/kernel-source-2.6.27.7_smp-noarch-1.tgz

wget http://mirrors.easynews.com/linux/slackware/slackware-current/slackware/d/kernel-headers-2.6.27.7_smp-x86-1.tgz

wget http://mirros.easynews.com/linux/slackware/slackware-current/slackware/a/kernel-modules-2.6.27.7-i486-1.tgz

 

3.  Get the unRAID source.  I'll put the code for downloading it directly, but if you have already downloaded it, save Tomm some bandwidth and try copying it over to your /usr/src directory some other way.

wget http://download.lime-technology.com/unRAID%20Server%204.4.zip

 

4.  Install the kernel packages:

installpkg kernel-source*
installpkg kernel-headers*
installpkg kernel-modules*

 

5.  Extract the unRAID source and copy to appropriate location:

mkdir unraid
mv unRAID* unraid
cd unraid
unzip unRAID*
mkdir bz-mod
cd bz-mod
zcat ../bzroot | cpio -i -d -H newc --no-absolute-filenames
cd /usr/src/unraid/bz-mod/usr/src/linux
cp -rf * /usr/src/linux/
cp -rf .config /usr/src/linux/ 

 

6.  Prepare the kernel config. 

cd /usr/src/linux
make clean
make menuconfig

 

In the last step of #5, we had copied the stock unRAID kernel config to /usr/src/linux.  Step #6 is to update this config to work under our development system settings.  At the very least, this will probably mean changing the modules for the filesystems, IDE/ATA, and SATA drivers to be compiled directly into the kernel instead of being modules loaded as needed.  This means changing the "M" to "*" under menuconfig.  When building my development system, I determined that I needed to make a number of changes to the Device Drivers areas of ATA/ATAPI, Serial ATA, and Graphics to get my system to boot (i.e., no kernel panics) and to get X working.  What I actually did was compare the settings in my stock Slackware 12.1 install to see what the settings needed to be in the upgraded (2.6.27.7) menuconfig.  This is one of the great things about using virtual machines because I was able to have the stock and the upgraded running side-by-side to make changes and test in real time.  I also made some changes to my networking drivers, but I think my router was just being flakey at the time and that I probably didn't have to make any changes here.

 

I've attached the .config file that works for my setup.  It may work without modification for yours.  You will need to rename it from config.txt to .config and put it in the /usr/src/linux directory prior to doing make clean here.  Hopefully, you will not need to make any changes under make menuconfig, but once your config is finished, proceed to Step #7.  Note, this may turn out to be an iterative process.  If your new kernel fails to boot, you'll probably need to come back to #6 to make changes.

 

7.  Compile the kernel:

make bzImage
cp arch/x86/boot/bzImage /boot/vmlinuz-2.6.27.7-unRAID
cp System.map /boot/System.map-2.6.27.7-unRAID
make modules
make modules_install

 

8.  Update LILO to include the new kernel as an option.  The LILO config file is located at /etc/lilo.conf.  It is specific to your install, and may look slightly different for different people.  Find a set of lines similar to this:

# Linux bootable partition config begins
image = /boot/vmlinuz
root = /dev/hde3
label = Linux
read-only
# Linux bootable partition config ends

 

The string for root is likely to be different.  Add another entry like the one below, below this current entry.  Make sure to keep your "root" the same as it was before.  (Note, the only changes are on the "image" line and the "label" line.

 

image = /boot/vmlinuz-2.6.27.7-unRAID
root = /dev/hde3
label = unRAID
read-only

 

Once you have edited and saved lilo.conf, you'll need to update lilo by running:

lilo -v

 

Your new kernel should now be added as a boot option.  BTW, I also change a couple of other things in my lilo.conf file.  I change my timeout value to 150.  This means a 15 second delay before booting the default kernel instead of the standard 2 minutes (120 seconds).  Also, once I made sure my unRAID kernel booted properly (and that X worked), I reversed the order of my kernel entries, to make unRAID the first entry and therefore the default kernel.  That way, if I fire up my development environment and get distracted, it boots into the unRAID kernel not the Slackware 12.1 kernel.

 

If all went well, you should have a booting unRAID kernel and a functional devlopment environment.  This was the extent of my development setup for me, but josetann goes on to give directions for getting unRAID (i.e., the array and parity and all) to run under this setup.  His goal was running unRAID on a hard drive not just building a development environment.  Please visit his site if you need that additional functionality.

 

 

 

Edited Jan 25 2009 to update attached config file

Link to comment
  • 2 weeks later...
  • Replies 53
  • Created
  • Last Reply

Top Posters In This Topic

musicmann,

 

This is some good stuff. A bunch of steps usually found in different places consolidated into one easy to follow post. I had to do this the hard way, I wished I had had your post a few weeks back.

 

Having said that, I think this can be expanded some. Maybe I can help, maybe you would like to do this, or maybe you want to still leave it as an exercise for the readers. I don't want to step on your toes here. What I think is missing are the steps that take you from this environment, to the bz* images you can put on a flash and then boot up unraid with your added packages.

 

However, maybe I got this wrong. If you are trying to create an environment solely for the purpose of created pkg files that you can install from your "go" script then this is a great guide.

 

Cheers,

Paul

Link to comment

Paul,

I'm a bit lazy.  I was just making an environment on which I could build packages that could be installed via the go script, so once I got what I needed, I stopped there.  ;D

 

Feel free to add anything that you see fit to this thread.  I think there still may be significant interest in home-rolled unRAID distros, and it may help to have it all located in one place.  If you'd like me to add your steps to the original post, that can be done too.

 

 

Link to comment
  • 2 weeks later...

I've updated the kernel config file in the original post.  This was to address an issue that I had building a VMware Server package in which VMware Server worked but bridged networking for virtual machines didn't (NAT worked). 

 

In the original config, I had piled in a lot of network device drivers thinking that, that was being "safe."  This version is much more like the stock unRAID wrt network device drivers.  I used a physical instead of a virtual development environment this time, but I tried to add support to the virtual NIC that VMware uses, so you should be able to make virtual development environments with networking support using this config file.

 

Also, Slackware 12.2 uses the 2.6.27.7 kernel (used by unRAID 4.4.2) by default, so you can avoid a few steps by installing it versus installing 12.1 and then upgrading to the 2.6.27.7 kernel.

Link to comment
  • 5 weeks later...

If you want to add raid Drivers, where in this process would you do it?

Where in the process would you create your say raid 5 or 10 drive for the Cache Drive?

Any chance unraid could also be used as a htpc?

The unRAID "md" device takes the place of the usual "md" (raid5) device.  As currently written, you cannot have both in the same kernel.  It would need a complete re-write.

 

Joe L.

Link to comment

Joel, If you were to do Hardware Raid 5 for the cache drive, then do you still need the MD command?

 

I have no experience with hardware raid5 on linux.  Sorry.  From what I understand, there is limited support for some cards.

 

unRAID needs the "md" device to be its own.

 

Joe L.

Link to comment
  • 2 weeks later...
Guest MrKenSan3

Hi All, New here, but have a simple question, and this seems like the right gang to ask...

 

I've not yet dug into the details of implementing musicmann's EXCELLENT tutorial, but I'd like to ask if *all* of the sources are available for the unRAID kernel mods/modules required. I ask because I'd like to recompile and run this on a NAS box with a non-intel processor.

 

Goal would be to support up to 4ish drives on a commercial NAS box (thinking the D-Link DNS series) as I like the form-factor and size of such a solution rather than hacking a machine together from bits... (which I have plenty of already ;-) )

 

A nod from those in the know would encourage me to dig in and give this a whirl... Of course with the goal of sharing the fruits with the community!

 

Any thoughts, or pointers to other attempts at something like this would be gratefully appreciated!

 

MrKenSan

 

Link to comment

Hi All, New here, but have a simple question, and this seems like the right gang to ask...

 

I've not yet dug into the details of implementing musicmann's EXCELLENT tutorial, but I'd like to ask if *all* of the sources are available for the unRAID kernel mods/modules required. I ask because I'd like to recompile and run this on a NAS box with a non-intel processor.

 

Goal would be to support up to 4ish drives on a commercial NAS box (thinking the D-Link DNS series) as I like the form-factor and size of such a solution rather than hacking a machine together from bits... (which I have plenty of already ;-) )

 

A nod from those in the know would encourage me to dig in and give this a whirl... Of course with the goal of sharing the fruits with the community!

 

Any thoughts, or pointers to other attempts at something like this would be gratefully appreciated!

 

MrKenSan

 

The source for the "md" kernel module is GPL and available, as is the config file.  It is written to overlay on top of a GPL Linux distribution of Slackware source.

 

Many have built their own kernel.

 

The source code for the shfs "User-Share" file-system is NOT available, and is not GPL.  The same for the "emhttp" management utility that performs all configuration and management of the array.  They represent lime-technologies product.  Unless those execuitables  are runable on the DLINK platform, you will not be able to run either.  Many of the smaller form-factor platforms do not include a memory management module...  I have no idea about your target, but a quick check shows it uses an ARM processor, so I'm fairly certain you will not get too far with your effort before hitting a snag. A port to a different platform will be a challenge unless Slackware is already ported.

 

Joe L.

Link to comment
The source code for the shfs "User-Share" file-system is NOT available, and is not GPL.  The same for the "emhttp" management utility that performs all configuration and management of the array.  They represent lime-technologies product.  Unless those execuitables  are runable on the DLINK platform, you will not be able to run either.  Many of the smaller form-factor platforms do not include a memory management module...  I have no idea about your target, but a quick check shows it uses an ARM processor, so I'm fairly certain you will not get too far with your effort before hitting a snag. A port to a different platform will be a challenge unless Slackware is already ported.
I'll admit I don't really know what's going on under the hood, but if I understand it correctly, the md module does all the work of keeping parity while emhttp does the work of configuring and controlling md. So in theory one could write their own management utility in order to run unRaid on another platform. Still challenging, but nowhere approaching impossible. And while the shfs is great, you could get by with just SMB or NFS disk shares.

 

</arm-chair coder> ;)

Link to comment
  • 5 weeks later...

Just getting going with unRAID here, and following this tutorial prompts me to suggest some alterations :)

 

The paths for the 2.6.27.7 files need changing as 'slackware-current' now points to 2.6.29.1;

 

wget http://mirrors.easynews.com/linux/slackware/slackware-12.2/slackware/k/kernel-source-2.6.27.7_smp-noarch-1.tgz

wget http://mirrors.easynews.com/linux/slackware/slackware-12.2/slackware/d/kernel-headers-2.6.27.7_smp-x86-1.tgz

wget http://mirrors.easynews.com/linux/slackware/slackware-12.2/slackware/a/kernel-modules-2.6.27.7-i486-1.tgz

 

 

Also, for those of us setting up a development environment under VMWare, there is an excellent list of what kernel options are necessary here: http://en.gentoo-wiki.com/wiki/HOWTO_Install_Gentoo_on_VMware#Kernel_options

 

It's for Gentoo, and a couple of kernel options have moved (but are still easily locatable based on the information there), but those options should work very well for any VMWare dev environment (mine, for example, is running under VMWare Fusion)

 

 

Aside from that the instructions were spot on - so thanks very much for writing it all up! :)

Link to comment
  • 2 weeks later...

Hi,

 

I am able to boot with my new kernel but I cannot connect with the internet. after typing "startx" KDE loads but Firefox is unable to connect to the internet.

 

ifconfig command showing LOOPBACK.

 

I am getting this error message during "make bzImage" on step #7

 

make[1]: warning:  Clock skew detected.  Your build may be incomplete.

make: warning:  Clock skew detected.  Your build may be incomplete.

 

Where should I start?

 

FYI: I am using Slackware 12.2 and I skip steps #2 and #4

 

Thanks!

Link to comment

I assume you've tried restarting networking and/or have restarted the machine itself.  After that, my first check would be to make sure that the approprate network driver was compiled into the kernel.  If you are using a virtual environment under a VMware product, check to make sure the pcnet32 driver has been included in the kernel.  If you are using a physical development environment (or if you're using a different virtualization platform...or even a VMware product and pcnet32 is already in there), find out what networking hardware or "hardware" is being used, and make sure that driver is included in the kernel.

Link to comment

Hello!

 

First, thanks to all who've worked hard to get the Development Environments up and running!

 

I'm having difficulty getting the kernel modules compiled.  I've basically followed the directions, with the following differences:

1.  Using 12.2 as the base slackware install

2.  Using unraid 4.5 Beta 4 (the latest available right now)

3.  Using the 2.6.29.2 kernel sources (the 'current' slackware sources)...I saw somewhere that Beta5 is being prepared with 2.6.29.1, but I couldn't locate that...and I've not been able to locate any info on what Beta 4 is built on...just that it's whatever was current on April 14th...

 

I tried the .config file shipped with beta4, and the .config file previously listed in this thread...and also tried config'ing my own...

 

The Kernel compiles fine, but when I make modules it failes while compiling md

I get an error "in function read file

error: 'struct tast_struct' has no member named 'fsuid' (a bunch of times)

and the build fails.

 

Am I just getting too agressive with my kernel sources? or is there something else to tweak?...should I just wait for Beta 5?

 

I should also mention that I'm doing this on Slackware running on VMWare esxi3.5 (not that it should matter yet...)  My goal is to get VMware working on top of unraid (I know, it won't run while running on esxi, but I don't have a dev box to play with this yet).

 

Any easy fixes?...

 

Thanks!

David

Link to comment

Hello!

 

First, thanks to all who've worked hard to get the Development Environments up and running!

 

I'm having difficulty getting the kernel modules compiled.  I've basically followed the directions, with the following differences:

1.  Using 12.2 as the base slackware install

2.  Using unraid 4.5 Beta 4 (the latest available right now)

3.  Using the 2.6.29.2 kernel sources (the 'current' slackware sources)...I saw somewhere that Beta5 is being prepared with 2.6.29.1, but I couldn't locate that...and I've not been able to locate any info on what Beta 4 is built on...just that it's whatever was current on April 14th...

 

I tried the .config file shipped with beta4, and the .config file previously listed in this thread...and also tried config'ing my own...

 

The Kernel compiles fine, but when I make modules it failes while compiling md

I get an error "in function read file

error: 'struct tast_struct' has no member named 'fsuid' (a bunch of times)

and the build fails.

 

Am I just getting too agressive with my kernel sources? or is there something else to tweak?...should I just wait for Beta 5?

 

I should also mention that I'm doing this on Slackware running on VMWare esxi3.5 (not that it should matter yet...)   My goal is to get VMware working on top of unraid (I know, it won't run while running on esxi, but I don't have a dev box to play with this yet).

 

Any easy fixes?...

 

Thanks!

David

Your "current kernel sources" probably have a typo. Or, you introduced an error in your editing of the "md" files.

  I'll bet "tast_struct" should be "task_struct"  (expecially since task_struct is a data structure that does have a member named fsuid)

 

Joe L.

Link to comment

Thanks Joe.

 

You're right about the 'k' ....I mistyped into my post...it is 'task_struct' in the actual error message...

 

I'll go re-grab the kernel sources and try again...I don't think I edited anything (other than the config file, and that--at least one of the times--was using make menuconfig...but perhaps I did something wrong...I certainly did not edit any of the md files...

 

I just wanted to make sure I wasn't beating my head against an impossible configuration...I was worried that the sources for md might not work with this kernel source...

 

Thanks again

-David

Link to comment

Thanks Joe.

 

You're right about the 'k' ....I mistyped into my post...it is 'task_struct' in the actual error message...

 

I'll go re-grab the kernel sources and try again...I don't think I edited anything (other than the config file, and that--at least one of the times--was using make menuconfig...but perhaps I did something wrong...I certainly did not edit any of the md files...

 

I just wanted to make sure I wasn't beating my head against an impossible configuration...I was worried that the sources for md might not work with this kernel source...

 

Thanks again

-David

I've not looked... but do you have the kernel header files?  They would have the structure definitions.  (I don't have a development env, so I can't tell you if the structure was renamed, or fsuid was renamed... but somehow, I doubt it)
Link to comment

Yup...got the kernel header files...I agree that it seems like something is missing here...

 

I tried dl'ing the packages from thetechguide.com (kernel 2.6.24.4) and they build just fine (and they have the unraid md code in them)...but even taking the ./md folder from them and porting it into the 2.6.29.2 structure doesn't build (same errors)...

 

I'm thinking there must be some other header file I'm missing...but I have no idea how to find it...

 

BTW, the kernel does compile with the 2.6.29.2 stock md, but as soon as I copy over the unraid md source the build dies...

Link to comment

To be on the safe side, have your development enviroment at the exact same kernel version as the target unRAID you're building for.  I'm not sure if this is vital, but it will remove another variable from the equation.

 

I haven't played around with 4.5b4, so I don't know the exact version but it may be in the release notes (looks like it might be 2.6.28.4).  If that doesn't work, just boot the 4.5b4 flash and you should see the kernel version either during the boot process or when you log in.

 

 

Link to comment

Thanks musicmann (for both the initial instructions and this note!)

 

Silly question though...where can I get depreciated kernel sources for slackware?...it looks like they freeze the kernel level with a dotted release, and the "slackware-current" folders only contain the most recent...I can't seem to locate any in-between releases...

 

This is my first time compiling kernels under slackware in about 12 years (yikes!---)...of course we used to have to compile them all the time ;-} -- Funny, it doesn't seem to have changed much (except don't have to "make dep" any more)

 

I'm guessing that Beta5 will be out any time now (I think I saw a note a few weeks ago saying it was due out in a few days...), so perhaps I'll just wait for that...then hopefully I'll be able to get "aligned" sources....

 

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.