For the love of unRAID, A union of root ram and a custom filesystem.

I haven't been around much lately as I've been working with rTorrent customizations on unRAID plus a new possibility.

This new possibility is not my own idea, but I decided to play with it and see what is possible.

I've started to play with a union fs and unraid.

It is not for the faint of heart. It can get a bit involved.

I've spent the last couple days wrestling with this, but I have something sorta working.

I'm mentioning this to see if I can peak some interest and experimentation among others.

I've started to play with union filesystems in an effort to merge a static disk based filesystem with the root filesystem.

The purpose for me is to create a static filesystem for addons on top of unraid so that I do not have to keep installing these packages every time.

For me it's no big deal to install the package.

Yet for some of the more involved packages, there is static data/configs that I only want to modify once.

Plus now that I have a battery backed gigabyte ram drive on the sata channels, I can have a useful filesystem without a mechanical spindle.

For those with a cache drive. A directory in the root named with a "." would work too.

I started playing with unionfs, but I was having weird issues with it.

The patch went in easily. once I applied it to a supported kernel.

One plus is it did union the flash drive directly.

The problem I had is it would hang when I tried to union the root filesystem a special way.

I have to play with this s'more.

http://www.filesystems.org/project-unionfs.html

http://en.wikipedia.org/wiki/UnionFS

I did some searching and saw that most people have moved over to aufs.

This was a bit of work to get into the kernel, but eventually made it in.

It's very separate and requires manual tweaking of kernel files to get it installed.

It does not seem that mature, or it seems a bit rough on the presentation, however Knoppix and a few distros are using this.

it seems capable. It does not union the FAT flash directly.

http://aufs.sourceforge.net/

So far, I spent huge amounts of time trying to union the root filesystem.

You cannot use pivot_root on rootfs. (initramfs) which is how unRAID is using the root filesystem.

Eventually I found you can use switch_root from busybox with a tmpfs.

So I complied busybox linked it as switch_root and went through another process of trying to use it.

What I ended up doing is this.

modify syslinux.cfg to add  a debug keyword so I could control startup and get debug shells as needed.

I added a rdinit=/linuxrc to jumpstart the root filesystem initialization and highjack the system initialization.

This has the effect of being able to turn unioning on or off by use of rdinit.

With it I can run a script before init, without it the normal init process runs.

What I had to do was create a tmpfs, mount it as aufs.

rsync root to the tmpfs/aufs/ramdisk, then switch_root to it. 

I got pretty far.

caveats of putting root on a tmpfs vs initramfs?

Yes there are some (good and bad points)

one point is a tmpfs has a fixed max size, if size is unspecified, it is half your available ram.

So your root filesystem can now never grow past a specified amount.

This can actually be a plus. I.E. Now you may not crash by running out of room from logs.

I don't know what the effect of this is.

Also, because your root filesystem is now limited, you may be limited as to how many addons you can install.

Frankly, the root filesystem in ram consumes 72MB on a fress bootup. So I do not consider this much an issue.

OK, back to the union,

After the linuxrc change and root switchover to tmpfs, I was able to build a union. but not directly from flash.

To build a union using flash I created a fielsystem on a file, mounted via loopback, then unioned the loopback filesystem.

It worked, albeit a lil slow and quirky. I think there is a .inode file that is updated and this sorta slows it down.

Still have more work to do.

The problem I had was breaking the union to the loopback filesystem.

I have not figured out how to delete the union so i can umount the loopback filesystem.

I would  love to see what others thing so I can experiment in the right direction.

Here's my linuxrc script so far.

#!/bin/sh

PATH=/sbin:/usr/sbin:/bin:/usr/bin:/ramdisk/bin

echo "
*******************************************************************************
"
echo "$0: pid=$$, ppid=$PPID $PATH"

touch /.aufs
modprobe aufs

mkdir /ramdisk
mount -v -t tmpfs none /ramdisk

# Use of mv will require changes to path or use of full filenames. 
# switch_root does a recursive delete on initramfs
# making actual mv unneeded
# mv / /ramdisk
rsync -ax / /ramdisk

# Setup union/aufs to ramdisk for merge.
mount -v -t aufs -obr:/ramdisk=rw /ramdisk /ramdisk


# Not done here, done in normal init rc.S process
# mount -v --move /proc /ramdisk/proc
# mount -v --move /sys  /ramdisk/sys 
# mount -v --move /     /ramdisk


# Temporatily mount proc, set root device to ramdisk
mount -t proc /proc /proc
echo "0x100" > /proc/sys/kernel/real-root-dev
umount /proc

# Update mtab to point to / so that remount later works
sed -i -e 's|/ramdisk|/|g' /ramdisk/etc/mtab

# Switch root
exec switch_root /ramdisk /sbin/init

#
# Does not work with initramfs
# cd /ramdisk
# mkdir /oldroot
# pivot_root . oldroot
#
echo "
*******************************************************************************
"

Because I had to recreate bzroot, I also modified the etc/rc.d/rc.S script to put in debug calls.

This allows me to add debug to the syslinux.cfg line so that I can call a shell where appropriate.

I'll probably modify the debug call soon, but here it is in case someone wants to experiment.

I added the following CHUNK after sysfs initialization

if [ -d /sys ]; then
  if grep -wq sysfs /proc/filesystems ; then
    if ! grep -wq sysfs /proc/mounts ; then
      /sbin/mount -v sysfs /sys -n -t sysfs
    fi
  fi
fi

# Read boot command line with builtin cat command
CMDLINE="$(cat /proc/cmdline)"
case "$CMDLINE" in *BOOT_IMAGE=debug\ *|*\ debug*) DEBUG="yes"; ;; esac

if [ -n "$DEBUG" ]; then
   echo "$0: pid=$$ ppid=$PPID"
   echo -e "\n\n"
   echo "Starting intermediate Shell as requested by \"debug\" option."
   echo "Type \"exit\" to continue with normal bootup."
   echo -e "\n\n"
   [ -x /static/ash ] && /static/ash || /bin/bash
fi 

My syslinux.cfg segment

label unRAID OS DEV
  menu default
  kernel bzimaged
  append initrd=bzrootd rootdelay=10 debug rdinit=/linuxrc

Also I modified my go script to handle the debug case.

considering I rebooted like ummm.. a million times, I had to stop empttp from initializing the array

Here's my go script

#!/bin/bash
# Start the Management Utility
sleep 1 ; 
if [ -e /proc/mdcmd ] 
   then echo trace > /proc/mdcmd; sleep 1
fi
if grep -wq debug /proc/cmdline
   then /boot/custom/etc/rc.d/S02-sync-etc
        /boot/custom/etc/rc.d/S20-install-sshd
        exit
fi
/usr/local/sbin/emhttp &
fromdos < /boot/custom/etc/rc.d/init.d/rc.local_startup | sh

Here is my S04-mount-custom.sh which creates the file based filesystem and mounts it.

For the record, you can use this now, without the union to allow /boot/custom to have a real filesystem with real permission control attributes. (and symlinks).

The script could use a variable to change the destination mount point...

But it was late and my sleep medicine was kicking in 

I can't believe I even got this far as I fell asleep at my desk a few times. 

#!/bin/bash
# Makes a disk file     
# if values are no then Run with    init_fs=yes  S04-fs.sh
# Mounts a swap file
# if values are no then Run with    mount_fs=yes S04-fs.sh

init_fs=yes
mount_fs=yes

fsfile=/mnt/disk1/.unraid.customfs
fsfile=/mnt/user/cache/.unraid.customfs
fsfile=/boot/config/unraid.customfs
size=256
size=128

if [ "${init_fs:=no}" == "yes" -a ! -f ${fsfile} ]
   then dd if=/dev/zero of=${fsfile} bs=1M count=${size:=128}
        chmod 600 ${fsfile}
        mkdir /mnt/custom
        mkreiserfs -f -f -l CUSTOMFS ${fsfile}
        # mount -o loop /mnt/custom
fi

if [ "${mount_fs:=no}" == "yes" -a -f ${fsfile} ]
   then [ ! -d /mnt/custom ] && mkdir /mnt/custom
        if ! mount | grep -wq ${fsfile}
           then reiserfsck -y ${fsfile} 
                mount -o loop ${fsfile} /mnt/custom
                # mount -L CUSTOMFS /mnt/custom
        fi
fi

if [ "${merge_fs:=no}" == "yes" ]
   then if grep aufs < /proc/mounts | grep -wq ramdisk 
           then echo "Merging ${fsfile} with /"
                mount -v -o remount,append:/mnt/custom=rw /
        fi
fi

I have a set of scripts for extracting, updating and rebuilding the bzroot. If there is interest I can make them available.

This is good stuff, WeeboTech.  It is pretty complex...I understood less than 10% of it, but looking at the wikipedia link, this would be a great addition to unRAID.  It would make developing packages a lot easier.  Good luck!

Forgive me for a stupid question, and it's probably already discussed elsewhere that I couldn't find, but why would you have to reinstall packages every time in the first place?  Once you're running unRAID on a full slackware distro can't you just install rtorrent and leave it be?  Is there something difficult about writing to the array from internal programs?

Once you're running unRAID on a full slackware distro can't you just install rtorrent and leave it be?

This is IF you are running on a full slackware distro.

I am not. Nor do I plan to. I like the minimal environment of a booting root ram filesystem.

There are just a few static files that I would like to leave somewhere and have the changes merged in. (Without having to write a sync script for every package.. that is getting old... LOL).

I have the gigabyte I-RAM drive and I want to use this for just a few packages and a few data files.

The purpose is to have as much in root ram as possible, while having the few static configs preserved in an easy to manage environment.

Gotchya. That I-RAM drive is pretty interesting - hadn't heard of it before.  What is the advantage of installing the packages every time and figuring out unionfs over having a static filesystem that you rsync to the ram drive on boot and back to the disk on shutdown?  If you're syncing the entire thing then you don't have to write individual sync scripts.

Gotchya. That I-RAM drive is pretty interesting - hadn't heard of it before.  What is the advantage of installing the packages every time and figuring out unionfs over having a static filesystem that you rsync to the ram drive on boot and back to the disk on shutdown?  If you're syncing the entire thing then you don't have to write individual sync scripts.

I think it's more for easier upgrading.

Files that don't need to reside in rootram, can be on a union filesystem somewhere.

I.E. shared libs, a compiler, the base files of a webserver (once a webserver is loaded, do I need to have all the files in ram?).

I was thinking more of having a filesystem image on the boot flash.

then you do a

ROOT=/boot/custom installpkg (your packagename)

and it's installed, but still lets the hard drives spin down.

Now when I upgrade to a later rev of unraid, I recompile the kernel with aufs, add in my scripts and my prior packages are there again.

Plus this lets me use (and write) packages that install (or look like they are installed) into /, but write through the root ram filesystem to a disk.

a mysql server for example. or with torrentflux or something that uses sqlite where you want the static data to exist on reboot.

> If you're syncing the entire thing then you don't have to write individual sync scripts.

True, but part of it is to allow incremental additions and testing so I can write packages to contribute.

Point is, as seasoned as I am, I don't want to run a full slackware distro on my file server.

if I'm going to do that, then I'll just use openfiler or something like that.

I want to install the bare minimum but also keep some static confg files accessible from the standard root tree.

Once you're running unRAID on a full slackware distro can't you just install rtorrent and leave it be? 

I'm not 100% opposed to running a full slackware install, but my thoughts are somewhat similar to WeeboTech's.  For me, the integrity of my data is the #1 concern.  My thought is that the more stuff that's running on a box, the more potential for bad interactions.  I'm not a linux expert, so maybe there's no good basis for my concern, but my approach has been to start with Tom's stable core software and then explicitly add only the specific functionality that I need.  Now that I have VMware Server running, I'm pretty happy because I should be able to put VMs on my box to do the other stuff knowing that they are playing in nice little protected boxes.  Had I not been able to get VMware onto my unRAID, I probably would have moved (eventually) to a full slackware install.  To me, it depends on what all you need to do, and for a lot of people it might make sense to run it one way or the other.

In my case, the potential of a union would just make some things easier.  For example, VMware is pretty heavy on the /tmp directory, so I have to move it to disk and create a link each reboot.  With union, I could just move it and it would be seamless.

My thought is that the more stuff that's running on a box, the more potential for bad interactions.  I'm not a linux expert, so maybe there's no good basis for my concern,

This is a Microsoft Windows thing. The more you install, add and run, the more unstable the OS is.

I've run linux servers with thousands of processes without instability of the OS.

What you have to watch for are files being used on the array.

my approach has been to start with Tom's stable core software and then explicitly add only the specific functionality that I need.

And this is my approach too. I just want to make it easier to add stuff and leave it there. LOL!!!