Request for Comments

Now that we have some experience with docker (it's great) we have come to the conclusion that we have not set it up optimally for unRaid.  In current version6-beta's we require docker to be on a btrfs volume, which is typically the 'cache' disk; and within a subdir called '/mnt/cache/docker'. This presents a number of problems which we need to solve asap.

 

Therefore what we want to do is the following:

 

- The "docker volume" will now be a loopback-mounted file, mounted at /var/lib/docker.  ie, $DOCKER_HOME=/var/lib/docker

 

- The Extensions/Docker page will let you specify the location of this file, which can be on any unRaid device: cache or array.  If the file does not exist it will be created, and you can specify the initial size (actually override a default).  After being created we will 'mkfs.btrfs' a btrfs file sytem (we still have DOCKER_OPTS="--storage-driver=btrfs")

 

- There is some status that tells you how full the volume file is with options to resize it.

 

These changes will solve some issues and add some flexibility:

 

- the 'New Permissions' script will not affect the contents of the docker volume

 

- it will be easier to ensure docker does not run out of free space.

 

- it will be easier to backup your docker volume - just copy it!

 

- you do not need to have any btrfs-formatted devices (but the docker volume could exist on a btrfs volume if you want)

 

This change will affect current plugins that manage docker, as well as perhaps some other minor tweaks.  We will be making these changes now and will release in -beta8.

 

Questions to the Community:

a) What do you think, see any major flaws with this?

b) How to migrate existing btrfs file system from /mnt/cache/docker to /var/lib/docker without forcing fresh recreation of all the containers?

Too technical for me.

 

I just want docker to work in a way where it doesn't start up the array when I use it or need to work with containers. How that is implemented doesn't concern me.

Too technical for me.

 

I just want docker to work in a way where it doesn't start up the array when I use it or need to work with containers. How that is implemented doesn't concern me.

 

What do you mean, that you don't want to have to have array Started?  Or that you don't want to have to have HDD's spinning up?

Too technical for me.

 

I just want docker to work in a way where it doesn't start up the array when I use it or need to work with containers. How that is implemented doesn't concern me.

 

What do you mean, that you don't want to have to have array Started?  Or that you don't want to have to have HDD's spinning up?

 

Mainly the second point, not spinning up.  I keep everything on the cache at the moment to ensure the array disks only spin up when a direct action kick to the array kicks in. (copy or move a file)

 

But it would be handy to keep the containers up even if I chose to stop the array.  It doesn't happen often but there have been time swhere the array stopped and started after configuring something and all containers came to a screaming halt.

Too technical for me.

 

I just want docker to work in a way where it doesn't start up the array when I use it or need to work with containers. How that is implemented doesn't concern me.

 

What do you mean, that you don't want to have to have array Started?  Or that you don't want to have to have HDD's spinning up?

 

Mainly the second point, not spinning up.  I keep everything on the cache at the moment to ensure the array disks only spin up when a direct action kick to the array kicks in. (copy or move a file)

 

But it would be handy to keep the containers up even if I chose to stop the array.  It doesn't happen often but there have been time swhere the array stopped and started after configuring something and all containers came to a screaming halt.

 

Keeping the docker volume on the cache will prevent spin up of the array, unless of course a running docker app accesses something on the array.

 

Keeping the docker volume on the cache will prevent spin up of the array, unless of course a running docker app accesses something on the array.

 

Yes, that would be my main request for anything new.  I live in the tropics and the server is in a room where aircon isn't always one.  Drives are in their low 40Cs when idling so I like to have them spun down as often as possible.

I think that is a good idea.  That would eliminate the requirement for btrfs on the cache volume and give you the flexibility to store the docker files anywhere you choose.

 

Would a simple rsync work for migrating the data from /mnt/cache/docker to the new path?

Would a simple rsync work for migrating the data from /mnt/cache/docker to the new path?

That or a simple 'cp' will do the job but problem is that each 'snapshot' (docker layer) will get 'expanded' in it's full glory.  Your 3GB of docker containers will end up at 60GB or more 

Since I am assuming this change does not impact the current container config files does it really matter if you need to wipe and rebuild the current containers? It takes a few minutes to download the container, which isn't a big deal. If we can still point to the current config folders then 95% of the work doesn't need to be done anyways, right?

 

Or am I missing something?

I think I sort of understand the solution, but I'm not entirely clear on what the problem is.

 

Is the problem caused by having the drive where docker lives too full?

 

Are there any other manifestations of this problem anticipated, or any other anticipated problems that would be addressed by this solution?

 

Is recreating the containers really a big issue? Don't they mostly store their persistent data on a drive somewhere and we could just point the new instances to that data? (I see bkastner already asked this but I will leave it as another way of saying the same thing in case it helps anyone.)

 

 

Since I am assuming this change does not impact the current container config files does it really matter if you need to wipe and rebuild the current containers? It takes a few minutes to download the container, which isn't a big deal. If we can still point to the current config folders then 95% of the work doesn't need to be done anyways, right?

 

Or am I missing something?

 

That is true, and with the current docker plugin using the template, it is pretty easy to recreate the docker just as it was.  ( At least for the handful I use.. )  Or even starting from scratch, isn't that big a deal... With using a beta, you have to be prepared for some work..

My containers are not on the cache drive. They are on a separately mounted SSD. This is where I want them to stay.

 

Will these changes affect me?

My containers are not on the cache drive. They are only a separately mounted SSD. This is where I want them to stay.

 

Will these changes affect me?

No.

I think I sort of understand the solution, but I'm not entirely clear on what the problem is.

There are two problems.  First is that 'New Permissions' script walks its way down every path on every device.  When it hits the 'docker/btrfs' directory it changes all the permissions such that executables in there no longer can execute ('x' bit is cleared) - this is not good.  The New Perm. script is necessary for a couple reasons: one to handle upgrade from earlier unRaid releases, and another more important, to fix things  up again when someone transitions from Active Directory mode back to workgroup.  There are other times it's nice to run this script.

 

Is the problem caused by having the drive where docker lives too full?

That's the other more serious issue.  Docker fails rather miserably when it runs out of space so it's important to monitor it and have tools to resize where it's kept.  When it's stored on the cache as is mostly done now (or on the array with beta7), if that device fills up it might be difficult to resize it larger, or move other stuff off to give docker more space.  By using a loop mounted file we can reserve space ahead of time (we use fallocate).

 

Another benefit is that you can stop docker, move the image file somewhere else, and then restart docker.  Moving the image file is a lot easier than moving the entire btrfs docker tree.  In addition this offers the possibility of creating a pre-built image file with a lot of popular docker apps already installed and ready to go.

 

Is recreating the containers really a big issue? Don't they mostly store their persistent data on a drive somewhere and we could just point the new instances to that data?

I dont' think it is, but I thought I'd ask the Community.

I like the basic idea of not tying docker to be run from a btrfs formatted drive.

 

Will it be possible to keep the cache drive mounted when the array is stopped (or is this a separate feature request).  I can see instances where this could be advantageous, particularily when running docker containers or VMs.

 

Thinking about it we might want to indicate whether individual docker containers should be stopped if the array is stopped because they need access to files on the array.  If the cache drive can be left mounted as indicated above then this might need to be another thing to be taken into consideration?

This seems like an elegant solution to me.

 

I would suggest though that whilst we are at this we consider the role and importance of the non containerized settings data i.e. what we have so far standardized on as /mnt/user/appdata

 

Arguably this is as critical that the docker containers themselves as they are typically the settings that make a generic application user specific. In fact almost none of the popular images here would work without /mnt/user/appdata volume imports.

 

The enhanced docker addon will store all settings under this path e.g /mnt/user/appdata/mariadb and whilst i think this specific storage scheme lacks the ability to run multiple identical dockers it is still very practical.

 

On top of this we have the template XML which is essentially the command line switch that will load the container from the image with the user specific volumes etc

 

IMHO all of these go hand in hand and should have a similar, if not identical, on disk storage solution.

Are there performance implications running docker and docker apps from the loopback filesystem?

 

Another option is to use mount -o bind, but that opens up other issues that the loop back solves.

 

Will there be webGui functionality to

1. mount/unmount

2. fsck the loop back.

 

 

 

I just got everything I wanted running in docker setup and working correctly using the web extension. What is going to happen to all of my installed apps? It looks like just the docker images are moving and they will still reference the appdata folder on the cache?

I was confused by this change and did some research to try and understand. I think I understand now and hope I can help others. If I'm wrong I'll just confuse everyone further.

 

Research indicates that a loopback file system is a file whose contents are mounted as a filesystem. I use VMs and this is basically what they do. Each disk in the VM is one large file in the host OS. And that file can start out small and grow over time as the logical disk inside the VM needs the space. Please correct if this isn't an accurate description of a loopback FS.

 

So if you created a btrfs loopback FS with the host file on the cache disk, it would solve your issues. In essence your loopback FS would appear as a separate dedicated disk for Docker containers. Users would still have theirDocker templates (maintained by the plugin) and recreate the containers which would dl whatever it needed and store in the loopback FS. This is necessary rather than trying to copy the existing Docker directory to the loopback FS. If, like me, you have a dedicated SSD or other disk mounted outside the array, you would not need to use the loopback and it would just access the actual filesystem.

 

Is this accurate?

 

Now it you didn't use btrfs, are you saying the Dockers would start taking up huge amounts of space? Why would anyone want that option?

 

Users using btrfs on their cache drives, would they need to reformat or anything? I'm thinking not. They could just delete the Docker directory when the time comes and let the 6b8 version do its magic. Them just load the plugin and recreate the containers and they ate loaded on the loopback FS. 10 minutes of DLing and done. You may need to adjust  paths in the Docker template if you appdata moved.

Research indicates that a loopback file system is a file whose contents are mounted as a filesystem. I use VMs and this is basically what they do. Each disk in the VM is one large file in the host OS. And that file can start out small and grow over time as the logical disk inside the VM needs the space. Please correct if this isn't an accurate description of a loopback FS.

 

For the most part you are correct.

As far as I understand, the loopback filesystem in linux is 'pre-created' to it's full size.

I could be wrong in the implementation that limetech will use.

I'm assuming the loopback filesystem file for docker will be pre-created to avoid running out of space in the docker filesystem.

 

An analogy is an ISO file, that is mounted via loopback and then becomes accessible as local files.

From another thread:

shares that will experience a lot of small file updates probably should not be cow.

 

Currently we are storing plex data and other Docker settings in /mnt/user/appdata, which for most people is on their BTRFS-formatted cache drive with cow.

 

What happens to this data as part of this change?  Does cow get disabled on the cache drive or do we need to move the data to another disk with a different fs?

 

 

Also:

Hmm, ok this is probably serious enough for us to get out a -beta7a asap.

 

Are we still doing a 7a or going straight to 8?  Are we expecting this to come quickly or is it more intense on your end? (I am getting the "No space left on device (28)" error and wondering what to do to fix it)

 

 

BTW, I think it is fine to just recreate the docker containers in the new location.  They don't contain any data and are easy to rebuild.  It is a beta after all

I am curious for me /mnt/user/appdata is only a few hundred MB but do other run to multi GB?

I am curious for me /mnt/user/appdata is only a few hundred MB but do other run to multi GB?

Mine is 2GB and I only have 4 dockers. 1.8GB of that is plex, and I know I have a small plex library compared to most.