Correct docker setup - Optimal?

I'm checking in to make sure that the way I'm running docker on my setup is the best / recommended / optimal way...?

It's a HP Microserver N54L with 4GB RAM, 4x 4TB drives (no cache) and 1x 60GB SSD (unassigned device(. I've a 10GB docker image on the SSD and the appdata is on the array. The appdata is backed up with CA, but nothing for the SSD. In terms of dockers, I've 3 running - Couchpotato, Sonarr and nzbHydra.

I'm planning to run a couple more dockers, but before adding any, I wanted to check that how I'm running it is correct? To me, something doesn't feel right...

I thought the 10GB docker image would grow by 10GB each time I added a new docker? Also, the SSD isn't backed up? How about speed, what is the SSD running and what is running from appdata? Do I even need the SSD?

Sorry for the barrage of questions, I'm just struggling to get my head around how dockers run and how I should really have it configured to ensure ease of use, redundancy etc.

Any help, suggestion or questions appreciated as keen to optimise, set and forget 

Edited to add that SSD is an unassigned device.

It sounds like your setup is fine. The Docker image is a BTRFS filesystem image. Think .iso file, but for a BTRFS filesystem instead of a DVD.

You don't need to backup the Docker image. When you click the add new Docker button there is a drop-down and you can pick the my-XXX for your previously installed Dockers (for example my-Couchpotato) and it will re-install the Docker with all your previous settings and start working again as if nothing has happened. So, if you can restore the appdata to a new SSD then you can re-install the Dockers.

Since you mentioned optimal...

Reading and writing to an SSD is a lot faster than the parity protected array.  You aren't getting optimal speeds by having either the Docker image file or appdata located on the array.  It'll work - but it's not optimal.  Typically both the Docker image and appdata are located on the cache drive or an unassigned SSD, and then the appdata is backed up periodically to the array.  If you're happy with the current setup, though, it's not inherently wrong.

Oh, and the Docker image does not grow by 10GB every time you add a docker.  A 10GB image will be fine for a half a dozen or more Docker containers, depending on how much space they require.

4x 4TB drives (no cache) and 1x 60GB SSD.

4x 4TB drives (no cache) and 1x 60GB SSD.

Could you clarify this? One possible interpretation is you don't have a cache disk and you have an SSD as part of the parity array. That will be less than ideal. I think everyone has responded so far assuming that your SSD is an Unassigned Device, but you didn't really say.

Sorry, yes, the SSD is an unassigned device.

Since you mentioned optimal...

 

Reading and writing to an SSD is a lot faster than the parity protected array.  You aren't getting optimal speeds by having either the Docker image file or appdata located on the array.  It'll work - but it's not optimal.  Typically both the Docker image and appdata are located on the cache drive or an unassigned SSD, and then the appdata is backed up periodically to the array.  If you're happy with the current setup, though, it's not inherently wrong.

 

Oh, and the Docker image does not grow by 10GB every time you add a docker.  A 10GB image will be fine for a half a dozen or more Docker containers, depending on how much space they require.

Great, thanks for the feedback. I guess this then leads to another couple of questions;

1. Should I just increase the docker image to the full size of the SSD, as that's all it's ever going to be used for? Or...

2. Should I move the appdata to the SSD too? Or...

3. It it 'really' worth using the SSD drive at all? I'm not too worried about squeezing out the last millisecond of speed if it's more 'complicated' to set it up on SSD and just put everything on the array?

What additional Dockers do you plan to use?

What additional Dockers do you plan to use?

Nothing too taxing. Plexconnect, PlexPY and perhaps Watcher & / or Radarr.

There are a couple of disadvantages to having Docker and appdata on the array.  First, Dockers are applications and you're asking them to do work.  This means they're constantly active and will keep array drives spun up - many folks like to let them spin down.  Also, I/O against the array tends to be slow, high latency I/O compared to an SSD.  I/O heavy Dockers like Plex transcoding can really bog down on the array.  VMs don't work well on the array, either.

If it were me I'd bump the Docker image to 20GB and move appdata back to the SSD using cache "prefer".  Use CA Backup to backup appdata periodically.  I don't know what kind of I/O Plexconnect does, so it may be fine to leave appdata on the array if the rest of these Dockers really don't tax the system much.  If you're not sitting watching a Docker do its work it probably doesn't matter how long it takes to do it.

There are a couple of disadvantages to having Docker and appdata on the array.  First, Dockers are applications and you're asking them to do work.  This means they're constantly active and will keep array drives spun up - many folks like to let them spin down.  Also, I/O against the array tends to be slow, high latency I/O compared to an SSD.  I/O heavy Dockers like Plex transcoding can really bog down on the array.  VMs don't work well on the array, either.

I'm not too worried about the drives spinning down, as they're accessed frequently, so not really a problem. I also won't be using Plex on the server, as it's not powerful enough to transcode.

If it were me I'd bump the Docker image to 20GB and move appdata back to the SSD using cache "prefer".  Use CA Backup to backup appdata periodically.  I don't know what kind of I/O Plexconnect does, so it may be fine to leave appdata on the array if the rest of these Dockers really don't tax the system much.  If you're not sitting watching a Docker do its work it probably doesn't matter how long it takes to do it.

OK, I've increased the size of the docker image to 20GB.

In terms of moving the appdata, what would be the correct process to do this in order to keep all the settings in place?

Also I see you mention cache, would this be how to do it rather than unassigned devices?

Also I see you mention cache, would this be how to do it rather than unassigned devices?

Also I see you mention cache, would this be how to do it rather than unassigned devices?

That is the "normal" way of setting it up in unraid, so most of the tutorials / directions / etc are going to assume you are using the cache slot for your docker and apps. You don't actually have to use the cache slot as a traditional cache drive for your array shares if you don't want to.

OK, makes sense and I do see more referrals to cache than unassigned devices when reading about dockers.

So I'd add the SSD as a cache drive, then set 'appdata' as preferred to the SSD? What about mover?

Also, can I do all this without losing my settings, or am I best to start again from scratch 

Also I see you mention cache, would this be how to do it rather than unassigned devices?

That is the "normal" way of setting it up in unraid, so most of the tutorials / directions / etc are going to assume you are using the cache slot for your docker and apps. You don't actually have to use the cache slot as a traditional cache drive for your array shares if you don't want to.

OK, makes sense and I do see more referrals to cache than unassigned devices when reading about dockers.

 

So I'd add the SSD as a cache drive, then set 'appdata' as preferred to the SSD? What about mover?

 

Also, can I do all this without losing my settings, or am I best to start again from scratch 

Also I see you mention cache, would this be how to do it rather than unassigned devices?

That is the "normal" way of setting it up in unraid, so most of the tutorials / directions / etc are going to assume you are using the cache slot for your docker and apps. You don't actually have to use the cache slot as a traditional cache drive for your array shares if you don't want to.

OK, makes sense and I do see more referrals to cache than unassigned devices when reading about dockers.

 

So I'd add the SSD as a cache drive, then set 'appdata' as preferred to the SSD? What about mover?

 

Also, can I do all this without losing my settings, or am I best to start again from scratch 

Stop the docker service. After setting appdata to cache-preferred, run mover and it will move appdata to cache. If you always referred to appdata as /mnt/user/appdata, then everything should find it there after it is moved to cache. Restart docker service.

Makes sense, thanks. However, what about the docker image, as this is already on the SSD through unassigned devices?

Also I see you mention cache, would this be how to do it rather than unassigned devices?

That is the "normal" way of setting it up in unraid, so most of the tutorials / directions / etc are going to assume you are using the cache slot for your docker and apps. You don't actually have to use the cache slot as a traditional cache drive for your array shares if you don't want to.

OK, makes sense and I do see more referrals to cache than unassigned devices when reading about dockers.

 

So I'd add the SSD as a cache drive, then set 'appdata' as preferred to the SSD? What about mover?

 

Also, can I do all this without losing my settings, or am I best to start again from scratch 

Stop the docker service. After setting appdata to cache-preferred, run mover and it will move appdata to cache. If you always referred to appdata as /mnt/user/appdata, then everything should find it there after it is moved to cache. Restart docker service.

Makes sense, thanks. However, what about the docker image, as this is already on the SSD through unassigned devices?

Also I see you mention cache, would this be how to do it rather than unassigned devices?

That is the "normal" way of setting it up in unraid, so most of the tutorials / directions / etc are going to assume you are using the cache slot for your docker and apps. You don't actually have to use the cache slot as a traditional cache drive for your array shares if you don't want to.

OK, makes sense and I do see more referrals to cache than unassigned devices when reading about dockers.

 

So I'd add the SSD as a cache drive, then set 'appdata' as preferred to the SSD? What about mover?

 

Also, can I do all this without losing my settings, or am I best to start again from scratch 

Stop the docker service. After setting appdata to cache-preferred, run mover and it will move appdata to cache. If you always referred to appdata as /mnt/user/appdata, then everything should find it there after it is moved to cache. Restart docker service.

Makes sense, thanks. However, what about the docker image, as this is already on the SSD through unassigned devices?

After stopping docker service, delete docker.img (on that page), reassign SSD to cache, then do the rest of the above except change the location of docker.img to cache before you restart docker service.

Great stuff, thanks. I've just tweaked my settings and all went well. I needed to reinstall the dockers, which I guess was the result of deleting the old docker.img, but all the settings remained. All seems well so far, thanks!

Am I correct in thinking 'mover' will still move the files from cache, or will they purely stay on cache now?

Am I correct in thinking 'mover' will still move the files from cache, or will they purely stay on cache now?

Try that and let me know if you don't understand how it applies to your question.

Am I correct in thinking 'mover' will still move the files from cache, or will they purely stay on cache now?

If you go to the settings page for a user share and turn on Help it will explain the various settings including the use cache setting. The Help button is a toggle and will stay on until you turn it off. If it is turned on the webUI will show you help for any of its pages.

 

Try that and let me know if you don't understand how it applies to your question.

Great, thanks for the pointer. I read the help, but I always thought that moving the data from the cache wasn't great for dockers? Of course, I could have misread or misunderstood. So if prefer is fine, then I'm fine 

I've also run a 'Fix common problems' plugins and it's complaining about permission for the appdata backup folder and similar names, both for SMB. Is this something I need to worry about?

Infact, thinking about permissions, that explains why CrashPlan is unable to backup the appdata backup due to these permissions. How can I fix this?

Am I correct in thinking 'mover' will still move the files from cache, or will they purely stay on cache now?

If you go to the settings page for a user share and turn on Help it will explain the various settings including the use cache setting. The Help button is a toggle and will stay on until you turn it off. If it is turned on the webUI will show you help for any of its pages.

 

Try that and let me know if you don't understand how it applies to your question.

Great, thanks for the pointer. I read the help, but I always thought that moving the data from the cache wasn't great for dockers? Of course, I could have misread or misunderstood. So if prefer is fine, then I'm fine 

 

I've also run a 'Fix common problems' plugins and it's complaining about permission for the appdata backup folder and similar names, both for SMB. Is this something I need to worry about?

 

Infact, thinking about permissions, that explains why CrashPlan is unable to backup the appdata backup due to these permissions. How can I fix this?

Maybe ask on FCP thread or CA Backup thread for your other question.

Am I correct in thinking 'mover' will still move the files from cache, or will they purely stay on cache now?

If you go to the settings page for a user share and turn on Help it will explain the various settings including the use cache setting. The Help button is a toggle and will stay on until you turn it off. If it is turned on the webUI will show you help for any of its pages.

 

Try that and let me know if you don't understand how it applies to your question.

Great, thanks for the pointer. I read the help, but I always thought that moving the data from the cache wasn't great for dockers? Of course, I could have misread or misunderstood. So if prefer is fine, then I'm fine 

 

I've also run a 'Fix common problems' plugins and it's complaining about permission for the appdata backup folder and similar names, both for SMB. Is this something I need to worry about?

 

Infact, thinking about permissions, that explains why CrashPlan is unable to backup the appdata backup due to these permissions. How can I fix this?

Sounds like maybe you have misunderstood somehow. cache-prefer will not move data from cache, it will move data to cache.

Sorry, my bad, I did indeed misread how the data is located / moved. I guess you gain speed of the SSD cache over the redundancy of the array.

Maybe ask on FCP thread or CA Backup thread for your other question.

Will do, as following from the above, with it being cache only, having the backup running is important.

Thanks again, your patience and help much appreciated!!!