Docker read-only errors

This is the error I get when I try to add a container:

Error: open /var/lib/docker/tmp/GetImageBlob443829641: read-only file system

I've tried redoing BTRFS on the cache and deleting/rebuilding the docker from scratch.

I attached the diagnostics from right after I tried to add a new container.

tower-diagnostics-20170109-1131.zip

This is the error I get when I try to add a container:

 

Error: open /var/lib/docker/tmp/GetImageBlob443829641: read-only file system

 

I've tried redoing BTRFS on the cache and deleting/rebuilding the docker from scratch.

 

I attached the diagnostics from right after I tried to add a new container.

Will do.

I was running through the syslog and also found:

    blk_update_request: I/O error, dev sdb, sector 9342280

sdb is my only cache drive at the moment. I had a 3 drive pool before moving to the new server.

I recently updated to the following server and after transplanting the drives all these errors started to happen.

Server Chassis/ Case

SuperMicro SC826E16-R800LPB Chassis (High Performance SAS2/6GbS Expander)

Back Plane

BPN-SAS2-826EL1 826 backplane with single LSI SAS2X28 expander chip

Motherboard

X8DTN+

CPU Processor

2x Intel Xeon Six Core E5645 2.4Ghz

RAM Memory

64GB RAM DDR3 ECC REG DIMM (8x 8GB)

Hard Drives/ Caddies

12x 3.5" Drive Caddies

RAID Controller

LSI 9211-8i JBOD IT Mode HBA Card connected to Expander backplane

NIC Ports

On board Intel® 82576 Dual-Port Gigabit Ethernet Controller

Here are some of the things I've tried so far:

-btrfs scrub on the cache drive and docker image

-btrfs check --repair on the cache drive

-redoing the drive (changing the file system to XFS and then back as per the Check Disk Filesystems wiki)

-ran a memtest overnight, result: 1 pass with no errors

The cache seems to be okay when I restore the appdata and create the docker image, but once I try to add my old template containers back, the docker gives me the error in my post above and the drive filesystem seems to become corrupted.

I think I'm going to try moving the drive to a different slot in the hot swap bay. A lot of posts on here suggest that an I/O error is hardware rather than software. The drive is fairly new but I'll run a SMART test on it later today.

I've been trying to fix this for a few days with no success so let me know what you think or if I should try something else!

So I updated ca.backup 

Also, from my general support thread johnnie.black told me to try connecting the SSD cache drive via SATA to see if it would work and it DID! I can now add the docker containers without getting the "/var/lib/docker/tmp read-only filesystem" error while trying to download a container.

It seems to be working okay now, I wonder what was happening. Once I configure my docker image and containers back, should I try using the expander/backplane again? I just hate having to use a SATA2 port on the mobo for a SSD and also I have nowhere to actually mount it physically.