Typing shell 'sync' command results in continuous disk writes

Open a telnet session, type 'sync' and from then on something is causing small number of writes to every disk.  As a result disks won't spin down.

 

See:

http://lime-technology.com/forum/index.php?topic=27720.msg244787#msg244787

 

I have been doing some tests trying to find something about this sync issue (infinite disk writes every 5 seconds after running sync) and will share what I did found, as I think it may eventually help or at least eventually save Tom some time doing tests...

 

- With unraid rc13 as base I did "repacked" it to a few different kernel versions and tested on each of them if this issue happened with each kernel (note that because unraid md driver would not compile with some different kernel versions then I did not included it at all with any kernel version, nor the patched libata file... then it was just vanilla kernel based on rc13 .config, also obviously because I had not unraid md driver then I endend testing it by just manually mounting a reiserfs partition - don't worry about unsyncing my array, I did it on a vm with test virtual hdds! - and then testing by running sync and looking at vmware activity "led" for that virtual hdd, very easy to see the writes every 5 seconds that way, after running sync), the result was:

 

3.9.5 - issue happens

3.9.1 - issue happens

3.8.13 - issue happens

3.7.10 - issue happens

3.4.26 - issue does NOT happen (tested this specific version as it's the one on rc11a that I still have on my real unraid server, without this problem for sure)

3.4.48 - issue does NOT happen

 

Seems really a kernel thing, despite I find it strange how it may be present on so many versions? weird... however, me thinking if that couldn't be caused by something else on unraid, something set on some init script or something, other than kernel... I did one more test, got ubuntu 13.04 live disc (with kernel 3.8.0), booted it on a vmware vm with my unraid test virtual hdd's, did manually mounted reiserfs partition from one of the virtual hdd's, exactly as I tested before with multiple kernels on unraid, virtual hdd activity "led" was off.. until I ran sync... after doing it it started blinking every 5 seconds exactly as with unraid...

 

just a last note: after tests I did booted normal rc13 again on vm with these test hdd's, did parity check and there was 3xx errors, ok I was surely expecting errors as I mounted partitions manually/directly from the hdd's, out of the array, but isn't it 3xx too much just for mounting partitions, without doing any kind of file write on them? (not very important probably, just a note)

 

sync --version on ubuntu returns 8.20.

 

Btw, on sync man page it says:

The sync program does nothing but exercise the sync(2) system call.

if all it does is doing a system call don't think version will do big difference?

 

Re mount options, I did already thought about that and searched a bit but could not find anything useful so far, think it's possible to increase commit time, that maybe would make difference but... we just want it does not write anything at all unless there is something to write (i.e. exactly like it is before we do sync).

hmm, no idea too, btw I did just looked a bit more at mount options and tried some... even tried disabling journalling (nolog) for testing and can always see the every 5 sec. disk access after sync (why only after sync? ...), the only way I could not see it was mounting it 'ro'

 

Edit: Searching a bit for:

https://www.google.com/search?q=reiserfs+%225+seconds%22

we can find some apparently related info but... most refer to atime related and I did surely tried to mount it with 'noatime' option and still can reproduce it.

 

Edit2: After searching a bit I'm wondering if it may be related with changes done to get rid of sync_supers since kernel 3.6.

 

Edit3: Just tried to repack rc13 with kernels 3.6.11, 3.5.7 and 3.5.1 and can confirm that the "problem" happens with both, apparently this behavior happens with all versions after 3.4.x...

hmm, no idea too, btw I did just looked a bit more at mount options and tried some... even tried disabling journalling (nolog) for testing and can always see the every 5 sec. disk access after sync (why only after sync? ...), the only way I could not see it was mounting it 'ro'

 

Edit: Searching a bit for:

https://www.google.com/search?q=reiserfs+%225+seconds%22

we can find some apparently related info but... most refer to atime related and I did surely tried to mount it with 'noatime' option and still can reproduce it.

 

Edit2: After searching a bit I'm wondering if it may be related with changes done to get rid of sync_supers since kernel 3.6.

 

Edit3: Just tried to repack rc13 with kernels 3.6.11, 3.5.7 and 3.5.1 and can confirm that the "problem" happens with both, apparently this behavior happens with all versions after 3.4.x...

Nice work nars, I have confirmed the same, that this problem was introduced in 3.5 kernel.  It also appears limited to reiserfs; I don't see this with ext3 or FAT.  I posted a 'bug report' to the reiserfs development newsgroup, you can follow it's progress here:

http://thread.gmane.org/gmane.comp.file-systems.reiserfs.general/24257

 

I'm all-but-certain this bug is also the cause of "Crash/hang when trying to Stop array".

hmm, no idea too, btw I did just looked a bit more at mount options and tried some... even tried disabling journalling (nolog) for testing and can always see the every 5 sec. disk access after sync (why only after sync? ...), the only way I could not see it was mounting it 'ro'

 

Edit: Searching a bit for:

https://www.google.com/search?q=reiserfs+%225+seconds%22

we can find some apparently related info but... most refer to atime related and I did surely tried to mount it with 'noatime' option and still can reproduce it.

 

Edit2: After searching a bit I'm wondering if it may be related with changes done to get rid of sync_supers since kernel 3.6.

 

Edit3: Just tried to repack rc13 with kernels 3.6.11, 3.5.7 and 3.5.1 and can confirm that the "problem" happens with both, apparently this behavior happens with all versions after 3.4.x...

Nice work nars, I have confirmed the same, that this problem was introduced in 3.5 kernel.  It also appears limited to reiserfs; I don't see this with ext3 or FAT.  I posted a 'bug report' to the reiserfs development newsgroup, you can follow it's progress here:

http://thread.gmane.org/gmane.comp.file-systems.reiserfs.general/24257

 

I'm all-but-certain this bug is also the cause of "Crash/hang when trying to Stop array".

 

So, that means back to the 3.4.x. kernel then?

 

Yes, to the latest in the series: 3.4.48

 

That may not be bad news though. If I understood you right, you went to 3.9.x mainly to solve those "slow writes" problem.

Right.

 

I believe that that can be achieved with 3.4.x, if you abandon (for now) support for more than 4GB RAM. (We'll use more RAM later, when you roll out the 64-bit unRaid.

 

I tested a custom kernel for the v5-RC12a, and my system was flying! The only difference in .config was:

CONFIG_HIGHMEM4G=y
# CONFIG_HIGHMEM64G is not set
CONFIG_X86_PAE=n

I am sure that you can make a very stable build this way.

 

There are two choices here:

 

The Red pill: leave PAE support in (because not all chipsets with >4GB RAM exhibit this issue), but document the correct 'mem' kernel setting to boot with (provide a boot option to do this)

 

The Blue pill: build kernel without PAE to limit all platforms to 4G. This makes things more robust and less confusing because there is no choice to be made by the user, at the expense of taking away a feature (greater than 4GB RAM support).

 

Either choice will be documented as a "workaround" until this sync/crash issue is fixed in later kernels.

 

So what do you think?  Take the Red pill or take the Blue pill?

Red pill , but how about cache pool ?

Red pill , but how about cache pool ?

Thanks for your input Neo 

 

Pill choice does not affect 'cache pool'.

Red pill

Red pill , but how about cache pool ?

 

Whats cache pool?

I would suggest the Red Pill. 

 

However perhaps the 4GB memory option should be the default boot option and the option to use more RAM be the alternative rather than the other wy round as in the current RC..  One can then include in the release notes how to edit the syslinux.cfg file to change the default?  That way anyone electing to use more than 4GB is electing to do so deliberately.

Red pill.

Perhaps this could be a poll.

 

Document the correct 'mem' kernel setting to boot with (provide a boot option to do this)

 

Also, update the syslinux.cfg with the option so novices can easily select which option to use.

However perhaps the 4GB memory option should be the default boot option and the option to use more RAM be the alternative rather than the other wy round as in the current RC..  One can then include in the release notes how to edit the syslinux.cfg file to change the default?  That way anyone electing to use more than 4GB is electing to do so deliberately.

 

I like that!

However perhaps the 4GB memory option should be the default boot option and the option to use more RAM be the alternative rather than the other wy round as in the current RC..  One can then include in the release notes how to edit the syslinux.cfg file to change the default?  That way anyone electing to use more than 4GB is electing to do so deliberately.

 

I like that!

+1, user needing >4GB should be an 'advanced' user anyway... needing to run more apps on server, etc... else 4GB would be more than suffice for normal stock unraid usage...

 

a 3rd option could be maybe compile two kernel images, one with PAE off (default boot) and other with PAE on, until 64-bit available, but it would be a bit more work for you... and if mem option is ok for all users with slowdown problem (I didn't follow all that thread as I have not the problem...) then it should be really suffice.

There was a question.  Red Pill / Blue Pill. I.E. Reaching out for what the community wants.

A poll provides that feedback very fast.

 

itimpi's suggestion makes it easy for people to use whatever functionality they choose with the most basic option being covered by default.

 

Can we detetc if >4GB is installed even when the 4GB oiption is the default? I would like to have emHTTP alert potential users of this.

 

I agree 4GB is alot compared to what we are used to but it is NOT a lot in terms of what people buy today. I bet you money more people have 8GB + of ram on new builds than dont

There was a question.  Red Pill / Blue Pill. I.E. Reaching out for what the community wants.

Purple pill.  Executive decision. 

 

Aren't you just dying to see this thing go final?

 

4GB is still a HUGE amount.  Seriously, we've lost perspective here.  And it's not like I haven't installed 16GB RAM on each of my servers.

 

I have not lost perspective. My perspective is a reliable release. I'm not dying for anything to go final when I know there are problems.  I'm using rc12 and it's pretty stable.

 

I've configured each of my virtualized machines to use 1GB, it's enough for now.

My bare metal is 4gb which is enough for a very busy torrent file server.

 

I don't think unRAID needs to have that much ram, however some people install a ton of plugins. They end up going into the Ram drive.  Why limit them if it's not necessary.

rc12 is pretty stable for most users. I think tom should replace rc12  to final for now.

 

For rc13 issues - have that fixed in 5.1 or 5.0.1

There are always going to be problems, I'm just trying to get to a stable release without any regressions.  Looks like I'm going to have to live with the 4GB RAM limitation on some platforms for the time being.  What I'm going to do then is release -rc14 with the stable 3.4.48 kernel and the Realtek-supplied drivers for Realtek NIC's.  I'm going to implements itimipi's suggestion of making 4GB RAM limitation the "default" and let you guys with the right platform and more RAM edit your syslinux.cfg files.  There are fixes in the code for some of the other issues, and I need to concentrate on getting those ironed out.  The reason I didn't post a poll is because the issue itself is fairly "esoteric" and probably the only people knowledgeable enough to cast a valid vote are reading this thread anyway.  At some point the reiserfs folks are going to go up to San Quentin and ask Hans wtf is wrong and when a patch gets released then I'll revisit the latest kernel, but this will not be until after 5.0 is released.

Nice work nars, I have confirmed the same, that this problem was introduced in 3.5 kernel.  It also appears limited to reiserfs; I don't see this with ext3 or FAT.  I posted a 'bug report' to the reiserfs development newsgroup, you can follow it's progress here:

http://thread.gmane.org/gmane.comp.file-systems.reiserfs.general/24257

 

I'm all-but-certain this bug is also the cause of "Crash/hang when trying to Stop array".

So is this going to be an ongoing thing, chasing kernel bugs related only to reiserfs, since reiserfs is the bastard stepchild of file systems? Are you investigating alternatives like XFS for the core protected unraid drives?

There are always going to be problems, I'm just trying to get to a stable release without any regressions.  Looks like I'm going to have to live with the 4GB RAM limitation on some platforms for the time being.  What I'm going to do then is release -rc14 with the stable 3.4.48 kernel and the Realtek-supplied drivers for Realtek NIC's.  I'm going to implements itimipi's suggestion of making 4GB RAM limitation the "default" and let you guys with the right platform and more RAM edit your syslinux.cfg files.  There are fixes in the code for some of the other issues, and I need to concentrate on getting those ironed out.  The reason I didn't post a poll is because the issue itself is fairly "esoteric" and probably the only people knowledgeable enough to cast a valid vote are reading this thread anyway.  At some point the reiserfs folks are going to go up to San Quentin and ask Hans wtf is wrong and when a patch gets released then I'll revisit the latest kernel, but this will not be until after 5.0 is released.

 

Works for me. Thanks for your hard work!!!

So is this going to be an ongoing thing, chasing kernel bugs related only to reiserfs, since reiserfs is the bastard stepchild of file systems? Are you investigating alternatives like XFS for the core protected unraid drives?

 

There have been very few kernel bugs isolated to reiserfs, and it's not the only file system to have issues.  Actually reiserfs really is very good for what we use it for.  Easy and fast to make and expand (try building ext3/4 on a 4TB hard drive - takes a loooong time).  Very resilient - I've helped people restore nearly all their files even after disk was mistakenly plugged into parity slot and parity-sync started.  I think it's main fault is simply it's name.  If it was named "NewFS" or something, then I'm pretty sure it would still be popular.  Same with Reiser4 - if was named "NewImprovedFS" there would not be a btrfs... IMHO, I don't mean to start a FS discussion, so let's not.  Yes other file system support is certainly at the top of the feature list to add.

 

Why is reiserfs marginalized?

 

http://serverfault.com/questions/68715/why-is-reiserfs-marginalized

Actually reiserfs really is very good for what we use it for.

...

Easy and fast to make and expand

...

Very resilient - I've helped people restore nearly all their files even after disk was mistakenly plugged into parity slot and parity-sync started.

I was never much of a fan for reiserfs.

A few years on the forum and I realized how well it's worked for unRAID.

 

We've seen people dd onto the first part of the drive erasing the partition table and still be able to recover most of their data.

 

Personally I had a bad spot on my hard drive that corrupted the superblock.

I was able to rebuild the superblock from the data on the drive and recover most of my data.

 

I don't mean to start a FS discussion, so let's not.  Yes other file system support is certainly at the top of the feature list to add.

Not to start a FS discussion, just a quick question, I'm curious: Does the md-mod driver even know about filesystems? Or about partitions for that matter? Is it only emhttp that's taking care of partitioning/formating/mounting etc?

 

From what I know the md-mod is derived from linux software raid modules.

Those modules do not care about the filesystem specifics. 

The partitioning/formating/mounting and expansion is done by emhttp.

I don't mean to start a FS discussion, so let's not.  Yes other file system support is certainly at the top of the feature list to add.

Not to start a FS discussion, just a quick question, I'm curious: Does the md-mod driver even know about filesystems? Or about partitions for that matter? Is it only emhttp that's taking care of partitioning/formating/mounting etc?

 

From what I know the md-mod is derived from linux software raid modules.

Those modules do not care about the filesystem specifics. 

The partitioning/formating/mounting and expansion is done by emhttp.

 

That much I knew.  I was hoping for a line from Tom, to confirm that the modifications weren't such as to make the md driver aware about partitions, or filesystems, or spinning stuff up and down, you know, things that aren't any of md's business.  I am not a C programmer so I can't easily look in that code. But if the above still holds true -- i.e., the md driver will work with whatever block devices you pass to it, be it partitions on a disk, or the disk itself -- then why do we really even bother with partitions? We can just as easily create the filesystem (be it reiserfs or something else) on the disk itself, and md should still do its job. Right?

 

I always thought same, but I did now tested it for curiosity on my test system to start array on maintenance mode, then manually format /dev/mdX device as ext4 (also tried ext3) and then manually mounting it... at 1st it appeared to work, even did an ls on it and did see lost+found folder, but quickly got a kernel crash "kernel: kernel BUG at drivers/md/unraid.c:443! ....", with etx2 it appeared to work all fine until I umount it then got same crash, then it is maybe something more complex than I though...