File system corruption

I truly feel bad asking this, as I'm happy about the progress being made with unRAID 6 beta...

Aren't the users now being exposed to far more possible failures due to new file system options?  Doesn't the inclusion of new file systems increase the development time (x2 as far as I can tell)?

If the intention is to move away from reiserfs going forward I can understand the changes, if it isn't, I don't understand how embracing new file systems is going to make the support situation any better.

It's entirely possible (and likely) that I'm wrong, but I'd like to know.

I truly feel bad asking this, as I'm happy about the progress being made with unRAID 6 beta...

 

Aren't the users now being exposed to far more possible failures due to new file system options?  Doesn't the inclusion of new file systems increase the development time (x2 as far as I can tell)?

 

If the intention is to move away from reiserfs going forward I can understand the changes, if it isn't, I don't understand how embracing new file systems is going to make the support situation any better.

 

It's entirely possible (and likely) that I'm wrong, but I'd like to know.

There have been many posts by Limetech and other forum members about this already!

• Limetech have stated that their intention IS to proactively move away from reiserfs as it has a maximum file system size of 16TB and disks of this size do not look to be that far away now.  The other file systems have far higher limits.  XFS looks like becoming the new default for unRAID in the short term as it is very mature and has good recovery tools.  Several major Linux distributions have now adopted XFS as their default format.
  
• Reiserfs is not really being maintained any more at the Linux level.  This introduces risks whenLimeTech want to to move to new Linux kernel versions.  The other file systems are far more mainstream and thus well tested) at the generic Linux level and issues are thus less likely to exist in the kernel versions used by unRAID.
  
• BTRFS (while still deemed experimental) is making its way as an option into the mainstream Linux distributions and is under active development.  It has advanced features that will protect against items like file corruption.  It is also particularly suitable to supporting virtualisation because of its Copy-on-write (COW) features.  A number of Linux distributions have already indicated that as it matures this may well become their default file system for these reasons.
  
• (My personal opinion) Once support for multiple file-systems has been integrated into unRAID then I suspect there is little additional development effort involved in adding new ones.  There will be additional testing effort, but even this can be minimized I expect as one can leverage what the mainstream Linux distributions are doing from a testing perspective.  Support for other file systems can be seen as very desirable from both a technical and marketing perspective.  I notice that a roadmap item in this space is support for NTFS and HFS+ which will be very attractive to Windows and Mac users who would like the ability to take an unRAID disk and add it to their PC/Mac and be able to read it natively without special drivers.
  

I might add that if we have to move to another filesystem a filesystem converter sure would come in handy. Offloading a disk, formatting, loading data back, seems too tedious and error-prone.

I might add that if we have to move to another filesystem a filesystem converter sure would come in handy. Offloading a disk, formatting, loading data back, seems too tedious and error-prone.

Although it might be a convenient tool to have I am not sure I would want to trust such a tool.

Since unRAID is not discontinuing Reiserfs and is allowing a mixture of file systems in the same array, I suspect that the majority of people will do the migration piece-meal and combine switching file systems with going to larger disks.  In such a case you would probably take the original disk out of the array (thus keeping its data intact), so the copy method would not be too onerous and would be far safer.

I truly feel bad asking this, as I'm happy about the progress being made with unRAID 6 beta...

 

Aren't the users now being exposed to far more possible failures due to new file system options?  Doesn't the inclusion of new file systems increase the development time (x2 as far as I can tell)?

 

If the intention is to move away from reiserfs going forward I can understand the changes, if it isn't, I don't understand how embracing new file systems is going to make the support situation any better.

 

It's entirely possible (and likely) that I'm wrong, but I'd like to know.

There have been many posts by Limetech and other forum members about this already!

• Limetech have stated that their intention IS to proactively move away from reiserfs as it has a maximum file system size of 16TB and disks of this size do not look to be that far away now.  The other file systems have far higher limits.  XFS looks like becoming the new default for unRAID in the short term as it is very mature and has good recovery tools.  Several major Linux distributions have now adopted XFS as their default format.
  
• Reiserfs is not really being maintained any more at the Linux level.  This introduces risks whenLimeTech want to to move to new Linux kernel versions.  The other file systems are far more mainstream and thus well tested) at the generic Linux level and issues are thus less likely to exist in the kernel versions used by unRAID.
  
• BTRFS (while still deemed experimental) is making its way as an option into the mainstream Linux distributions and is under active development.  It has advanced features that will protect against items like file corruption.  It is also particularly suitable to supporting virtualisation because of its Copy-on-write (COW) features.  A number of Linux distributions have already indicated that as it matures this may well become their default file system for these reasons.
  
• (My personal opinion) Once support for multiple file-systems has been integrated into unRAID then I suspect there is little additional development effort involved in adding new ones.  There will be additional testing effort, but even this can be minimized I expect as one can leverage what the mainstream Linux distributions are doing from a testing perspective.  Support for other file systems can be seen as very desirable from both a technical and marketing perspective.  I notice that a roadmap item in this space is support for NTFS and HFS+ which will be very attractive to Windows and Mac users who would like the ability to take an unRAID disk and add it to their PC/Mac and be able to read it natively without special drivers.
  

Thanks for taking the time itimpi, truly appreciated.  I try to keep up with the forum, but things slip by. 

I think a convertor is a nice idea in theory but a very bad one in practice.

It is not impossible to imagine that when a conversion goes wrong you end up in a data loss state (potentially unrecoverable). Also compare the time put into debugging and coding moving files from A to B. It surely has to be one of the most reliable operations ever. Could we ever say this about a convertor?

I wouldn't trust one at all ever.

Better would be tools to run like the mover script that gradually migrates and tests as it goes whilst maintianing parity the whole time

I think a convertor is a nice idea in theory but a very bad one in practice.

 

It is not impossible to imagine that when a conversion goes wrong you end up in a data loss state (potentially unrecoverable). Also compare the time put into debugging and coding moving files from A to B. It surely has to be one of the most reliable operations ever. Could we ever say this about a convertor?

 

I wouldn't trust one at all ever.

 

Better would be tools to run like the mover script that gradually migrates and tests as it goes whilst maintianing parity the whole time

+1 .. One of the things I miss in unraid is drive balancing tools... At some point in time you decide you want a different drive setup and/or want to move some data around.. rebalance..

At the moment there is no such tool though it would be a very simple one (I THINK... I could not do it myself) to build..

Think of options like:

- reballance data.. Make sure drives are equally full (percentwise of in hard mbs)

- empty drive.. Actively move data around to empty a drive (and then possible take it out of the array to replace it)

They are basically copy and move commands so should not be that hard to implement..

 

- reballance data.. Make sure drives are equally full (percentwise of in hard mbs)

 

This is actually "usually" a very bad idea.. or at least not a very good one. Most people use unRAID for media and most of them have videos. In this example you are ballancing the data onto differernt drives (which doesn't really achieve anything beneficial) but you in up in a situation that if you sit down to watch a tv series you could have every ep on a seperate drive. Imagine that with music.

In an ideal world you want sets of data pooled onto the same drive and then it get difficult as it requires the tool to know what defines a set.

Not having a go as this idea comes up every now and again and whilst its nice for OCD its not actually very beneficial.

 

 

- empty drive.. Actively move data around to empty a drive (and then possible take it out of the array to replace it)

 

This is possibly my #1 missing feature with unRAID. You should be able to do far more with adding and replaceing drives without spinning all disks or blasting away your parity... so here here:)

Most people use unRAID for media and most of them have videos. In this example you are ballancing the data onto differernt drives (which doesn't really achieve anything beneficial) but you in up in a situation that if you sit down to watch a tv series you could have every ep on a seperate drive. Imagine that with music.

Also, data in a media store tends to be very static - write once, read many - when a device fills, just stop writing to it.

Nas:

There are circumstances where I use it. For example at some point I figured it would be a good idea to have most of me frequently accessed data on one disk to make sure only one spins up most of the time.

At some point I also figured that it would be good to have disks with only movies and another set with only series; this because series need spare room, since they grow and movies do not, a movies disk can be completely full..

At some point I even moved old series to a specific disk set, because old series also do not grow..

All these settings can be set using unraid options but there is no way to enforce these options when you change them.

So there is enough reason to need this in my use case..

Nas:

 

There are circumstances where I use it. For example at some point I figured it would be a good idea to have most of me frequently accessed data on one disk to make sure only one spins up most of the time.

 

At some point I also figured that it would be good to have disks with only movies and another set with only series; this because series need spare room, since they grow and movies do not, a movies disk can be completely full..

 

At some point I even moved old series to a specific disk set, because old series also do not grow..

 

All these settings can be set using unraid options but there is no way to enforce these options when you change them.

 

So there is enough reason to need this in my use case..

Yeha your right. I have mentioned a few times before that this is an area that we could improve on and consequently give users a better experience.

A couple of years back i wrote a crap (but useful) script called undist that would tell you whats disk shares a folder was on. I use this to balance like with like usually at the time when I add system capacity i.e. add a  new disk.

The meta effect (much as you have written) is a cleaner usage experience.

The problem is it is very hard to teach a system what "files" should be "clumped".

I suggest we take this to a new thread as its damn interesting but OT

Since this is more of a Q&A discussion about file system corruption, then a request for a new road mapped feature, I'm going to move it to the general support area.

When the information can be formulated into a concise feature request, repost with pointers to relevant discussion.