built-in automatic server replication

i have searched this topic but haven't found anything.

i cannot believe this appears to never have been requested, i must need new glasses...

anywho, just in case i am the first one to ask for this:

 

whilst i understand one should use his/her unraid server as much as possible and not be wasting cpu cycles (hence the vm technology that has been baked in), to me, unraid is still an expandable & fault-tolerant storage  technology first and foremost. 

 

stable dual-parity makes for a more robust unraid server (and this is much appreciated, thanks Tom!), but the ultimate in fault-tolerance is still redundancy and more specifically *server redundancy*.  now, one can already build a second unraid box and set up some scripts to copy over data to the secondary server, but i think we could do better.

 

it would be nice if there was some built-in functionality to automate data replication between servers, between a "master" and a "slave". that functionality would be managed through the gui and in fact could involve more than a single pair of servers -- i would say in my post-holiday dazed state that this could be considered clustering.  "cluster" members could be local (same rack) or remote (different building).  local data replication could be considered by some as a rather simple problem to tackle, but remote replication might be something different due to available bandwidth and so on.

 

curious to know what others think of this idea.

 

cheers -- and a happy 2017 to everyone.

 

I think a lot of us would agree with your points.  And no, you're not the first, but just guessing, perhaps the third, which is not enough to get it going.  There are some doing it I know, my estimate would be about a dozen or less, and I think you could find discussions of the best methods somewhere, mostly based on rsync daemons I believe.

+1

 

I do all my replication manually.... it's less fun than it sounds.

There is a place for complete replication, and by extension, high availability. I'm not sure unRaid is the best place to start for that though. Replication implies that deletions and corruptions would flow through to the second server, which isn't the best use of resources for most environments where unraid is currently used.

 

I would much rather see a built in limetech maintained system for automatic versioning backups, with some level of isolation so it takes some effort to make changes to saved versions. Something where you specify a second unraid box IP address and desired max network bandwidth usage, and the two servers communicate with each other about available and needed space, time to complete current backup task, and options for restoring content to the original box from the backup set.

 

There are so many providers in the "cloud" backup space, it would be nice to have our own privately managed unraid "cloud" designed specifically for our servers. I know there are plenty of roll your own options out there, but I'd really like to see limetech step up and offer a supported option built in. Backup is an integral part of being a NAS, imho.

unRaid Time Machine. They'd sell more licenses. It's a win/win.

to sort of quote someone, i thought of "one more thing":

 

replication could be complete or partial.  this because we can't always build two servers with the exact same configuration (number of disks, etc.).

i know the redundant/backup server i would like to build as soon as i can would be re-using many parts of my previous unraid build. obvious consequence: less drive space than the current/main server.

 

i also know some of my network shares are definitely preferable to others.

i'd prefer to replicate/backup a share containing my cv, personal photos and whatnot, instead of, say, the one containing the "reality" tv shows recorded for my gf (the things we do for love...).

 

btw, jonathanm brought up the need for versioning (à la vms, i presume?) to deal with the case of finger/mouse slip.  that would be a good optional feature for replication/backup.

 

cheers.

 

btw, jonathanm brought up the need for versioning (à la vms, i presume?) to deal with the case of finger/mouse slip.  that would be a good optional feature for replication/backup.

 

Currently, versioning has become FAR more important than just a recovery for mistyping.  Ransomware attacks have become (in my opinion) the number one threat, and the old ways of doing backups aren't good enough, if they just replicate files down the line.  Even scheduled nightly backups aren't safe, if they give you roughly only half a day to detect and react.  You the professional aren't the biggest problem, it's the ignorant and foolish user on your network that can expose the entire network to a very sophisticated attack.  And your unversioned backups won't help you, may actually make things worse.  The attack may not be able to reach your inaccessible backup locations, but that's OK, your backup tools can do the work for the attacker, replicating the damage to the otherwise inaccessible locations!

 

Your backups don't have to use a traditional versioning scheme, but they have to use some sort of grandfathering, that produces and keeps inaccessible backups from previous periods.

 

It's my opinion that many current computer professionals are under-estimating this threat.  The old ways HAVE to change.  And it's been proven that the new phishing methods are capable of fooling ANYONE, as they use authentic graphics and dialogs, and can act as a middle-man intercepting and passing along all of the authentication dialogs, including multi-factor authentications.

I agree that many computer professionals may be underestimating the threat of ransomware, I would say that by now, its the ones that are either living under rocks or in denial as ransomware has gotten so much media attention its difficult to ignore.

 

When it comes to security, one can not rely on one tool to save the day; backups, no matter how good they are, aren't going to be any good if they aren't protected and deployed properly. If you follow the 3-2-1 rule, and this is an 'old way' you will be protected. If you go a step further and keep a version of your backups offline, you will be protected.  However, as I've said, relying on just these methods to protect your network and your backups is not enough, one must practice defence in depth and use many other  methods for protection. Issues are exacerbated by the fact that some software requires ridiculous user privilege and wide open sharing for which there is no solution but to deal with it best you can and educate your users. The more educated and informed users are of the potential threats, the better it is for everyone.

so the point made in the last two posts (re. ransomware attacks) not only put the emphasis on versioning, but also on *version protection*.  finding a way to ensure that older versions of files are left untouched is a good idea.  i too wonder sometimes if/when i'll fall victim to a reprobate's actions, considering i was once the victim of a "drive-by download" a few years back, despite my precautions.  if my network shares were to get mucked up by ransomware, if my server were somehow infected (i wonder how, but anything network has to be considered vulnerable to malice nowadays), i would definitely not be happy.

 

/* not-fully thought-out brainstorming below */

 

i'd wonder if a mechanism to detect ransomware actions (i.e., sequential, mass-encryption of files & directories) could be implemented?

and to resist ransomware attacks, could file ops be implemented in an oracle-like fashion, i.e., "<command>, <command>, <command>, commit", the 'commit' command be issued by unraid (transparently/hidden from the user) once a certain volume of ops be reached and no suspicious pattern detected?

 

/* end of possible brain-fart */

 

*** so now the feature-request becomes "built-in & automated server or share replication & versioning", the optional versioning being built with a level of protection of prior versions. 

 

i would also refine & add to my previous ideas of being able to specify which share to replicate, and for "future unraid" to be able to have multiple replication servers: to be able to specify specific network share ---> replication server associations.  some share could be replicated locally, others could be replicated remotely. 

 

or am i going overboard with this?

 

i'd wonder if a mechanism to detect ransomware actions (i.e., sequential, mass-encryption of files & directories) could be implemented?

https://lime-technology.com/forum/index.php?topic=52462.0

and to resist ransomware attacks, could file ops be implemented in an oracle-like fashion, i.e., "<command>, <command>, <command>, commit", the 'commit' command be issued by unraid (transparently/hidden from the user) once a certain volume of ops be reached and no suspicious pattern detected?

 

That's an interesting idea I haven't heard before.

 

Two problems I can see, first it would have to be at a very low level, at the kernel level (requires kernel programmers) or at the file system level or just above it, intercepting all file I/O through that file system.  That requires some significant low level programming, and thorough testing before release.  The other problem is that it will clearly impact performance, because it requires delays on the first I/O's of every group of I/O's.  I'm not sure the commits are needed, just hold enough I/O's to monitor for given patterns, then release.

An implicit/"under the hood" commit command to release i/o ops after pattern analysis was just one way to express the idea of not committing all i/o to disc as they come in.  i agree that such an approach (delay actual writes until deemed safe) would need to be low-level enough to be transparent to the user. one has to ask: is this an unraid or a filesystem function?

 

anyway, that's just a side consideration.  whilst ransomware attacks are not a subject to dismiss lightly, protecting data from hardware failure is still the main reason behind the use of unraid for many of us.  hence my initial request for built-in replication/back-up between unraid servers.  and i can see the need for protection from finger/mouse slips, provided by versioning, which could greatly augment the value of data replication between servers.  i'm not dismissing resilience against ransomware attacks, i think such a function builds on previously developed features, and that limetech should make """clustering""" a higher priority.

 

i also have a gut feeling that built-in automated replication/backups coupled with versioning could help sell unraid in enterprise environments.  how many times have i heard over the years upper management dismiss gnu-linux or any other floss as toys that would never be deployed in a corporate environment... only to see it nowadays replacing solaris, hp/ux, tru64, vms, etc. wholesale?  maybe we'll end up seeing unraid have a significant presence in data centres?

 

maybe we'll end up seeing unraid have a significant presence in data centres?

Would be nice, but with the basic premise of unRaid handling drive recovery the way that it does (more akin to RAID4), there's a huge learning curve for old-school IT guys to think of it as anymore than just a toy.

there's a huge learning curve for old-school IT guys to think of it as anymore than just a toy.

 

i *am* an old-school it guy (trained on punched cards, we still had a pdp-8 with a tty in the corner of the terminal/keypunch room, know the joys of getting a batch job rejected because of a jcl error, etc.), just not in management. in this era of shrinking budgets and gross mismanagement, even i figured out that unraid could become a very capable, low-cost (due to be able to use/re-use generic hardware) solution for enterprise network storage.

 

i kept mentioning to upper management to keep an eye on the various freenices (*bsd, gnu-linux, etc.) over the years, as they could become useful and possibly could be used to replace more expensive solution.  the suits kept telling me those were toys and they would never use that.

 

guess what's been happening for the last 4-5 years? they have been phasing out hp/ux, tru64, solaris, even vms, in favour or red hat entreprise linux.

 

anything can happen, as long as unraid keeps evolving.

 

anything can happen, as long as unraid keeps evolving.

I agree.  Was mainly referring to unRaid's concept of no striping and dedicated parity. 

 

And as far as virtualization in an enterprise environment, I don't foresee IT dropping ESXi in favour of unRaid.

 

For the masses however, unRaid has no equal....

 

i don't think either that unraid would replace vmware as a virtualization platform, but with clustering-like features mentioned earlier in this thread, it could become a low-cost, highly flexible + reliable form of entreprise network storage.  like for network shares for corporate desktops. your average cubicule dweller doesn't need super-high throughput to save his/her latest spreadsheet, for example.

 

*edit: forgot the word "entreprise".

I'd like to add one more thing to multi-verion backup/replication ... make it multi-threaded to increase performance. I absolutely hate being bound to one cpu's power for backups. It turns what could be a 5-6 hour initial backup into a day and a half.... costing more overall energy in the long run.