comet424

wish for a manual or automatic parity pause

20 posts in this topic Last Reply

Recommended Posts

wish for manual or automatic pause of parity checking when other tasks needs the disk capacity.

 

since  copying 2TB to the unraid..  unraid at same time is doing a PArity check with 12TB parity drive.. and trying to access Plex server  all at the same time

and plex stops working it breaks down  from this scenario

Share this post


Link to post

Dell's PERCs have a user-defined 'rebuild rate' (default of 30%), which as far as I understands is a QoS system for the I/O to the disks. I've no clue if something like this is possible with unRAID, but it would be functionally pretty similar to what you're suggesting: If other stuff is happening, prioritise that, rather than the parity check.

Share this post


Link to post

not sure what all that means what you said I just a simple person simple words lol..  just frustrating lol   

took 3 days 2 hours 45min to parity check 12TB drive  of 24TB of data  so had to wait

Share this post


Link to post
6 hours ago, comet424 said:

took 3 days 2 hours 45min to parity check 12TB drive

That's a long time for a 12TB parity, I have a server with 8TB drives and it takes 15 Hours.

Share this post


Link to post
Posted (edited)

im guessing because i have 2 2TB drives  1 6TB  1 1.5TB and 1 12TB data drives 

and 1 12TB parity drive

plus while it was doing parity   i transfered 2.7TB of data to the array

oh and it averaged parity speed was like 40mb/s it said

Edited by comet424

Share this post


Link to post
24 minutes ago, comet424 said:

while it was doing parity   i transfered 2.7TB of data to the array

 

This is the main factor for the slowdown - during this time, the disks had to seek like crazy between the parity scan position and the file transfer position.

 

And it was because of this that I suggested you should open a feature request thread - it would have taken way less time to pause the parity scan, perform the file copy, and then continue the parity scan. And with less disk wear.

 

The file /proc/diskstats shows how much work different disks does. So it would be quite easy for unRAID to look at this file and notice that it's meaningful to step down on the parity scan work.

Share this post


Link to post

well i kinda already did ask for a feature request.  at the top i asked for a automatic or pause feature of the parity scan.. 

is not what you mentioned the same..

 

ya i dont really check the server  i just copy to it and let it be   so i noticed when plex want working parity was like 26 done  and i was copying at that time 2.7tb of data to the system..

i still very new to unraid i used to windows

 

Share this post


Link to post
4 minutes ago, comet424 said:

i still very new to unraid i used to windows

 

Windows also suffers badly when doing multiple transfers using the same disk. Linux too. The OS doesn't have any optimization to prioritize one transfer before another. So it's mostly up to the end user.

 

In enterprise installations, it's often handled by tiered storage using cache drives where some storage manager software spreads the load over multiple cache drives. And it's up to the system designer to decide max burst capacity that the caching system should be able to handle, and design enough overcapacity of the main storage to be able to synchronize between cache and main storage while maintaining the required real-time requirements. This is also why many larger storage solutions skips disk arrays and instead uses large pools of disks and stores multiple copies of the file data spread over "random" disks. Then a load balancer can spread read/write jobs over the disk pool.

 

One note is that the native RAID support in Linux do support bandwidth-limitation of rebuilds.

Share this post


Link to post

ok some of that confused the heck out of me lol 

i have a 250gb cache drive i added to the unraid  i found if i copy the 2tbs of stuff it filles that cache  but it doesnt transfer right away to say the 12tb drive  and i had to manually transfer it.. so i guess doesnt work for big file copies like that

 

i do ultimately want to have say on my unraid media server here  if 3 or 4 people in the home accessing the server or watching the same show by odd chance i dont want no slow down..

do i need large pool and the file spread over random disks..

or would upgrading to like 10gb nic and 10gb switch fix that  or do you need this link agression switch with multiple nics

as id like only 1 copy of the file on the network but to speed up performance i plan on changing out like the 1.5tb drive with another 12TB drive  i using WD Gold 12TB

and  would have tried these SASs but i cant get them from canada computers.. and not many comps support it anyways

 

Share this post


Link to post
16 minutes ago, comet424 said:

i do ultimately want to have say on my unraid media server here  if 3 or 4 people in the home accessing the server or watching the same show by odd chance i dont want no slow down..

 

The disks have more bandwidth than what single views needs. And use of read-ahead (in the viewer and by Linux itself) means each viewer will normally read in enough data that the drive has ample time to perform a seek and move to a different file to stream out to another viewer before later return back and fetch more data for the first stream.

 

And if multiple people are viewing the same show, then unRAID will also make use of RAM to buffer data.

 

When parity is checked, RAM buffers doesn't matter because unRAID wants to walk through all data on the disks once. So every new read is for new data from the disks.

And same with your 2.7TB copy - it can't make use of any RAM buffers because there is constantly new data to write.

So this special situation ends up with the parity disk and the data disk you copy to constantly multiplexing between the two tasks. With infinite seek speeds, you would end up with around half the speed - 50% of transfer capacity consumed for the copy and 50% for the parity check (actual percentage depending on disk raw capacity and used network interface speeds).

 

But HDD disk seeks are very, very slow. So a majority of the time, the disks will not transfer any data because they are waiting for the disk heads to move to the new location. You get something called thrashing - you might end up with 20 MB/s total transfer rate and 80% of the time waiting for the disks to finish seek operations. That is why SSD manufacturers likes to present their 80000 OP/s figures that allows a SSD to access multiple files "concurrently" without the transfer rates free-falling.

Share this post


Link to post

ok i understand a little

but for my questions  if i wanna be able to access same file and not have a slow down what do i need to do with un raid would upgrading the nics help

i did notice  if i paused my copying  of the 2tb  plex worked  a bit  least it wasnt choppy quality  i guess i was maxing out the 1gbps nic

but  if its better for SSD

can plex or unraid  if a file is accssed  plex copys to the ssd  and then reads it that way

 

as they dont make 12TB SSDs yet

Share this post


Link to post

Best is if you start your copying before going to bed (but preferably avoiding a parity-check night).

 

And if you have a cache disk, and configure your shares to use the cache (option "Yes") then you can add reasonable amounts of new data to unRAID by writing to the cache drive, and have unRAID move them to the array next night.

 

Note that your current issue was a special case. You had two file operations that each wanted 100% disk transfer capacity that did run concurrently with you trying to use Plex.

Share this post


Link to post

ah ok...  

and what about accessing same file 4 times odds are it wont happen for plex  cant the hard drives handle that can the nic handle it  as i doubt it could handle and copy files over a single nic  without dropping quality

 

i did try the cache  i new to SSDs  i used to regular hard drives i bought my first SSD  a month ago  250 WD blue 3D nano  not sure what 3d nano means but has 5 yr warrenty

i found when i link all my shares  to the cache and was copying the 2 tb it filled that 250gb cache then  unraid complained it didnt automaticlly move it right away had to manually move it..  i figured it automaticlly move  when it gets close to full so there be no performance slow down  etc

Share this post


Link to post
2 minutes ago, comet424 said:

i figured it automaticlly move  when it gets close to full

It will move the files over on a schedule you set (Settings -> Scheduler).  You can have unRaid move files over when the cache drive gets to a certain fill level by installing the Mover Tuning plugin.

Share this post


Link to post

ok ill try that thanks

 

and is there anything i can do to make it so if multiple people accessign the unraid to watch tv shows or movies or the same movie there is now slow down in quality

i dont need it to stream online  just on the network 

via xbox  computer cell tv etc

 

Share this post


Link to post
51 minutes ago, comet424 said:

as i doubt it could handle and copy files over a single nic  without dropping quality

 

Normal 1080p quality video can be streamed over a 100Mbit/s network. And "standard" networking nowadays is 1gbit/s.

 

So the NIC shouldn't be the limiting factor. In most situations you should be fine to stream multiple playback streams at the same time,  unless Plex needs to transcode the data to a resolution and encoding standard that the different clients support. If transcoding is needed, then the limiting factor would normally be the available computational power since it takes a huge amount of computations to decode and encode video data.

 

In the end, I'd say just set cache option to "Yes" for your media shares and you are likely to be fine without need to do anything else.

Share this post


Link to post

oh and under cache  where do u set if it gets full it transfers  there is a Critical disk utilization threshold (%): set to 90  is that what it is

if so i guess wouldnt work for me since when i was copying it was 2TB and the cache was 250  so it be always full as its transfering right

Share this post


Link to post
8 minutes ago, comet424 said:

oh and under cache  where do u set if it gets full it transfers  there is a Critical disk utilization threshold (%): set to 90  is that what it is

if so i guess wouldnt work for me since when i was copying it was 2TB and the cache was 250  so it be always full as its transfering right

Once the CacheFloor setting is reached, all transfers will go to the array.  The file(s) on the cache drive won't get moved until the mover runs (Scheduler - Mover Settings).  The plugin previously mentioned you can schedule it to run far more often, and begin moving files once a % threshhold is reached.

Share this post


Link to post

ah yes i forgot i thought was part of the unraid program

thanks its installing now

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


Copyright © 2005-2018 Lime Technology, Inc.
unRAID® is a registered trademark of Lime Technology, Inc.