DZMM

Quick High Water Question

9 posts in this topic Last Reply

Recommended Posts

Let's say I have two 8TB disks, disk 1 is 90% full and the other is empty.  If I move and rename a file stored on disk 1 stored at /mnt/user/share1/file.txt to /mnt/user/share2/new name.txt, will it move it quickly on disk 1 between shares and ignore the high water rule, or will it move and copy (slower) to disk 2, obeying the high water rule?

 

I'm getting into bottlenecks with renaming and moving downloads, and I'm wondering if my setup where I keep imports, movie_adults, movie_kids, tv_adults etc on different disks is causing problems i.e.  a new TV show is currently going /mnt/disk1/imports/show1.mkv to /mnt/disk2/TV shows/show_renamed.mkv which takes time to copy across, whereas mnt/disk1/imports/show1.mkv to /mnt/disk1/TV shows/show_renamed.mkv should be faster if unraid always moves a file on the same disk even if another disk has more space.

 

Thanks

Share this post


Link to post
2 hours ago, DZMM said:

If I move and rename a file stored on disk 1 stored at /mnt/user/share1/file.txt to /mnt/user/share2/new name.txt, will it move it quickly on disk 1 between shares and ignore the high water rule, or will it move and copy (slower) to disk 2, obeying the high water rule?

Depends how you do it.  If you move it via the network, then it will obey high water.  If you do it via command line, mc, krusader etc then it won't

Share this post


Link to post

Thanks.  What about dockers?

 

I guess if a disk is selected to extract a file from nzbget based on HW, then when it's time to move and rename that file that disk will probably still win the HW selection a few mins later.

Share this post


Link to post

It'll honor it as you're going through using 2 different mount points in the container. /Downloads and /movies

Sent via Tapatalk because I'm either at work or enjoying the summer

Share this post


Link to post

Thanks.  I'm hoping the changes I've made to not exclude disks from shares, or at least matching them up, and also to turn hardlinks on so hopefully less copying, will improve my import speeds as my backlog is approaching a couple of days which is annoying when you know something is there, but plex can't see it yet.

Share this post


Link to post

This might help to understand when the allocation method and include/exclude settings come into play.

 

 

Share this post


Link to post

Thanks.  I think I've got my flows setup the way I want and optimised:

 

nzbget: 

  1. downloads to /mnt/user/downloads on disk 4 - /mnt/downloads/ set to disks 4 only
  2. extracts to /mnt/user/import on disks 1-7 based on High Water choosing a disk.  99% of the time this shouldn't be on disk 4 as this has the least amount of space, so extraction should be faster as not occurring on same disk (I'm tempted to do downloads to an UD to ensure this is always the case)
  3. sonarr/radarr/lidarr/lazylibrarian move files to respective media shares.  HW rules should mean this is on the same disk that was chosen for the import, so just a rename and not a copy so fast

Deluge:

  1. downloads direct to /mnt/user/import on disks 1-7
  2. sonarr/radarr/lidarr/lazylibrarian copys/hardlinks files to respective media shares.  When hardlinks works should be fast as no copy

The bit I'm not sure about yet, is for some of my files the media share is a unionfs mount e.g. for kids_movies it's 

 

/mnt/user/movies_kids_upload=RW:/mnt/disks/rclone_vfs/movies_kids_gd=RO /mnt/disks/unionfs_movies_kids
unionfs -o cow,allow_other,direct_io,auto_cache,sync_read 

so sonarr/radarr etc are moving the file to /mnt/disks/unionfs_movies_kids i.e. not a /mnt/user location, although in reality the file behind the scenes is being written to /mnt/user/movies_kids_upload (rclone move job then moves it to google drive mounted at /mnt/disks/rclone_vfs/......). 

 

Do the dockers/unraid know this and still follow the HW rule i.e. the files are just renamed rather than copied?  I'm assuming hardlinks won't work as the docker will think /mnt/disks/ is on a different filesystem?

Share this post


Link to post
Posted (edited)
12 minutes ago, DZMM said:

Do the dockers/unraid know this and still follow the HW rule i.e. the files are just renamed rather than copied?  I'm assuming hardlinks won't work as the docker will think /mnt/disks/ is on a different filesystem?

Rather than spend forever trying to understand your workflow and inferring the host/container mappings, here's the rules in a nutshell

 

If you passthrough to the containers 2 mount points, one for downloads, and one for the final destination then:

  • Highwater / include / exclude rules will be honored
  • Hardlinks between downloads and final destinations will not work (because the mountpoints between the source and destination are different)

If you passthrough to the containers a single mount point, and reference downloads and the final destination from it then, (assuming everything in the container winds up referencing /mnt/user/)

 

  • Highwater / include / exclude rules will not be honored (files on the final destination will wind up being on the same disk that the download happened on)
  • Hardlinks between downloads and final destinations will work

My thoughts on hardlinks between a rclone mount and /mnt/user would be that even if you passthrough /mnt mapped to /mnt then it will not work at all as ultimately /mnt/disks and /mnt/user are different filesystems.

Edited by Squid
  • Like 1

Share this post


Link to post

Thanks that helped.  tweaking slightly to get desired result of nzb extraction onto different disk to the acutal download then a hardlink to the media folder, and torrents hardlinking for all, except for /mnt/disk rclone moves:

 

nzbget: /downloads mount point --> /mnt/user/import ---> /mnt/user/media

deluge:  /mnt/user/import ---> /mnt/user/media

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


Copyright © 2005-2018 Lime Technology, Inc.
unRAID® is a registered trademark of Lime Technology, Inc.