live4ever

Try installing google-auth-oauthlib: pip install google-auth-oauthlib

So after I upgraded my main uploading unRAID server from 6.9 series to 6.11 I noticed how permissions could mess up using the scripts from GitHub. My mount script is a bit different - I create a “backup” folder acting as sort of Dropbox for uploads to gdrive at: /mnt/user/local/gdrive_vfs/backup This folder would get deleted after rclone_upload completes and then get recreated with rclone_mount (with root:root). In DZMM rclone_mount the MountFolders bit: MountFolders=\{"downloads/complete,downloads/intermediate,downloads/seeds,movies,tv"\} # comma separated list of folders to create within the mount In DZMM rclone_upload: Command1="--exclude downloads/**" So the paths from MountFolders: /mnt/user/local/gdrive_vfs/movies /mnt/user/local/gdrive_vfs/tv will get deleted (after uploading) - then the rclone_mount script will create them again with root:root (and dockers trying to use them will give permission errors. Is there a way for the rclone_mount script to create the empty MountFolders with nobody:user permissions?

The above was on unRAID 6.10.3 - another server I have on 6.9.3 doesn't have "dig" (not in nerd tools either) so the GES script won't run (in case anyone tries on the 6.9.x release). *********************************************** ******** Google Endpoint Scanner (GES) ******** *********************************************** [GES] [INFO] The application started on Thu Jul 7 10:22:21 EDT 2022. [GES] [WARNING] Hosts backup file not found. Backing it up. [GES] [ERROR] The command 'dig' is not installed or not reachable in this user's $PATH. *********************************************** * Finished Google Endpoint Scanner (GES) * Message: Install dig or make sure its executable is reachable, then try again. *********************************************** Script Finished Jul 07, 2022 10:22.21

Some test from Canada this morning: ******** Google Endpoint Scanner (GES) ******** *********************************************** [GES] [INFO] The application started on Thu Jul 7 10:02:04 EDT 2022. [GES] [WARNING] Hosts backup file not found. Backing it up. [GES] [WARNING] Did not find a permanent blacklist at /root/blacklisted_google_ips. Creating a new one. [GES] [INFO] Please wait. Downloading the test file from 142.251.41.42... [GES] [INFO] Parsing connection with 142.251.41.42. [GES] [INFO] 1.538 MiB/s. Below criterion endpoint. Blacklisting IP '142.251.41.42'. [GES] [INFO] Please wait. Downloading the test file from 142.251.32.74... [GES] [INFO] Parsing connection with 142.251.32.74. [GES] [INFO] 8.334 MiB/s. Above criterion endpoint. Whitelisting IP '142.251.32.74'. [GES] [INFO] Please wait. Downloading the test file from 172.217.1.10... [GES] [INFO] Parsing connection with 172.217.1.10. [GES] [INFO] 1.564 MiB/s. Below criterion endpoint. Blacklisting IP '172.217.1.10'. [GES] [INFO] Please wait. Downloading the test file from 172.217.165.10... [GES] [INFO] Parsing connection with 172.217.165.10. [GES] [INFO] 1.545 MiB/s. Below criterion endpoint. Blacklisting IP '172.217.165.10'. [GES] [INFO] Please wait. Downloading the test file from 142.251.41.74... [GES] [INFO] Parsing connection with 142.251.41.74. [GES] [INFO] 10.32 MiB/s. Above criterion endpoint. Whitelisting IP '142.251.41.74'. [GES] [INFO] Please wait. Downloading the test file from 142.251.33.170... [GES] [INFO] Parsing connection with 142.251.33.170. [GES] [INFO] 1.566 MiB/s. Below criterion endpoint. Blacklisting IP '142.251.33.170'. [GES] [INFO] The fastest IP is 142.251.41.74. Putting into the hosts file. *********************************************** * Finished Google Endpoint Scanner (GES) * Message: Reached EOF without errors *********************************************** Script Finished Jul 07, 2022 10:04.31 Full logs for this script are available at /tmp/user.scripts/tmpScripts/Gdrive_api/log.txt Script Starting Jul 07, 2022 10:11.25 Full logs for this script are available at /tmp/user.scripts/tmpScripts/Gdrive_api/log.txt *********************************************** ******** Google Endpoint Scanner (GES) ******** *********************************************** [GES] [INFO] The application started on Thu Jul 7 10:11:25 EDT 2022. [GES] [INFO] Hosts backup file found. Restoring it. [GES] [INFO] Found permanent blacklist. Parsing it. [GES] [INFO] Please wait. Downloading the test file from 142.251.32.74... [GES] [INFO] Parsing connection with 142.251.32.74. [GES] [INFO] 33.19 MiB/s. Above criterion endpoint. Whitelisting IP '142.251.32.74'. [GES] [INFO] Please wait. Downloading the test file from 142.251.41.74... [GES] [INFO] Parsing connection with 142.251.41.74. [GES] [INFO] 39.40 MiB/s. Above criterion endpoint. Whitelisting IP '142.251.41.74'. [GES] [INFO] The fastest IP is 142.251.41.74. Putting into the hosts file. *********************************************** * Finished Google Endpoint Scanner (GES) * Message: Reached EOF without errors *********************************************** Script Finished Jul 07, 2022 10:11.39 So initially it was the 50MB file then the 200MB file (at 10:11.25) - been having some buffering the past couple weeks.

Scroll down to the bottom of page “On a different network” here: https://support.plex.tv/articles/200288586-installation/

I’ve got two unRAIDs: 6.9 b22 47d uptime 6.9 b35 33d uptime That have been still working fine here.

@M1kep_ Thanks, the command: root@Tower:~# find /mnt/user/appdata/other/rclone/remotes -name mount_running* /mnt/user/appdata/other/rclone/remotes/gdrive_media_vfs/mount_running find /mnt/user/appdata/other/rclone/remotes -name mount_running* -delete allowed the rclone_mount script to start and download/install the mergerfs software.

@DZMM After a server reboot I can’t seem to get the the rclone_mount script (same one in my last post) to work all I’m seeing is: ————- Script Starting Nov 14, 2020 23:51.50 Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_mount/log.txt 14.11.2020 23:51:50 INFO: Creating local folders. 14.11.2020 23:51:50 INFO: *** Starting mount of remote gdrive_media_vfs 14.11.2020 23:51:50 INFO: Checking if this script is already running. 14.11.2020 23:51:50 INFO: Exiting script as already running. Script Finished Nov 14, 2020 23:51.50 Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_mount/log.txt _______ Thanks

@DZMMI do it after the mount script is up and running - and I did another test last night created two directories with the following paths: /mnt/user/mount_mergerfs/gdrive_media_vfs/test/video.mkv /mnt/user/mount_mergerfs/gdrive_media_vfs/test2/ The directory ../test/video.mkv (with a file within it) was there in this morning while the ../test2/ directory (without any sub-folders or files) was deleted/removed. Not a big deal really though, was just a little concerned that (empty) folders I created in mergerfs were disappearing. Also, I noticed that the mount script created shares/folders only on /mnt/disk14 and /mnt/disk15 (the least full disks of my array) - can I manually create the: /mnt/local/gdrive_media_vfs share on disks1-13? Or how do I get the mergerfs share to appear on all disks (to prevent moving files between disks before upload)? Thanks again

@DZMM had to reboot unRAID server to get it to clear up - unmounting with fusermount wasn't enough. Quick question, does the mount script have to be used to create the subfolders within the mergerfs folder? For example my mount script does: MountFolders=\{"downloads/complete,downloads/incomplete,backup"\} and if I manually create a folder like /mnt/user/mount_mergerfs/gdrive_media_vfs/test/video.mkv the /test/video.mkv disappears (I guess when the mount script runs after the next 10min?) Thanks again

Using rclone 1.53.1 Here's the rclone_mount script: #!/bin/bash ###################### #### Mount Script #### ###################### ### Version 0.96.8 ### ###################### ####### EDIT ONLY THESE SETTINGS ####### # INSTRUCTIONS # 1. Change the name of the rclone remote and shares to match your setup # 2. NOTE: enter RcloneRemoteName WITHOUT ':' # 3. Optional: include custom command and bind mount settings # 4. Optional: include extra folders in mergerfs mount # REQUIRED SETTINGS RcloneRemoteName="gdrive_media_vfs" RcloneMountShare="/mnt/user/mount_rclone" LocalFilesShare="/mnt/user/local" MergerfsMountShare="/mnt/user/mount_mergerfs" DockerStart="radarr sonarr" MountFolders=\{"downloads/complete,downloads/incomplete,backup"\} # OPTIONAL SETTINGS # Add extra paths to mergerfs mount in addition to LocalFilesShare LocalFilesShare2="ignore" # without trailing slash e.g. /mnt/user/other__remote_mount/or_other_local_folder. Enter 'ignore' to disable LocalFilesShare3="" LocalFilesShare4="" LocalFilesShare5="" LocalFilesShare6="" # Add extra commands or filters Command1="--rc" Command2="" Command3="" Command4="" Command5="" Command6="" Command7="" Command8="" CreateBindMount="N" # Y/N. Choose whether to bind traffic to a particular network adapter RCloneMountIP="192.168.1.252" # My unraid IP is 172.30.12.2 so I create another similar IP address NetworkAdapter="eth0" # choose your network adapter. eth0 recommended VirtualIPNumber="2" # creates eth0:x e.g. eth0:1. I create a unique virtual IP addresses for each mount & upload so I can monitor and traffic shape for each of them ####### END SETTINGS ####### ############################################################################### ##### DO NOT EDIT ANYTHING BELOW UNLESS YOU KNOW WHAT YOU ARE DOING ####### ############################################################################### ####### Preparing mount location variables ####### LocalFilesLocation="$LocalFilesShare/$RcloneRemoteName" RcloneMountLocation="$RcloneMountShare/$RcloneRemoteName" MergerFSMountLocation="$MergerfsMountShare/$RcloneRemoteName" ####### create directories for rclone mount and mergerfs mounts ####### mkdir -p /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName #for script files mkdir -p /mnt/user0/mount_rclone/cache/$RcloneRemoteName #for cache files if [[ $LocalFileShare == 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Not creating local folders as requested." else echo "$(date "+%d.%m.%Y %T") INFO: Creating local folders." eval mkdir -p $LocalFilesLocation/"$MountFolders" fi mkdir -p $RcloneMountLocation mkdir -p $MergerFSMountLocation ####### Check if script is already running ####### echo "$(date "+%d.%m.%Y %T") INFO: *** Starting mount of remote ${RcloneRemoteName}" echo "$(date "+%d.%m.%Y %T") INFO: Checking if this script is already running." if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Exiting script as already running." exit else echo "$(date "+%d.%m.%Y %T") INFO: Script not running - proceeding." touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running fi ####### Create Rclone Mount ####### # Check If Rclone Mount Already Created if [[ -f "$RcloneMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Success ${RcloneRemoteName} remote is already mounted." else echo "$(date "+%d.%m.%Y %T") INFO: Mount not running. Will now mount ${RcloneRemoteName} remote." # Creating mountcheck file in case it doesn't already exist echo "$(date "+%d.%m.%Y %T") INFO: Recreating mountcheck file for ${RcloneRemoteName} remote." touch mountcheck rclone copy mountcheck $RcloneRemoteName: -vv --no-traverse # Check bind option if [[ $CreateBindMount == 'Y' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}" ping -q -c2 $RCloneMountIP > /dev/null # -q quiet, -c number of pings to perform if [ $? -eq 0 ]; then # ping returns exit status 0 if successful echo "$(date "+%d.%m.%Y %T") INFO: *** IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}" else echo "$(date "+%d.%m.%Y %T") INFO: *** Creating IP address ${RCloneMountIP} for remote ${RcloneRemoteName}" ip addr add $RCloneMountIP/24 dev $NetworkAdapter label $NetworkAdapter:$VirtualIPNumber fi echo "$(date "+%d.%m.%Y %T") INFO: *** Created bind mount ${RCloneMountIP} for remote ${RcloneRemoteName}" else RCloneMountIP="" echo "$(date "+%d.%m.%Y %T") INFO: *** Creating mount for remote ${RcloneRemoteName}" fi # create rclone mount rclone mount \ --allow-other \ --dir-cache-time 720h \ --log-level INFO \ --poll-interval 15s \ --cache-dir=/mnt/user0/mount_rclone/cache/$RcloneRemoteName \ --vfs-cache-mode full \ --vfs-cache-max-size 100G \ --vfs-cache-max-age 336h \ --bind=$RCloneMountIP \ $RcloneRemoteName: $RcloneMountLocation & # Check if Mount Successful echo "$(date "+%d.%m.%Y %T") INFO: sleeping for 5 seconds" # slight pause to give mount time to finalise sleep 10 echo "$(date "+%d.%m.%Y %T") INFO: continuing..." if [[ -f "$RcloneMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Successful mount of ${RcloneRemoteName} mount." else echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mount failed - please check for problems." docker stop $DockerStart find /mnt/user/appdata/other/rclone/remotes -name mount_running* -delete rm /mnt/user/appdata/other/scripts/running/fast_check exit fi fi ####### Start MergerFS Mount ####### if [[ $MergerfsMountShare == 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Not creating mergerfs mount as requested." else if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount in place." else # check if mergerfs already installed if [[ -f "/bin/mergerfs" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs already installed, proceeding to create mergerfs mount" else # Build mergerfs binary echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs not installed - installing now." mkdir -p /mnt/user/appdata/other/rclone/mergerfs docker run -v /mnt/user/appdata/other/rclone/mergerfs:/build --rm trapexit/mergerfs-static-build mv /mnt/user/appdata/other/rclone/mergerfs/mergerfs /bin # check if mergerfs install successful echo "$(date "+%d.%m.%Y %T") INFO: *sleeping for 5 seconds" sleep 10 if [[ -f "/bin/mergerfs" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs installed successfully, proceeding to create mergerfs mount." else echo "$(date "+%d.%m.%Y %T") ERROR: Mergerfs not installed successfully. Please check for errors. Exiting." docker stop $DockerStart find /mnt/user/appdata/other/rclone/remotes -name mount_running* -delete rm /mnt/user/appdata/other/scripts/running/fast_check exit fi fi # Create mergerfs mount echo "$(date "+%d.%m.%Y %T") INFO: Creating ${RcloneRemoteName} mergerfs mount." # Extra Mergerfs folders if [[ $LocalFilesShare2 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare2} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare2=":$LocalFilesShare2" else LocalFilesShare2="" fi if [[ $LocalFilesShare3 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare3} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare3=":$LocalFilesShare3" else LocalFilesShare3="" fi if [[ $LocalFilesShare4 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare4} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare4=":$LocalFilesShare4" else LocalFilesShare4="" fi if [[ $LocalFilesShare5 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare5} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare5=":$LocalFilesShare5" else LocalFilesShare5="" fi if [[ $LocalFilesShare6 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare6} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare6=":$LocalFilesShare6" else LocalFilesShare6="" fi # mergerfs mount command mergerfs $LocalFilesLocation:$RcloneMountLocation$LocalFilesShare2$LocalFilesShare3$LocalFilesShare4$LocalFilesShare5$LocalFilesShare6 $MergerFSMountLocation -o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=partial,dropcacheonclose=true # check if mergerfs mount successful echo "$(date "+%d.%m.%Y %T") INFO: Checking if ${RcloneRemoteName} mergerfs mount created." if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount created." else echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mergerfs mount failed." docker stop $DockerStart find /mnt/user/appdata/other/rclone/remotes -name mount_running* -delete rm /mnt/user/appdata/other/scripts/running/fast_check exit fi fi fi ####### Starting Dockers That Need Mergerfs Mount To Work Properly ####### echo "$(date "+%d.%m.%Y %T") INFO: Starting dockers." docker start $DockerStart rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running echo "$(date "+%d.%m.%Y %T") INFO: ${RcloneRemoteName} Script complete" exit Here is rclone.conf [gdrive] type = drive client_id = xxxx client_secret = xxxx scope = drive token = {"access_token":"xxxxx","token_type":"Bearer","refresh_token":"xxxx","expiry":"2020-10-23T11:50:00.72593565-04:00"} [gdrive_media_vfs] type = crypt remote = gdrive:crypt filename_encryption = standard directory_name_encryption = true password = xxxx password2 = xxxx

Just set this up using the new mount script (posted a few pages back by @DZMM) seems to working and uploading fine. Created two folders (via mount script backup and downloads) in gdrive_media_vfs crypt remote, and question is should I be seeing all these other folders in /mnt/user/mount_mergerfs/gdrive_media_vfs (attached images)? Thanks

I can't seem to get the ADDITIONAL_PORTS variable to work. Tried with Telly port 6077 and ChannelsDVR port 8089 Neither of these are accessible at unRAID.lan.ip:6077 or 8089

Any updates to this? Still getting: docker exec -it -u abc tvheadend /usr/bin/tv_grab_zz_sdjson_sqlite --configure Can't locate LWP/UserAgent/Determined.pm in @INC (you may need to install the LWP::UserAgent::Determined module) (@INC contains: /usr/local/lib/perl5/site_perl /usr/local/share/perl5/site_perl /usr/lib/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib/perl5/core_perl /usr/share/perl5/core_perl) at /usr/bin/tv_grab_zz_sdjson_sqlite line 4927. BEGIN failed--compilation aborted at /usr/bin/tv_grab_zz_sdjson_sqlite line 4927. Using the newest update from the weekend (4.2.7-34~g036b9cbab). Thanks again.

Is anyone using the sqlite SchedulesDirect grabber? Linux 4.14.49-unRAID. Last login: Mon Nov 5 07:26:05 -0500 2018 on /dev/pts/3. root@Tower:~# docker exec -it -u abc tvheadend /usr/bin/tv_grab_zz_sdjson_sqlite --configure Can't locate LWP/UserAgent/Determined.pm in @INC (you may need to install the LWP::UserAgent::Determined module) (@INC contains: /usr/local/lib/perl5/site_perl /usr/local/share/perl5/site_perl /usr/lib/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib/perl5/core_perl /usr/share/perl5/core_perl) at /usr/bin/tv_grab_zz_sdjson_sqlite line 4927. BEGIN failed--compilation aborted at /usr/bin/tv_grab_zz_sdjson_sqlite line 4927. Maybe the docker image doesn't have the perl bits in it? Thanks for all the ls-io dockers!