Redxd

Members
  • Posts

    36
  • Joined

  • Last visited

Converted

  • Gender
    Undisclosed

Redxd's Achievements

Noob

Noob (1/14)

0

Reputation

  1. I'd also been keen to see the imfile module included in the default OS so I can onforward logs like /var/log/ipmifan which generates temp readings from my motherboards IPMI and adjust the fans accordingly.
  2. With Youtube URL's now changed, has anyone figured out how to download a full channel when 'Usernames' are no longer being accepted? {"level":"error","msg":"There was an error downloading the video: ERROR: u'' is not a valid URL. Set --default-search \"ytsearch\" (or run youtube-dl \"ytsearch:\" ) to search YouTube\n[youtube:user] destinws2: Downloading channel page\n[youtube:playlist] UU6107grRI4m0o2-emgoDnAA: Downloading webpage\n[download] Downloading playlist: Uploads from SmarterEveryDay\n[youtube:playlist] UU6107grRI4m0o2-emgoDnAA: Downloading page #1\nERROR: u'content_html'\n","time":"2020-07-27T13:50:46Z"} {"level":"info","msg":"returning response: {Error ERROR_DOWNLOADING There was an error downloading the target Download: DownloadVideo: ERROR: u'' is not a valid URL. Set --default-search \"ytsearch\" (or run youtube-dl \"ytsearch:\" ) to search YouTube\n[youtube:user] destinws2: Downloading channel page\n[youtube:playlist] UU6107grRI4m0o2-emgoDnAA: Downloading webpage\n[download] Downloading playlist: Uploads from SmarterEveryDay\n[youtube:playlist] UU6107grRI4m0o2-emgoDnAA: Downloading page #1\nERROR: u'content_html'\n}","time":"2020-07-27T13:50:46Z"}
  3. I second this request. I have an X11 that has FAN1-7 and also FANAB
  4. I've just updated the docker as well and i'm getting the same error message. Docker doesn't even start. usermod: no changes groupmod: GID '100' already exists {"level":"info","msg":"Initiating db","time":"2020-01-14T07:23:08Z"} {"level":"error","msg":"database.load: could not read file: json: cannot unmarshal string into Go struct field DownloadTarget.CheckingInterval of type int","time":"2020-01-14T07:23:08Z"} {"level":"fatal","msg":"\u003cnil\u003e","time":"2020-01-14T07:23:08Z"} I also removed the image completely and re-downloaded it just in case, still same issue. I checked the permissions on the json config files and they are exactly the same as what's set in the docker config.
  5. I've just got back from holidays and the docker is working pretty good now. One thing I have started to notice is that every once and a while (maybe when it does it's check for new videos) the docker will consume 100% CPU on my server (As shown on the dashboard) and slow down everything else. This has happened before though while pre-clearing disks and it wasn't the CPU that was maxing it, it was the IO wait time increasing. It goes straight back to normal when i kill the auto-yt docker. What information can I provide to help diagnose it?
  6. I just tried manually within the docker using the same command but changing the 'bestvideo[ext=mp4]' to just 'best'. This downloaded normal like any other time but it included video. I think when choosing the 'bestvideo' option it's just doing video and it may also be missing the audio part and then needing to join the two?
  7. I'm able to get the whole channel to download now by choose the quality as MP4, it doesn't work when set to 'Any'. With MP4 selected the videos download in the background but my guess is the 'channel' dosn't get added to the bottom of the gui until all videos are downloaded. I have noticed that all the videos it does download don't have any audio. Nothing in the logs about it. {"level":"info","msg":"received a request to add a target","time":"2019-12-20T11:02:27Z"} {"level":"info","msg":"executing youtube-dl command: /usr/local/bin/youtube-dl -j --playlist-end 1 https://www.youtube.com/user/destinws2","time":"2019-12-20T11:02:27Z"} {"level":"info","msg":"DOWNLOAD: {https://www.youtube.com/user/destinws2 Video And Audio Uploads from SmarterEveryDay mp4 [] 12-20-2019 11:02:35 Channel %(uploader)s [%(channel_id)s]/%(upload_date)s %(title)s_[%(id)s].%(ext)s}","time":"2019-12-20T11:02:35Z"} {"level":"info","msg":"downloading video","time":"2019-12-20T11:02:35Z"} {"level":"info","msg":"{ youtube-dl --ignore-errors bestvideo[ext=mp4] downloads/%(uploader)s [%(channel_id)s]/%(upload_date)s %(title)s_[%(id)s].%(ext)s https://www.youtube.com/user/destinws2}","time":"2019-12-20T11:02:35Z"} {"level":"info","msg":"executing youtube-dl command: /usr/local/bin/youtube-dl --ignore-errors -f bestvideo[ext=mp4] -o downloads/%(uploader)s [%(channel_id)s]/%(upload_date)s %(title)s_[%(id)s].%(ext)s https://www.youtube.com/user/destinws2","time":"2019-12-20T11:02:35Z"} {"level":"info","msg":"DOWNLOAD: {https://www.youtube.com/user/destinws2 Video And Audio Uploads from SmarterEveryDay mp4 [] 12-20-2019 11:02:35 Channel %(uploader)s [%(channel_id)s]/%(upload_date)s %(title)s_[%(id)s].%(ext)s}","time":"2019-12-20T11:02:35Z"} {"level":"info","msg":"downloading video","time":"2019-12-20T11:02:35Z"} {"level":"info","msg":"{ youtube-dl --ignore-errors bestvideo[ext=mp4] downloads/%(uploader)s [%(channel_id)s]/%(upload_date)s %(title)s_[%(id)s].%(ext)s https://www.youtube.com/user/destinws2}","time":"2019-12-20T11:02:35Z"} {"level":"info","msg":"executing youtube-dl command: /usr/local/bin/youtube-dl --ignore-errors -f bestvideo[ext=mp4] -o downloads/%(uploader)s [%(channel_id)s]/%(upload_date)s %(title)s_[%(id)s].%(ext)s https://www.youtube.com/user/destinws2","time":"2019-12-20T11:02:35Z"}
  8. I have tried a few different custom paths in troubleshooting this. With a /channels/ folder (created and not created) No channels folder so just / /Downloads/ but then in the logs it must already prefix Downloads show it was showing up as downloads/downloads/
  9. i've tried multiple scenarios with the custom output path. Even the default it dosn't work.
  10. This is the docker settings I have set. Downloads location pointing to my array and config file pointing to my cache directory.
  11. No matter what i set the download folder path to be, or what customised path I set, I'm always getting the same error messages in the logs. I've tried a two different youtube channels but get the same error. {"level":"info","msg":"received a request to add a target","time":"2019-12-14T08:09:58Z"} {"level":"info","msg":"executing youtube-dl command: /usr/local/bin/youtube-dl -j --playlist-end 1 https://youtube.com/users/destinws2","time":"2019-12-14T08:09:58Z"} {"level":"error","msg":"From GetMetadata(): unexpected end of JSON input","time":"2019-12-14T08:10:02Z"} {"level":"info","msg":"adding {https://youtube.com/users/destinws2 Video And Audio any [] 12-14-2019 08:10:02 Channel /%(uploader)s_[UC2LVhJH_9cT2XKp0VAfsKOQ]/%(title)s_[%(id)s].%(ext)s%(uploader)s/video/%(title)s.%(ext)s} to DB","time":"2019-12-14T08:10:02Z"} {"level":"info","msg":"DOWNLOAD: {https://youtube.com/users/destinws2 Video And Audio any [] 12-14-2019 08:10:02 Channel /%(uploader)s_[UC2LVhJH_9cT2XKp0VAfsKOQ]/%(title)s_[%(id)s].%(ext)s%(uploader)s/video/%(title)s.%(ext)s}","time":"2019-12-14T08:10:02Z"} {"level":"info","msg":"downloading video","time":"2019-12-14T08:10:02Z"} {"level":"info","msg":"{ youtube-dl --ignore-errors bestvideo[ext=any] downloads/%(uploader)s_[UC2LVhJH_9cT2XKp0VAfsKOQ]/%(title)s_[%(id)s].%(ext)s%(uploader)s/video/%(title)s.%(ext)s https://youtube.com/users/destinws2}","time":"2019-12-14T08:10:02Z"} {"level":"info","msg":"executing youtube-dl command: /usr/local/bin/youtube-dl --ignore-errors -f bestvideo[ext=any] -o downloads/%(uploader)s_[UC2LVhJH_9cT2XKp0VAfsKOQ]/%(title)s_[%(id)s].%(ext)s%(uploader)s/video/%(title)s.%(ext)s https://youtube.com/users/destinws2","time":"2019-12-14T08:10:02Z"} {"level":"error","msg":"exit status 1/usr/local/bin/youtube-dl --ignore-errors -f bestvideo[ext=any] -o downloads/%(uploader)s_[UC2LVhJH_9cT2XKp0VAfsKOQ]/%(title)s_[%(id)s].%(ext)s%(uploader)s/video/%(title)s.%(ext)s https://youtube.com/users/destinws2","time":"2019-12-14T08:10:05Z"} {"level":"info","msg":"updating latest downloaded video id","time":"2019-12-14T08:10:05Z"} {"level":"info","msg":"latest downloaded video id updated successfully","time":"2019-12-14T08:10:05Z"} {"level":"info","msg":"updating download history","time":"2019-12-14T08:10:05Z"} {"level":"info","msg":"[{https://youtube.com/users/destinws2 Video And Audio any [] 12-14-2019 08:10:02 Channel /%(uploader)s_[UC2LVhJH_9cT2XKp0VAfsKOQ]/%(title)s_[%(id)s].%(ext)s%(uploader)s/video/%(title)s.%(ext)s}]","time":"2019-12-14T08:10:05Z"} {"level":"info","msg":"returning response: {Error ERROR_DOWNLOADING There was an error downloading the target Download: DownloadVideo: exit status 1}","time":"2019-12-14T08:10:05Z"} {"level":"info","msg":"got this data: {channels}","time":"2019-12-14T08:10:06Z"} {"level":"info","msg":"getting all channels from channels.json","time":"2019-12-14T08:10:06Z"} {"level":"info","msg":"successfully read all channels","time":"2019-12-14T08:10:06Z"} {"level":"info","msg":"[{https://youtube.com/users/destinws2 Video And Audio any [] 12-14-2019 08:10:02 Channel /%(uploader)s_[UC2LVhJH_9cT2XKp0VAfsKOQ]/%(title)s_[%(id)s].%(ext)s%(uploader)s/video/%(title)s.%(ext)s}]","time":"2019-12-14T08:10:06Z"} {"level":"info","msg":"got this data: {channels}","time":"2019-12-14T08:10:43Z"} {"level":"info","msg":"getting all channels from channels.json","time":"2019-12-14T08:10:43Z"} {"level":"info","msg":"successfully read all channels","time":"2019-12-14T08:10:43Z"} {"level":"info","msg":"[{https://youtube.com/users/destinws2 Video And Audio any [] 12-14-2019 08:10:02 Channel /%(uploader)s_[UC2LVhJH_9cT2XKp0VAfsKOQ]/%(title)s_[%(id)s].%(ext)s%(uploader)s/video/%(title)s.%(ext)s}]","time":"2019-12-14T08:10:43Z"}
  12. For this drive and the other 12TB that successfully worked I was seeing easy 120MB/s. The drives were plugged into the same SAS card as all the other drives in the array. all the speeds I was seeing both read and write seems normal. It's just this one drive grinded everything to a halt once writing zeros. I've currently got it booted off a trial unraid instance on my main PC doing its preclear.
  13. See attached tower-diagnostics-20191201-1919.zip
  14. I just shucked two 12TB WD White's and only one seems to have preclear'd without any issues. I've tried the old Preclear plugin, then the preclear docker when I saw notes that the plugin wasn't working well with 6.7 and then the most up to date plugin once updates were made for the MBR issues others were receiving. This one 12TB drive keeps on slowing down my unraid to a crawl. Hogs all I/O. When using the docker if left on for a while it takes minutes to run the docker stop command. The pre-read seems to run fine but it's not until it starts writing zero's that 1-2 minutes in the dashboard shows all CPU's maxed out. iotop shows the dd process writing the zero's at a high % and another process labeled [kworker/u16:7+flush-8:32]. see attached screenshot. Logs don't show much. I'm unable to find how to turn debugging on based on what gfjardim said a few posts back. Here are the preclear plugin logs for the drive itself Dec 01 19:10:01 preclear_disk_8CJLTRWE_10041: Zeroing: dd if=/dev/zero of=/dev/sdc bs=2097152 seek=26138902528 count=11973999722496 conv=notrunc iflag=count_bytes,nocache,fullblock oflag=seek_bytes Dec 01 19:10:01 preclear_disk_8CJLTRWE_10041: Zeroing: dd pid [12894] Dec 01 19:12:50 preclear_disk_8CJLTRWE_10041: SIGTERM received, exiting... Dec 01 20:45:44 preclear_disk_8CJLTRWE_10404: Command: /usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh --notify 2 --frequency 2 --cycles 1 --no-prompt /dev/sdc Dec 01 20:45:44 preclear_disk_8CJLTRWE_10404: Preclear Disk Version: 1.0.6 Dec 01 20:45:45 preclear_disk_8CJLTRWE_10404: S.M.A.R.T. info type: default Dec 01 20:45:45 preclear_disk_8CJLTRWE_10404: S.M.A.R.T. attrs type: default Dec 01 20:45:45 preclear_disk_8CJLTRWE_10404: Disk size: 12000138625024 Dec 01 20:45:45 preclear_disk_8CJLTRWE_10404: Disk blocks: 2929721344 Dec 01 20:45:45 preclear_disk_8CJLTRWE_10404: Blocks (512 byte): 23437770752 Dec 01 20:45:45 preclear_disk_8CJLTRWE_10404: Block size: 4096 Dec 01 20:45:45 preclear_disk_8CJLTRWE_10404: Start sector: 0 Dec 01 20:45:47 preclear_disk_8CJLTRWE_10404: Pre-Read: dd if=/dev/sdc of=/dev/null bs=2097152 skip=0 count=12000138625024 conv=notrunc iflag=nocache,count_bytes,skip_bytes Dec 01 20:47:40 preclear_disk_8CJLTRWE_10404: SIGTERM received, exiting... Dec 01 20:49:38 preclear_disk_8CJLTRWE_32581: Command: /usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh --load-file /tmp/.preclear/8CJLTRWE.resume /dev/sdc Dec 01 20:49:38 preclear_disk_8CJLTRWE_32581: Preclear Disk Version: 1.0.6 Dec 01 20:49:38 preclear_disk_8CJLTRWE_32581: Restoring previous instance of preclear Dec 01 20:49:38 preclear_disk_8CJLTRWE_32581: S.M.A.R.T. info type: default Dec 01 20:49:38 preclear_disk_8CJLTRWE_32581: S.M.A.R.T. attrs type: default Dec 01 20:49:38 preclear_disk_8CJLTRWE_32581: Disk size: 12000138625024 Dec 01 20:49:38 preclear_disk_8CJLTRWE_32581: Disk blocks: 2929721344 Dec 01 20:49:38 preclear_disk_8CJLTRWE_32581: Blocks (512 byte): 23437770752 Dec 01 20:49:38 preclear_disk_8CJLTRWE_32581: Block size: 4096 Dec 01 20:49:38 preclear_disk_8CJLTRWE_32581: Start sector: 0 Dec 01 20:49:40 preclear_disk_8CJLTRWE_32581: Continuing disk read from byte 26449281024 Dec 01 20:49:40 preclear_disk_8CJLTRWE_32581: Pre-Read: dd if=/dev/sdc of=/dev/null bs=2097152 skip=26449281024 count=11973689344000 conv=notrunc iflag=nocache,count_bytes,skip_bytes Dec 01 21:11:45 preclear_disk_8CJLTRWE_32581: SIGTERM received, exiting... Dec 01 21:11:56 preclear_disk_8CJLTRWE_25713: Command: /usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh --cycles 1 --skip-preread --no-prompt /dev/sdc Dec 01 21:11:56 preclear_disk_8CJLTRWE_25713: Preclear Disk Version: 1.0.6 Dec 01 21:11:56 preclear_disk_8CJLTRWE_25713: S.M.A.R.T. info type: default Dec 01 21:11:56 preclear_disk_8CJLTRWE_25713: S.M.A.R.T. attrs type: default Dec 01 21:11:56 preclear_disk_8CJLTRWE_25713: Disk size: 12000138625024 Dec 01 21:11:56 preclear_disk_8CJLTRWE_25713: Disk blocks: 2929721344 Dec 01 21:11:56 preclear_disk_8CJLTRWE_25713: Blocks (512 byte): 23437770752 Dec 01 21:11:56 preclear_disk_8CJLTRWE_25713: Block size: 4096 Dec 01 21:11:56 preclear_disk_8CJLTRWE_25713: Start sector: 0 Dec 01 21:11:58 preclear_disk_8CJLTRWE_25713: Zeroing: emptying the MBR. Dec 01 21:11:58 preclear_disk_8CJLTRWE_25713: Zeroing: dd if=/dev/zero of=/dev/sdc bs=2097152 seek=2097152 count=12000136527872 conv=notrunc iflag=count_bytes,nocache,fullblock oflag=seek_bytes Dec 01 21:11:58 preclear_disk_8CJLTRWE_25713: Zeroing: dd pid [28276] Dec 01 21:15:12 preclear_disk_8CJLTRWE_25713: SIGTERM received, exiting... And disk log information Nov 21 18:52:32 Tower kernel: sd 1:0:1:0: [sdc] 23437770752 512-byte logical blocks: (12.0 TB/10.9 TiB) Nov 21 18:52:32 Tower kernel: sd 1:0:1:0: [sdc] 4096-byte physical blocks Nov 21 18:52:32 Tower kernel: sd 1:0:1:0: [sdc] Write Protect is off Nov 21 18:52:32 Tower kernel: sd 1:0:1:0: [sdc] Mode Sense: 73 00 00 08 Nov 21 18:52:32 Tower kernel: sd 1:0:1:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Nov 21 18:52:32 Tower kernel: sd 1:0:1:0: [sdc] Attached SCSI disk Nov 21 18:52:58 Tower emhttpd: WDC_WD120EMAZ-11BLFA0_8CJLTRWE (sdc) 512 23437770752 Nov 28 21:35:27 Tower emhttpd: WDC_WD120EMAZ-11BLFA0_8CJLTRWE (sdc) 512 23437770752 Nov 28 21:35:48 Tower emhttpd: WDC_WD120EMAZ-11BLFA0_8CJLTRWE (sdc) 512 23437770752 Nov 28 21:39:09 Tower emhttpd: WDC_WD120EMAZ-11BLFA0_8CJLTRWE (sdc) 512 23437770752 Nov 28 21:39:42 Tower emhttpd: WDC_WD120EMAZ-11BLFA0_8CJLTRWE (sdc) 512 23437770752 Nov 30 09:05:32 Tower emhttpd: WDC_WD120EMAZ-11BLFA0_8CJLTRWE (sdc) 512 23437770752 Nov 30 09:07:45 Tower emhttpd: WDC_WD120EMAZ-11BLFA0_8CJLTRWE (sdc) 512 23437770752 Nov 30 09:07:57 Tower emhttpd: WDC_WD120EMAZ-11BLFA0_8CJLTRWE (sdc) 512 23437770752 Dec 1 19:34:59 Tower preclear_disk_8CJLTRWE[16263]: Command: /usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh --cycles 1 --skip-preread --no-prompt /dev/sdc Dec 1 19:35:03 Tower preclear_disk_8CJLTRWE[16263]: Zeroing: dd if=/dev/zero of=/dev/sdc bs=2097152 seek=2097152 count=12000136527872 conv=notrunc iflag=count_bytes,nocache,fullblock oflag=seek_bytes Dec 1 19:39:59 Tower preclear_disk_8CJLTRWE[10041]: Command: /usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh --load-file /tmp/.preclear/8CJLTRWE.resume /dev/sdc Dec 1 19:40:01 Tower preclear_disk_8CJLTRWE[10041]: Zeroing: dd if=/dev/zero of=/dev/sdc bs=2097152 seek=26138902528 count=11973999722496 conv=notrunc iflag=count_bytes,nocache,fullblock oflag=seek_bytes Dec 1 21:15:44 Tower preclear_disk_8CJLTRWE[10404]: Command: /usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh --notify 2 --frequency 2 --cycles 1 --no-prompt /dev/sdc Dec 1 21:15:47 Tower preclear_disk_8CJLTRWE[10404]: Pre-Read: dd if=/dev/sdc of=/dev/null bs=2097152 skip=0 count=12000138625024 conv=notrunc iflag=nocache,count_bytes,skip_bytes Dec 1 21:19:38 Tower preclear_disk_8CJLTRWE[32581]: Command: /usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh --load-file /tmp/.preclear/8CJLTRWE.resume /dev/sdc Dec 1 21:19:40 Tower preclear_disk_8CJLTRWE[32581]: Pre-Read: dd if=/dev/sdc of=/dev/null bs=2097152 skip=26449281024 count=11973689344000 conv=notrunc iflag=nocache,count_bytes,skip_bytes Dec 1 21:41:56 Tower preclear_disk_8CJLTRWE[25713]: Command: /usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh --cycles 1 --skip-preread --no-prompt /dev/sdc Dec 1 21:41:58 Tower preclear_disk_8CJLTRWE[25713]: Zeroing: dd if=/dev/zero of=/dev/sdc bs=2097152 seek=2097152 count=12000136527872 conv=notrunc iflag=count_bytes,nocache,fullblock oflag=seek_bytes