MovieGrabber - xml and discussion


Recommended Posts

I installed the MovieGrabber image from eroz.  My xml template is attached for anyone that wants to use it.

 

I was able to connect to moviegrabber, and make several changes to the config, and they all seemed to 'stick.

 

Once finished, I hit "Run" and got...

 

500 Internal Server Error

The server encountered an unexpected condition which prevented it from fulfilling the request.

Traceback (most recent call last):
  File "/opt/moviegrabber/lib/site-packages/cherrypy/_cprequest.py", line 656, in respond
    response.body = self.handler()
  File "/opt/moviegrabber/lib/site-packages/cherrypy/lib/encoding.py", line 188, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/opt/moviegrabber/lib/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/opt/moviegrabber/MovieGrabber.py", line 6133, in run_now
    SearchIndexThread().checks()
  File "/opt/moviegrabber/MovieGrabber.py", line 6331, in checks
    self.run()
  File "/opt/moviegrabber/MovieGrabber.py", line 6373, in run
    search_index_function = getattr(SearchIndex(self.download_type,self.index_site_item), self.search_index_function)
  File "/opt/moviegrabber/MovieGrabber.py", line 1454, in __init__
    self.config_bad_report = (config_parser.get(download_type, index_site_item + "_bad_report")).decode("utf-8")
  File "/usr/lib/python2.7/ConfigParser.py", line 618, in get
    raise NoOptionError(option, section)
NoOptionError: No option 'nzb_su_1_bad_report' in section: 'usenet'

 

Not sure if this is a problem with the docker, with my setup, or with moviegrabber. :(

my-MovieGrabber.xml.txt

Link to comment

No more errors.  After clicking "Run Now", it updates the "Last Run On:" date, but nothing much else seems to happen.

 

Perhaps I have my IMDB settings too high, but I've never run this program before, so I could just have it set all wrong.

 

I expected it would be a bit more like CouchPotato, and let me search for a movie by name, but after going thru the config, it seems it's ONLY meant to find movies that meet the criteria I set, which is not quite as useful as I thought it would be.

 

I'll let it run for a while and see if I get any hits, and report back later.

 

Thanks for fixing it for me!

Link to comment

No more errors.  After clicking "Run Now", it updates the "Last Run On:" date, but nothing much else seems to happen.

 

Perhaps I have my IMDB settings too high, but I've never run this program before, so I could just have it set all wrong.

 

I expected it would be a bit more like CouchPotato, and let me search for a movie by name, but after going thru the config, it seems it's ONLY meant to find movies that meet the criteria I set, which is not quite as useful as I thought it would be.

 

I'll let it run for a while and see if I get any hits, and report back later.

 

Thanks for fixing it for me!

No problem, it was all binhex.  We found a bug and he was able to correct it quickly.
Link to comment

Hi Justinchase most people get hung up on the search criteria, can you pm or post here a screenshot of your Usenet settings page, make sure to delete/hide your api key

 

One other thing to note is that your queue date should be a later date than your good date otherwise you won't queue anything, all matching movies will instead be downloaded.

 

History is where you should be looking to view matching/skipped movies BTW.

 

Link to comment

NP OK I see the issue straight away, you have no search criteria, try entering something like 1080p,bluray into the must exist search criteria save and then run now

 

still no results.  I added "1080p, BluRay" but still nothing.  I changed max size to 100000, still no results.

 

isn't the hostname for nzb.su  https://api.nzb.su

 

I tried changing to that also, still no results.  I also changed to https://www.nzb.su, still no results.

 

Here is my IMDB page.  I added some information in some of the fields, but still get no results.  (This info was added before the 1080p, bluray change I just made.

imdb.jpg.3ffaa0ab6e4336a9bba31f7a4fd40ee3.jpg

Link to comment

isn't the hostname for nzb.su  https://api.nzb.su

 

Thanks for this eroz, It used to be http://nzb.su/api if this has changed then for now Justinchase will need to create a custom site (in Usenet section) and fill in the correct URL.

 

Please let me know so I can change the template if required.

 

Edit - It has indeed changed and they have now disabled forwarding to the new URL.

 

Still no luck, I amended the original search provider, and added a new, custom one with http://nzb.su/api, but still no results.

Link to comment

binhex and I have exchanged a few PM's to try to resolve this, and I think he's gotten it fixed...

 

i think i have found the issue, well there are two issues in total, the first one you know about which is the url has changes for the api, the second issue is with the search criteria and how it deals with spaces between items, in short i wasnt stripping spaces from the comma seperated items in the "must exist" search criteria, so your search criteria of "1080p, bluray" actually gets translated into "1080p%20%20bluray" which matches nothing, i have now put in additional code checks for this so the issue should now be fixed.

 

so in short you can either get eroz to force a build for moviegrabber docker, then delete container and image and pull down again, or simply change your search criteria so it doesnt have a space in it, so just to be clear it should look like this (excluding quotes) "1080p,bluray" NOT this "1080p, bluray"

 

please double check you have the url still correctly set as https://api.nzb.su and port set to 443 then hit the run now, this works fine on my dev box :-).

 

thanks for your patience with this, i hope this sorts the issue for you, let me know either way.

 

I have removed the space and tested, and it is now pulling movies into the search history; yay!

 

If eroz can/will recreate the container with the updates, future users will not have to worry about this.

 

 

As for functionality, the History is showing lots of movies, some seem to be outside my criteria, like too low of rating, too old, etc.  Also, some are duplicates.  I suspect the duplicates are because they are duplicated on nzb.su, but it would be nice if MovieGrabber would recognize them as duplicates and only show one version in the History tab.

 

Also, I'm not sure if they history should be showing stuff that doesn't match my criteria, of if that is, in fact, normal.  I would suggest from a usability standpoint, it should not show them.

 

Finally, now that there is a history, there is also a search box on that page.  As a test, I typed Field of Dreams and got No items found.  Not sure if it's only looking in the history, or if it's looking at nzb.su.  i also don't know if I need the year at the end, like the example shows, but if so, I would suggest removing this requirement, as I have no idea what year most movies are released off the top of my head :)

 

I'll keep playing around, and report back any issues, or suggestions I have.

 

It does look like an interesting program, and I look forward to watching it develop.

Link to comment

As for functionality, the History is showing lots of movies, some seem to be outside my criteria, like too low of rating, too old, etc. 

 

History will list ALL movies, this is not what is accepted for download/queue but is just a list of processing of the movies showing why its status is set to "Skipped", "Queued" or "Downloaded".

 

i noticed from your config.ini your votes are very low, i would advise starting with a minimum vote count of around 10000 keep in mind imdb is massive and even a min vote count of 10000 will give you plenty of matching movies.

 

Also, some are duplicates.  I suspect the duplicates are because they are duplicated on nzb.su, but it would be nice if MovieGrabber would recognize them as duplicates and only show one version in the History tab.

 

moviegrabber does track posts by postname NOT imdb name, so you may see duplicate imdb movie entries but you should not be seeing duplicate postname entries, this is to allow the user to choose the "best" copy, maybe from their favorite scene group.

 

Also, I'm not sure if they history should be showing stuff that doesn't match my criteria, of if that is, in fact, normal.  I would suggest from a usability standpoint, it should not show them.

 

as stated earlier History shows ALL processed items, if you want to see what has matched your criteria you can sort by "status" which should show you queued and downloaded items at the top of the History.

 

Finally, now that there is a history, there is also a search box on that page.  As a test, I typed Field of Dreams and got No items found.  Not sure if it's only looking in the history, or if it's looking at nzb.su.  i also don't know if I need the year at the end, like the example shows, but if so, I would suggest removing this requirement, as I have no idea what year most movies are released off the top of my head :)

 

the search box is for searching the processed movies in History, it does not search the index site, moviegrabber is all about automation and not about plugging away searching for particular movies i.e. like couchpotato

 

you can do historical processing though, this can be achieved by increasing the number of posts processed in configuration/general "posts to process", if you want to do a large backlog process try setting this to something like 4000, i would HIGHLY advise you tweak your imdb settings though before you do this or you could end up with a MASSIVE queue in your usenet binary downloader.

 

lastly thanks for hanging in there whilst we squashed a couple more bugs, glad we got there in the end! :-)

Link to comment

History will list ALL movies, this is not what is accepted for download/queue but is just a list of processing of the movies showing why its status is set to "Skipped", "Queued" or "Downloaded".

 

That makes sense, I just wasn't sure.

 

i noticed from your config.ini your votes are very low, i would advise starting with a minimum vote count of around 10000 keep in mind imdb is massive and even a min vote count of 10000 will give you plenty of matching movies.

 

Done, thanks for the tip.

 

moviegrabber does track posts by postname NOT imdb name, so you may see duplicate imdb movie entries but you should not be seeing duplicate postname entries, this is to allow the user to choose the "best" copy, maybe from their favorite scene group.

 

Okay, that makes sense also.

 

if you want to see what has matched your criteria you can sort by "status" which should show you queued and downloaded items at the top of the History.

 

good idea.  I actually sorted by rating, which is also useful.

 

the search box is for searching the processed movies in History, it does not search the index site, moviegrabber is all about automation and not about plugging away searching for particular movies i.e. like couchpotato

 

I like the automated-ness of the program, it finds stuff I wouldn't find on my own.  I would suggest that perhaps a 'forced search' feature would be useful also.  That would allow us to remove CouchPotato, and just use this one program.  Personally, I don't use CouchPotato since it never seems to work, but it would be nice to have this search capability back (without going to nzb.su directly).

 

lastly thanks for hanging in there whilst we squashed a couple more bugs, glad we got there in the end! :-)

 

No problem at all.  Thank you for your prompt attention and resolution.  This is a very cool program and I'm looking forward to watching it get better over time!

Link to comment

np and glad your enjoying it, one other thing, im not sure if you tweaked your "queue date" after my suggestion earlier but i think it was set to the same date as your "good date" which basically disables queuing, or at least it does based on imdb year (queue genre will still work if defined), if you want to make use of the queue feature i would set it to a later date, it will then queue all items from good date to queue date e.g.:-

 

good date = 1960

queue date = 1990

 

all items will be queued for review by yourself (obviously putting them in the Queue tab) before being downloaded, this does mean there is a delay to the automation of moviegrabber, but you can receive email notification so you are made aware of queued items, this is the way i use it and it works really well, also i think you may need to also enable queuing the configuration/switches section, i will be removing this requirement shortly and automatically enabling it if either queue date or genre are defined.

 

i hear your suggestion for being able to do a search of the index site, i have had this as to-do item for some time, i will make sure its on the list, no promises though as i got plenty that needs to be done before it. :-)

Link to comment

@binhex,  I'm getting this error when pressing the run now button.

Traceback (most recent call last):
  File "/opt/moviegrabber/lib/site-packages/cherrypy/_cprequest.py", line 656, in respond
    response.body = self.handler()
  File "/opt/moviegrabber/lib/site-packages/cherrypy/lib/encoding.py", line 188, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/opt/moviegrabber/lib/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/opt/moviegrabber/MovieGrabber.py", line 6140, in run_now
    SearchIndexThread().checks()
  File "/opt/moviegrabber/MovieGrabber.py", line 6338, in checks
    self.run()
  File "/opt/moviegrabber/MovieGrabber.py", line 6380, in run
    search_index_function = getattr(SearchIndex(self.download_type,self.index_site_item), self.search_index_function)
  File "/opt/moviegrabber/MovieGrabber.py", line 1489, in __init__
    self.movies_downloaded_cache = list(itertools.chain.from_iterable(os.walk(root_path) for root_path in movies_downloaded_dir_list))
  File "/usr/lib/python2.7/os.py", line 294, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "/usr/lib/python2.7/os.py", line 294, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "/usr/lib/python2.7/os.py", line 294, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "/usr/lib/python2.7/os.py", line 284, in walk
    if isdir(join(top, name)):
  File "/usr/lib/python2.7/posixpath.py", line 80, in join
    path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 11: ordinal not in range(128)

Any idea what I'm doing wrong?

Link to comment

@binhex,  I'm getting this error when pressing the run now button.

Traceback (most recent call last):
  File "/opt/moviegrabber/lib/site-packages/cherrypy/_cprequest.py", line 656, in respond
    response.body = self.handler()
  File "/opt/moviegrabber/lib/site-packages/cherrypy/lib/encoding.py", line 188, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/opt/moviegrabber/lib/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/opt/moviegrabber/MovieGrabber.py", line 6140, in run_now
    SearchIndexThread().checks()
  File "/opt/moviegrabber/MovieGrabber.py", line 6338, in checks
    self.run()
  File "/opt/moviegrabber/MovieGrabber.py", line 6380, in run
    search_index_function = getattr(SearchIndex(self.download_type,self.index_site_item), self.search_index_function)
  File "/opt/moviegrabber/MovieGrabber.py", line 1489, in __init__
    self.movies_downloaded_cache = list(itertools.chain.from_iterable(os.walk(root_path) for root_path in movies_downloaded_dir_list))
  File "/usr/lib/python2.7/os.py", line 294, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "/usr/lib/python2.7/os.py", line 294, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "/usr/lib/python2.7/os.py", line 294, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
  File "/usr/lib/python2.7/os.py", line 284, in walk
    if isdir(join(top, name)):
  File "/usr/lib/python2.7/posixpath.py", line 80, in join
    path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 11: ordinal not in range(128)

Any idea what I'm doing wrong?

 

hi eroz, yep i know what this is due to as i got tripped up on this too!, its basically that your docker image is not setting the locale correctly, i see from your dockerfile that you set the locale with this line:-

 

RUN locale-gen en_US en_US.UTF-8

 

however i dont think its setting the locale correctly, and thus the locale is falling back to ascii char set only, my code is then tripping up when it comes across a non ascii character in your movies downloaded folder, e.g. a movie folder named "Æon Flux (2005)" would do it but any non ascii movie title will cause this to happen.

 

if you want to see whats going on you can debug your docker image using the following:-

 

docker run -i -t eroz/moviegrabber /bin/bash

 

then try running the script ive attached to this post, this will identify if your locale is set correctly or not.

 

edit - hmm one interesting thing to note is that Needo is not setting the locale for any of his dockerfiles, which is a bit odd, either he has a workaround for it that is outside of his dockerfile or he just isnt aware that he is running a default ascii char set!.

 

 

 

 

locale-check.sh.txt

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.