Spotweb Module anyone?? News index php/web

has anyone seen this https://github.com/spotweb/spotweb

 

It might help with low power servers not having to do all the indexing .... spots are provided via news itself!

(People have it running on a raspberry pi)

 

Looks like sab/headphones/couch have been integrated

 

Thoughts?

 

Tony

Looks interesting..

Trying it out now.

 

So far, it's pretty easy to install as long as you have the simple feature web server and mysql installed.

this "Retrieve" is a lot easier than newznab's update scripts..dont need to worry about all the regex it seems..

 

and looks like it can be used with couchpotato/sickbeard according to https://github.com/spotweb/spotweb/wiki/Spotweb-als-Newznab-Provider

It's in dutch but chrome translate fairly well

well.. 'retrieve' button doesnt work too well.

it's better to use screen and run php retrieve.php

I was trying it on my qnap last night, coudnt get the admin page so show usenet setting. This may be a good option.

got it working on my unraid - well, its downloading the spots as we speak.....

I'm actually liking this more than newznab for now.

I haven't tested with sickbeard or couch potato yet though.

 

For those of you who is interested.. found the rewrite for api's and cron job to schedule retrieve.php on their wiki.

 

lighttpd rewrite

url.rewrite-once = ( "^/api\?(.*)" => "/index.php?page=newznabapi&$1" )

 

cron (Every 4 hours)

0   */4 *   *   *   /usr/bin/php /var/www/spotweb/retrieve.php >> /var/log/spotweb

 

It's getting a lot more data than i actually need.. but looking good

A step by step quide would be great for us Linux impaired.

Got sabnzb integration working .... looking good!.

API and cron working too, thanks.

 

More spots downloading, its at 1y9m and getting closer!

Got sabnzb integration working .... looking good!.

API and cron working too, thanks.

 

More spots downloading, its at 1y9m and getting closer!

 

Yea.. I didnt think I would need anything before 1 year, so i set the date to start at 1/1/2012. Retrieve was done in 30 mins maybe?

And I hope you're running retrieve.php in either command line or screen. Just like newznab's backfilling, first time will take forever because it goes back few years by default..

 

 

And batt01.. i will try to write something up

Here is the write up

 

1. Requirements

You need to have a web server and mysql running on your server to get SpotWeb to work.

 

Get SimpleFeatures web server here: http://lime-technology.com/forum/index.php?topic=12698.0

Get MySQL plug in for unRAID here: http://lime-technology.com/forum/index.php?topic=20603.0

 

2. Database Setup

SpotWeb does not create a database for you, so you need to create an empty database.

Telnet to your unRAID server

Run this command

mysql -u root -p

Type in mysql password.

you should see something like this if you're connected

mysql>

 

Type this command in to create database

CREATE DATABASE spotweb;

this will create 'spotweb' database.

 

Type

exit

To exit out of mysql command.

 

3. Download SpotWeb

You can either go here: https://github.com/spotweb/spotweb then click on 'ZIP' to download the whole repository.. or use GIT to download

 

I'm going to assume that you have sickbeard plug in installed from here (http://lime-technology.com/forum/index.php?topic=21260.0).

The reason why is that I don't want to go through necessary packages to install 'git', and sickbeard plug in from influencer already installs them.

So, if you have sickbeard plug in installed, follow this instruction to get the file. otherwise, probably easier just to go to the github site to download them

 

From here, I am assuming that you have Web Root directory as '/mnt/cache/web/'. If it's somewhere else, change the command according to that

 

Telnet to your unRAID if you haven't already, and type in this command to go to web root

 

cd /mnt/cache/web/

 

then type in this command to download files needed:

 

git clone -b master https://github.com/spotweb/spotweb spotweb

 

This will create a sub directory (spotweb) in your webroot and download files from github.

 

4. Install SpotWeb

 

If you followed above steps, you should be able to open install.php file in your web browser with this address:

http://tower:81/spotweb/install.php

(Change Tower:81 to your simplefeature web server setting)

 

Install.php

It's okay to have 'NOT OK' for DB::pgsql and Own settings file. They don't matter.

 

If anything else says something else, try to fix it.

 

Other wise, hit 'Next'.

 

Database Settings

 

Type: mysql

Server: localhost (if localhost doesnt work, type in the name of your server)

database: spotweb

username: root

password: root (or whatever your mysql password is)

 

You can probably create a new user just to access this, but if you're just trying this out or using it internally, shouldnt matter

 

Usenet server settings

You can either choose a server from the drop down  menu and type in the info

or choose 'Custom' and put in your usenet server info.

 

DO NOT put the port number. It will cause some error. You can set up SSL and some others after you have successfully installed it

 

Then hit 'Verify Usenet Server'

 

Spotweb type/Administrative user

Self explanatory. Choose whatever setting you want to use, create a username, then 'Create 'System'

 

Installation succesful

When you see this message, you're done.

This page does not have a link to go to SpotWeb page though.. weird.

so just go to

http://tower:81/spotweb/

 

5. Retrieve NZB's

Do not press 'Retrieve' button on the page just yet. This is going to take a long time at first.

By default, it will get everything from 11/1/2009 (or 1000 days? i dont know). If you don't need to go back that far, you can choose the date by going

 

config -> setting -> retrieve

 

just select the date.

 

Then open up the telnet connection to your unRAID machine.

 

Go to screen by just typing

screen

 

then navigate to your spotweb directory. in my case it's /mnt/cache/web/spotweb/

 

cd /mnt/cache/web/spotweb/

 

and then run this command

 

php retrieve.php

 

It will start gathering stuff and you should be able to see it on SpotWeb as it updates. This may take a while. This is why I wanted you to run it under screen.. so just let it run for awhile

 

I'm sure you know screen commands.. but just in case you dont know

 

Detach Screen: Ctrl + A -> Ctrl + D

Back to Screen: screen -r

 

5. Rewrite Setup

If you want to use SpotWEb with CouchPotato or SickBeard, you need to set this rewrite.

 

open up lighttpd.cfg file in '/boot/config/plugins/simpleFeatures/'

or it's in your flash drive.. hopefully you know where to find it

 

and add this rewrite rule

 

url.rewrite-once = ( "^/spotweb/api\?(.*)" => "/spotweb/index.php?page=newznabapi&$1" )

 

This is if you have it set up so the address for spotweb is http://tower:81/spotweb/

If not.. well change your rewire rule lol

 

Test your API by opening this link

 

http://server/spotweb/api?t=c

 

5. Schedule Retrieve

 

You can run these commands to add cron job to run retrieve every two hours.

You can also add these line to go file (/boot/config/go) to set it automatically on boot up

 

crontab -l > /tmp/crontab
echo "# SpotWeb Retrieve" >> /tmp/crontab
echo "0 */2 * * * /usr/bin/php /mnt/cache/web/spotweb/retrieve.php >> /var/log/spotweb" >> /tmp/crontab
crontab /tmp/crontab

 

MAKE SURE TO CHANGE '/mnt/cache/web/spotweb/' TO WHEREVER YOU INSTALLED YOUR SPOTWEB

 

Also, if you want to make it to run every hour, you can change */2 to */1.. if you need to run it sooner than that, read this page: http://en.wikipedia.org/wiki/Cron

 

END

 

Is this good enough?

Thanks tor taking the time to write up instructions. I installed spotweb in the webroot with newznab and I get 404 when I try to go to the install page http://tower:81/install.php. '>http://tower:81/install.php.  http://tower:81 will bring up newznab so the webserver works.

you shouldn't be installing them in the same folder. you should be trying to open up http://tower:81/spotweb/install.php

 

also do you have rewrite setup for your newznab?

 

That's going to cause problem with spotweb's rewrite.

either delete newznab's rewrite and use only spotweb's rewrite. (newznab won't be accessible)

 

or put newznab and spotweb into a seprate folder under web root, so it would be tower:81/newznab/ and tower:81/spotweb/

Ok I got it, the /install.php is going. I re installed  the webserver to get the default config. Now its pulling down data.  I have spotweb and newznab both working.

Question, what the difference  in running the php script and clicking the retrieve button?

 

Spotweb looks pretty good as a alternative to Newznab.

 

Looks like  spotweb does not like Chrome. I had to switch to IE to login.

 

My setup is pretty much the same, however I have a "." on the directory that mysql and www are installed in to stop mover from interfering.

This is pretty sweet.  Just installed it on my Ubuntu guest, and it's working really well.  I dig this a lot but... seems like many of the releases and just about all the comments are in Dutch!  Chrome doesn't auto-translate either, since the whole page isn't in Dutch (maybe I should switch the language setting?). 

 

Thanks for sharing.  I'm going to be playing with this for a while.

Hmm.  Well, I have the rewrite going for the API, but SB just can't find anything from it.  Returns a whole bunch of irrelevant results, according to the log.  I wonder what's wonky with my setup.

Hmm.  Well, I have the rewrite going for the API, but SB just can't find anything from it.  Returns a whole bunch of irrelevant results, according to the log.  I wonder what's wonky with my setup.

 

What's the 'Irrelevant Results' ?

If I search within SB for a specific episode of a specific show, for example, it will return (found in the log) words that may exist within the show name, but it doesn't seem to find the actual show.  If I search within my spotweb install, it finds it fine.  If I use a true Newznab provider, it finds it fine.  It just seems to be with my spotweb install that Sickbeard can't find anything (via the API)...

interesting..

i've never actually tested it against sb before so i wouldnt know about htis problem too well..

thanks for the writeup

 

But I can't find if and where you can filter out certain newsgroups?

thanks for the writeup

 

But I can't find if and where you can filter out certain newsgroups?

 

Yea that's one bad thing about this.. it doesnt have that option..

i heard that's because how spotnet works? idk much about it lol

The spot basically has an nfo and a link to the nzb's posted on usenet, correct? I'm curious how old are the posts it retrieves compared to real time indexing?

Can anyonne help me with respect to the "rewrite rules" ?  I have been able to get the spotweb install working but only by deleting the newznab rewrite rules (which causes newznab to not function anymore).

 

Can anyone tell me how they have been succesfull in having both active ?

 

Would also like to add that this instruction was 100% usefull ! Very easy install and I am thinking this might be a better way to go then newznab..

This is my lighttpd.cfg

 

# User customizations
# Add your own settings here. These won't get overwritten upon reboot
#

$SERVER["socket"] == "172.16.1.65:8123" {
server.document-root = "/mnt/cache/web/newznab/www/"

url.rewrite-once = (
	"^/.*\.(css|jpg|jpeg|gif|png|js|ico)" => "$0",
	"^/(admin|install).*$" => "$0",
	"^/([^/\.]+)/?(?:\?(.*))$" => "index.php?page=$1&$2",
	"^/([^/\.]+)/?$" => "index.php?page=$1",
	"^/([^/\.]+)/([^/]+)/?(?:\?(.*))$" => "index.php?page=$1&id=$2&$3",
	"^/([^/\.]+)/([^/]+)/?$" => "index.php?page=$1&id=$2",
	"^/([^/\.]+)/([^/]+)/([^/]+)/?$" => "index.php?page=$1&id=$2&subpage=$3"
)
}

$SERVER["socket"] == "172.16.1.65:8321" {
server.document-root = "/mnt/cache/web/owncloud/"
}

$SERVER["socket"] == "172.16.1.65:7123" {
server.document-root = "/mnt/cache/web/SpotWeb/"

url.rewrite-once = ( 
	"^/api\?(.*)" => "/index.php?page=newznabapi&$1" 
)
}

 

I'm just going to throw this right in here... heh..

 

My actual webroot is set as "/mnt/cache/web/main"

My unRAID machine's IP is "172.16.1.65"

 

i'm sure you guys will figure out what i'm doing here.. and i am sure this setting will be very helpful for some of you

 

if you guys want an explanation on this setting.. I'll be happy to, but it's just fun for you to figure out u know

The rewrite rule appears to be wrong, this link describes how it should be :

 

https://github.com/spotweb/spotweb/issues/1649