Introducing boiler, a plugin packaging and distribution system for unRAID

Project: http://getboiler.com/

Guides: http://getboiler.com/guides/

Registry: http://registry.getboiler.com/

Code: https://github.com/nicinabox/boiler

Install:

// curl
curl -s http://getboiler.com/install | sh

// Or wget
wget -qO- http://getboiler.com/install | sh -

We've desperately needed a simple way to distribute plugins for a long time. During my work on Boxcar I found that I needed a simple way to fetch and install system packages, and trolley was born. Trolley and boiler go hand in hand.

At the time, I thought building a plugin manager into boxcar was the way to go. After a lot of careful consideration, I've opted not to do this. Instead, I decided to go for a more generic system, so even existing plugins can be distributed easily (hooray!). Enter boiler.

Boiler standardizes how packages are designed. It solves a lot of common problems for you, so you can focus on making a killer app, rather than dealing with annoying problems. It paves the way for better packages, apps, and services on unRAID.

Plugin Philosophy

1. Plugins are tarballs, compatible with the native Slackware package system.

2. Plugins follow best practices for versioning and releases.

3. Plugin code is publicly available for review and improvement.

How it works

In simplest terms, boiler acts as a url shortener for your plugin. Your project can be in any git repo you like (Github, Bitbucket, Gitlab, etc), as long as it's publicly accessible. Anyone can run `boiler install NAME` and get your package installed. Distributed packages are tarballs designed to be standalone-compatible with Slackware's native pkgtools.

API driven

The boiler cli is a client tool that consumes a public JSON API.

Awesome git at the top showing it's use...

If there is some kind of testing I can help with, let me know.

Awesome git at the top showing it's use...

 

If there is some kind of testing I can help with, let me know.

Thanks! For anyone interested in contributing, I've put some todo's in the readme: https://github.com/nicinabox/boiler#todo

I would like to install it but can't get it to work

root@Tower:/boot/extra# boiler
/usr/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- bundler (LoadError)
from /usr/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/local/boiler/lib/boiler/cli.rb:1:in `<top (required)>'
from /usr/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/local/bin/boiler:12:in `<main>'

as far as I understood the install script installs gems in

/boot/custom/gems

but when executing

boiler

it tries to load gems from /usr/lib/ruby/gems. Here is an excerpt of

gem environment

  - GEM PATHS:
     - /usr/lib/ruby/gems/1.9.1
     - /root/.gem/ruby/1.9.1

Try reopening your terminal session or run `source /etc/bundlerc`. This is due to env variables that need to take effect in your session.

ok .. now I get a different error

/usr/local/boiler/lib/boiler/version.rb:2:in `read': No such file or directory - boiler.json (Errno::ENOENT)
from /usr/local/boiler/lib/boiler/version.rb:2:in `<module:Boiler>'
from /usr/local/boiler/lib/boiler/version.rb:1:in `<top (required)>'
from /usr/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/local/boiler/lib/boiler/cli.rb:5:in `<top (required)>'
from /usr/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/local/bin/boiler:12:in `<main>'

also -

bundle list

is returning a warning and no meaningful information:

/boot/custom/gems/ruby/1.9.1/gems/bundler-1.3.5/lib/bundler/vendor/thor/shell/basic.rb:336: warning: Insecure world writable dir /boot/custom/gems/ruby/1.9.1/bin in PATH, mode 040777
Could not locate Gemfile

i copied /var/log/boiler/boiler/boiler.json to /usr/local/boiler/lib/boiler .. then it works

actually the boiler.json file has to be in the working directory .. hmm

btw a lot of boilers in this path

I changed the reference to boiler.json in version.rb to be absolute:

module Boiler
  config = JSON.parse File.read "/var/log/boiler/boiler/boiler.json"

  VERSION = config['version']
end

then it works regardless of the file being present in the working dir

Hah, I was just just coming back to say I found another error while looking into that one. Just released 0.2.1 to fix that.

i copied /var/log/boiler/boiler/boiler.json to /usr/local/boiler/lib/boiler .. then it works

 

actually the boiler.json file has to be in the working directory .. hmm

 

btw a lot of boilers in this path

Truth squad. Let me explain because this looks a little goofy:

This is the path all boiler packages metadata gets copied to

 /var/log/boiler

This is the path of a specific package, called boiler

 /var/log/boiler/boiler

Each package will have a manifest called boiler.json

 /var/log/boiler/boiler/boiler.json

This was done to mirror the way the slackware tracks package installs in /var/log/packages. I didn't want to muddle up that directory, so I kept with the same idea.

Neat. Your work is always top notch.

ok - thanks for clarifying that

I have another question:

I want to create a boiler package out of a plugin. I start by converting the plugin to the boiler structure with

boiler convert plugin

. That works good so far.

BTW I'm "testing" it with the SABNZBd plugin from influencer. I noticed that the convert command doesn't recognise all dependencies and also doing things wrong.

I noticed this:

...
"dependencies": [
    "http://slackware.cs.utah.edu/pub/slackware/slackware-13.37/slackware/d/python-2.6.6-i486-1.txz",
    "http://slackware.cs.utah.edu/pub/slackware/slackware-13.37/slackware/ap/sqlite-3.7.5-i486-1.txz",
    "http://slackware.cs.utah.edu/pub/slackware/slackware-13.37/slackware/a/infozip-6.0-i486-1.txz",
    "http://slackware.cs.utah.edu/pub/slackware/slackware-13.37/slackware/n/openssl-0.9.8r-i486-3.txz"
  ],
...

There are two things that are interesting. According to the documentation dependencies should be "A hash of dependencies ("openssl": ">=1.0.1c")..." but in this case the outcome is an array of strings. Does this work?

Secondly there are missing dependencies - as far as I see only packages in the slackware.cs.utah.edu package management are converted - all others are ignored. Unfortunately SABNZBd for example has a few dependencies that originate in a different repository - just some examples:

http://connie.slackware.com/~alien/slackbuilds/python-cheetah/pkg/13.37/python-cheetah-2.4.4-i486-1alien.tgz
http://www.slackware.com/~alien/slackbuilds/unrar/pkg/13.37/unrar-4.2.4-i486-1alien.tgz
http://unraid-greenleaf-repository.googlecode.com/files/par2cmdline_tbb-0.4-i486-1kh.tgz

How would I use boiler configuration to install those? With a post install step?

Also I'm playing with the idea of providing boiler packages for all dependencies like unrar to improve the package version handling

ok - thanks for clarifying that

 

I have another question:

 

I want to create a boiler package out of a plugin. I start by converting the plugin to the boiler structure with

boiler convert plugin

. That works good so far.

 

BTW I'm "testing" it with the SABNZBd plugin from influencer. I noticed that the convert command doesn't recognise all dependencies and also doing things wrong.

 

I noticed this:

 

...
"dependencies": [
    "http://slackware.cs.utah.edu/pub/slackware/slackware-13.37/slackware/d/python-2.6.6-i486-1.txz",
    "http://slackware.cs.utah.edu/pub/slackware/slackware-13.37/slackware/ap/sqlite-3.7.5-i486-1.txz",
    "http://slackware.cs.utah.edu/pub/slackware/slackware-13.37/slackware/a/infozip-6.0-i486-1.txz",
    "http://slackware.cs.utah.edu/pub/slackware/slackware-13.37/slackware/n/openssl-0.9.8r-i486-3.txz"
  ],
...

 

There are two things that are interesting. According to the documentation dependencies should be "A hash of dependencies ("openssl": ">=1.0.1c")..." but in this case the outcome is an array of strings. Does this work?

So, basically, these lines get copied into install/doinst.sh as `trolley install url`. Trolley doesn't support this syntax yet, but it's something I was planning on adding today or tomorrow. Essentially it would act as a proxy to wget. I'd also like to support `name: url` along with `name: version`. Since plgs don't have a standardized API, parsing the xml may not work 100%. It's more of a 90% tool and then you need to review the difference. I'm up for making converting better, but I do need some help from the community on this.

Secondly there are missing dependencies - as far as I see only packages in the slackware.cs.utah.edu package management are converted - all others are ignored. Unfortunately SABNZBd for example has a few dependencies that originate in a different repository - just some examples:

 

http://connie.slackware.com/~alien/slackbuilds/python-cheetah/pkg/13.37/python-cheetah-2.4.4-i486-1alien.tgz
http://www.slackware.com/~alien/slackbuilds/unrar/pkg/13.37/unrar-4.2.4-i486-1alien.tgz
http://unraid-greenleaf-repository.googlecode.com/files/par2cmdline_tbb-0.4-i486-1kh.tgz

 

How would I use boiler configuration to install those? With a post install step?

Ideally you would put them in them under dependencies. This is another good reason to support urls there because then you can specify packages outside the official repo.

Also I'm playing with the idea of providing boiler packages for all dependencies like unrar to improve the package version handling

I've been considering this, but I'm not sure it's a good idea (I'm open to discussion on this). When you specify system packages by version they come directly from the source. If the system package is then wrapped in a boiler package, that author is required to keep the package up to date, and that doesn't bode well years down the line.

There are some packages where this is ideal, however. Overbyrn's openssh plugin comes to mind. It provides unraid specific functionality (dealing with config, key persistence, etc)

thanks for the reply

I started converting the external dependencies to boiler packages and noticed that it is pretty straight forward

all i had to do was mkdir packageName - unpack (with tar ...) - boiler init packageName - boiler pack packageName

there are only a few things that have to be done manually in this process:

• define the version (should be possible to parse??)
  
• define author(s) - remains manually although could be set to a configured name (dawiki in my case:))
  
• define license
  
• define home page
  

I guess this is far down the road but it could be automated - at least for version bumps since it should be possible to take license and home page information from the previous version boiler.json

BTW do I have to register a package or is it possible to use a dry run mechanism and check if everything is working without publishing it to a git repo?

Version defaults to 0.1.0. I'd like to be able to parse from a plg (if converting), but that has been unreliable.

Author defaults to what's defined in your ~/.gitconfig (see here for info on setting that)

License defaults to MIT

Home page defaults to nothing since this could be anything.

Re dry run: I keep a vm around for testing. That's the only way I've been able to really test my packages. This is how I test boiler (it can pack itself):

bin/boiler pack . && scp boiler-0.2.1-noarch-unraid.tgz [email protected]:/root && rm boiler-*

And then run `installpkg boiler-*` on the vm. You can do this without a git repo. Your repo must exist and be public otherwise boiler will reject it.

I managed to get sabNZBd to work with only boiler packages.

Thank you for providing these really easy-to-use tools

I have a few remarks though but I don't want to sound ungrateful

I created a repository for each dependency that is not available in the official slackware repos - I ended up creating these repos:

sabnzbd_unplugged

unrar

python-cheetah

python-yenc

pyopenssl

par2cmdline

It works pretty well and is usable however I expected to only need to install sabnzbd_unplugged but then it presented me the following messages:

...
=> Installing
No package named 'par2cmdline'
No package named 'pyopenssl'
No package named 'python-cheetah'
No package named 'python-yenc'
No package named 'unbar'
=> Installed!

It would be nice if the boiler dependencies were resolved automatically and installed before the "leaf" package (sabnzbd in my case)

Another minor thing is that at first I created a single github repo with just the tgz-packages and linked to the single files from boiler register - well that didn't work and I noticed that the packages are created on the client immediately after executing boiler install

I guess the boiler client is also using git for communication and there is no server part that does pre-packaging and so on

Nevertheless I am quite satisfied with what I accomplished and I thank you again for providing the tools

I think I will continue with the other plugins that I have installed including couchpotato, sickbeard, transmission, openssh and btsync

good night from the center of europe

I managed to get sabNZBd to work with only boiler packages.

 

Thank you for providing these really easy-to-use tools

 

I have a few remarks though but I don't want to sound ungrateful

 

I created a repository for each dependency that is not available in the official slackware repos - I ended up creating these repos:

sabnzbd_unplugged

unrar

python-cheetah

python-yenc

pyopenssl

par2cmdline

 

It works pretty well and is usable however I expected to only need to install sabnzbd_unplugged but then it presented me the following messages:

...
=> Installing
No package named 'par2cmdline'
No package named 'pyopenssl'
No package named 'python-cheetah'
No package named 'python-yenc'
No package named 'unbar'
=> Installed!

 

It would be nice if the boiler dependencies were resolved automatically and installed before the "leaf" package (sabnzbd in my case)

Awesome! Thanks for your feedback

It looks like we need some more robustness in dependencies. Currently, dependencies are for slackware dependencies. They're fetched by trolley according to the package API. The way you're specifying boiler packages along side slackware packages currently doesn't work, but I think it should.

This is what would be ideal in my book:

* resolve slackware packages by name or url

* resolve boiler packages by name

* resolve array of urls (proxy to wget)

I suspect there will end up being a second dependencies entry for boiler packages. They should function identically.

...
"dependencies": {},
"boilerDependencies": {}
...

`dependencies` would be resolved first, then `boilerDependencies` (and each of those packages dependencies).

Another minor thing is that at first I created a single github repo with just the tgz-packages and linked to the single files from boiler register - well that didn't work and I noticed that the packages are created on the client immediately after executing boiler install

 

I guess the boiler client is also using git for communication and there is no server part that does pre-packaging and so on

That's right. Each package must be its own repo. This is better in the long run because it keeps things segregated, keeps the diffs clean, and gives issue tracking to each package. Not to mention versioning is simpler this way.

unRAID itself does the packaging actually. Git is merely the carrier. This is why it's important to use git tags to mark your versions. Boiler will checkout the specified tag (or latest if none is specified) and package that code. By piggybacking off git in this way it makes the entire process a whole lot simpler.

Nevertheless I am quite satisfied with what I accomplished and I thank you again for providing the tools

<3. Thanks again for your feedback! It's very helpful to me

I added direct linking of unofficial packages in dependencies and sent you a pull request (https://github.com/nicinabox/boiler/pull/2)

I converted the transmission_unplugged plugin and this is also registered and works as expected

I would like to refactor sabnzbd and get rid of the other packages I created, but I guess you will have to manually remove the boiler packages since there is no unregister command

Anyway I will post here when I finished work for sabnzbd and no longer need the other boiler packages to make it work

best regards

Thanks, I merged your PR.

If you'd like to unregister a package, just comment on this issue with the ones you'd like to have removed.

Released boiler 0.2.2 and trolley 0.1.11.

dawiki's unplugged (converted) packages above work great!

The registry is now browsable on your favorite internet-capable device: http://registry.getboiler.com/

Publish your plugins with boiler and they'll show up automagically!

Just installed boiler (and filled out the form in your signature).  I'm far from a software developer, but let me know what testing can be done.

I know a lot of people would kill for an updated transmission plugin (currently influencer's plugin is for 2.76, and I think 2.82 is the current version).

Anyway, let me know what I can do to help.

Ogi

Just installed boiler (and filled out the form in your signature).  I'm far from a software developer, but let me know what testing can be done.

 

I know a lot of people would kill for an updated transmission plugin (currently influencer's plugin is for 2.76, and I think 2.82 is the current version).

 

Anyway, let me know what I can do to help.

 

Ogi

I've been running transmission 2.82 for a while now but editing influencer's plugin.  Here it is. Credits to influencer and ockingshay

https://www.dropbox.com/s/5z31u0mlhxpgqq8/transmission_unplugged.plg

Sent from my SM-N9005 using Tapatalk

Just installed boiler (and filled out the form in your signature).  I'm far from a software developer, but let me know what testing can be done.

 

I know a lot of people would kill for an updated transmission plugin (currently influencer's plugin is for 2.76, and I think 2.82 is the current version).

 

Anyway, let me know what I can do to help.

 

Ogi

 

I've been running transmission 2.82 for a while now but editing influencer's plugin.  Here it is. Credits to influencer and ockingshay

https://www.dropbox.com/s/5z31u0mlhxpgqq8/transmission_unplugged.plg

 

 

Sent from my SM-N9005 using Tapatalk

Huh, well that was easy.

The situation above is dangerous and part of what boiler seeks to fix.

1. Centralized registry

No one is going to come to this thread to find a plg download linked to someone's dropbox that may or may not be around in days/months/years. A central registry provides an interface for reliably, trustworthy, versioned code.

2. Out of date packages

If the above package truly is an updated transmission, the appropriate action would be to send a Pull Request to Influencer with the updated code so that a common and recognized package is update more globally.

Boiler let's you get updates directly from the official plugin easily by running

boiler update NAME

2. Accountable developers

There are only a handleful of devs creating worthwhile packages. Boiler creates accountability for devs and gives them better tools to manage their code, including version control through git, issue management if you choose to use Github, and authorship credit defined in the package.

3. Safe and secure code

All source code is public and reviewable. If a malicious package is found, it can be easily stopped at the source. The exchange above is scary because it's just a random guys on an internet forum linking a distributable package from a personal, unverifiable source. The package above should be regarded as volatile: it could disappear at any time or even if it will be replaced by different code. There is no traceable history, no contact, and no accountability for upkeep.

4. Easy distribution

Boiler lets you search, install, and update packages so you can quickly install the packages you need and keep them up to date. Hunting through random threads on a forum is not an acceptable solution.