Open Sourcery

My recent love affair with AMQP & nanite has been quite a nice boost to daemon-kit. First it got a neat AMQP consumer generator, and today it got a nanite agent generator with sample actor.

Here is how to get going quickly:

$ sudo gem install kennethkalmer-daemon-kit -s http://gems.github.com
$ daemon_kit myagent -i nanite_agent
$ cd myagent
$ ruby ./bin/myagent run

And in another console:

$ nanite-mapper -i
>> request('/sample/echo', 'ping') { |r| puts r.inspect }

You should get a ping message back from the sample actor.

The daemon-kit nanite agent implementation is more than just a replacement for the nanite-agent command line tool. It allows you to build more functionality into your agents as well. In my case I’m combining the cron features of daemon-kit with  the nanite agents to build a multi-purpose power daemon.

Fine print

This generator is still in it’s infancy and will undoubtedly have some changes moving forward, but so far it works just great in my simple tests. I’m publishing this to solicit some feedback from the nanite community in general.

Please follow the nanite README closely to ensure your nanite installation is working correctly. Ezra merged my changes to nanite, so the examples are working out the box again.

Disclaimer: I’m still learning nanite and AMQP, so I’m definitely no authority on either and asking me for help might yield longer than expected replies while I figure stuff out myself!

I started conceptualizing and playing with a pet project called daemon-kit earlier this year, with the aim to ultimately be the preferred way of assembling daemon processes written in Ruby.

Today I took the opportunity to add two more generators to daemon-kit, as well as fix some small annoying issues. The first generator is a ‘cron’ generator, which I’ll cover in this article. The second is an AMQP consumer, that my day job requires.

Running cron-style daemon processes seems to be a common need in the Ruby world, and my first ever daemon process was a cron-style implementation using the remarkable rufus-scheduler gem by John Mettraux. The second, an SQS client. Writing Ruby daemon processes is quick and simple, but getting to know the ins and outs of these hidden beasts can be quit a nightmare.

As of late I’ve been threatening in #ruote that daemon-kit will sport a ‘cron’ style generator when I get the time. Today I made time, and you can now get a simple cron daemon up and running in minutes, heres how:

1. Get daemon-kit & co

$ sudo gem install kennethkalmer-daemon-kit
$ sudo gem install rufus-scheduler

rufus-scheduler is not a direct dependency of daemon-kit, but required by the daemons generated using the cron generator.

2. Generate a stub daemon

$ daemon_kit mycrond -i cron

This creates a project layout in a directory named ‘mycrond‘. You can populate the lib folder with your custom code. What matters though is that your generated ‘cron’ daemon lives in libexec/mycrond.rb.

3. Profit

Open up libexec/mycrond.rb to reveal a fully functional cron-style daemon, complete with sample 1 minute task.

Behind the scenes

All of daemon-kit is basically two things: abstracting daemonizing routines and environment configurations, and wrapping supporting libraries in thin wrapper classes for easing their use inside daemon processes. The cron wrapper class is extremely thin, in part due to the excellent implementation of the rufus-scheduler gem.

As with all projects, I cannot imagine every possible use for the gem and rely on feedback from the community. If you are going to attempt using ActiveRecord inside the cron daemon, beware that you might have to juggle some balls with ActiveRecord and threads. Please report these issues on the github tracker and I’ll attempt to find solutions for you. I highly recommend using ActiveRecord 2.3.2 or later to benefit from the connection pooling and thread safety improvements.

UPDATE 2009-04022: Ruby Enterprise Edition 20090421 released a couple of hours after the original post. Now includes the OpenSSL patch from Gentoo’s Portage.

UPDATE: The upcoming release of Ruby Enterprise Edition might include the suggested patch, if we get enough +1’s on the list. Test and go vote!

Keeping your gentoo up to date almost means sacrificing Ruby Enterprise Edition because it doesn’t compile against OpenSSL 0.9.8i or later. This isn’t the fault of the Phusion guys, it comes from upstream Ruby.

Gentoo’s portage contains some patches for compiling stock Ruby 1.8.6 with later versions of OpenSSL, backported from Ruby 1.8.7. This patch also applies cleanly to Ruby Enterprise Edition 20090201, and here is how to do it from inside your unpacked Ruby Enterprise Edition tarball.


$ cd source
$ patch -p3 < /usr/portage/dev-lang/ruby/files/ruby-1.8.6-openssl.patch
$ cd ..
$ sudo ./installer


This keeps your OpenSSL library up to date, and keeps Ruby Enterprise Edition happy.

Some time ago I started a series of blog posts [1,2] on Ruby and messaging. Most of the series is still in draft format and has to be completed. It’s a topic I’ve been very interested for some time, and have monitored the space quite closely looking for new implementations and collecting presentations and links as I go along. I’ve been a fan of Jabber (XMPP) for some time now, but it has some smells, to me at least…

First, some background. I’m the chief utopian at a South African ISP, inX, and most of my current work focusses on finishing off the fourth generation of our flagship project ISP in a Box. ISP in a Box is essentially a Rails frontend to a variety of services we procure and reprovision to resellers, which in turn sell to their clients. Apart from coping with a lot of models and extremely complex business processes, we also have to cope with procuring and provisioning services.

In this post I’ll aim to give some insight into our provisioning processes, and more specifically how I plan to implement our hosting provisioning over the coming days.

We exclusively use the excellent ruote Worflow Engine by John Mettraux to drive our procurement/provisioning processes, instead of overly complex state machines. As part of this I’ve implemented a Jabber Participant/Listener pair of classes for ruote, allowing any part of the workflow to be shelled out to a Jabber bot for processing, and continuing the workflow when done. This has worked great so far, but the requirements of our hosting services are somewhat more. This made me sit down and think, and I announced today that I’ll be doing a “pen and paper” SWOT on XMPP, AMQP and REST. What I came up with, is not a textbook SWOT, but at least a look at AMQP from an XMPP point of view. And it was interesting indeed, so much so that I’ve made up my mind to move to AMQP from XMPP. Here’s what happened…

The smells of XMPP

For the record, I really love XMPP as a protocol and the possibilities made possible by it. I’ve written a good couple of clients, for fun and for serious production use at our company. In my mind, here is a “SWOT” on XMPP from someone who works with it frequently.

NOTE: I personally feel that most of the times we implement XMPP where we should be implementing a real queueing protocol. This holds especially true, if you like me, counts on ejabberd’s persisting of messages while the client is down.

For me personally the biggest smell in XMPP is the roster, at least for automated clients. This shows in other implementations too, Vertebra implemented a gatekeeper called Herault. I assume that with Herault in place you don’t care about roster management, and all bots will freely accept invitations from each other, and ask Herault whether the commands should be accepted or not. This is a perfectly sound plan, permitted your XMPP server is well secured and doesn’t allow inter-domain communications, ie you’re running inside a wall of trust.

The promised land of AMQP

I’m typing this without ever having written a single piece of AMQP code (be it producer or consumer). The “SWOT” below is what I’ve extracted from various online sources over the last couple of weeks, and especially today. I’ve got a number of requirements that needs to be fulfilled first:

• Peer to peer implementation of “bots” or “agents”
• Identity driven commands -> “Server 1: Create new site”, “Server 2: Suspend site”
• Unrestricted “bot” to “bot” communication
• File transfers (big files)
• No more roster management

Now the scary part is that all the above requirements can be fulfilled with XMPP. The implementation would smell however.

I hope the table above shows how blurred the lines are, and how difficult it can be to make a decision between the two protocols. Below is my non-textbook “SWOT” of AMQP.

In Summary

Both protocols are great, and the differences are not obvious at first. XMPP is easily mistaken as a message queue, especially because of the “offline chat” features in Jabber daemons and the inherently “targetted messages”. AMQP supports all of this, and so much more, without the need for roster management or Heraults. The only thing that I’ll miss from XMPP is the presence features. All my bots currently “talk to me and members of our support team”. I’ll implement this in my AMQP producers/consumers irrespective.

XMPP is built for one-on-one communication, with “broadcasting” supported by a protocol extension called PubSub. AMQP inherently supports one-on-one and one-to-many communication through it’s clever message/exchange system.

Going forward, I’m starting with the excellent nanite project from Ezra. I’ll probably work on bringing AMQP comsumers/producers into ruote this week, as well as implementing a nanite actor/master combo for ruote as well.

A great and timely post on RubyInside higlights RabbitMQ and some presentations to watch.

Please chime in on the comments, admittedly I might have buttered up AMQP due to my recent excitement, but I have a “gut feeling” it will payoff better in the future than XMPP. XMPP still rocks, I’ve just been using it as the wrong tool for the job, and I fear others have done the same.