Looking beyond daemon-kit 0.2

Works Ahead - Singapore Department of Transport

I wrote daemon-kit to solve two big issues with writing daemon processes in Ruby:

  1. Everyone is re-inventing the daemons gem
  2. Individual daemons share a lot common code, apart from the daemonizing bits

As for #1, daemon-kit at first wrapped the daemons gem, and later ripped it out completely as it was difficult to wrap up the worst of the daemons gem properly.

#2 seemed to be a twofold problem that daemon-kit has also addressed with great success. The first was addressing the all too common issues of logging, pid file management, umasks, signal traps, config files, exception handling and other “low-level” issues. Everyone was implementing these things to a limited extent in their daemons and this could cause a lot of frustration when done wrong. The second part was the need, even if undiscovered, for higher level re-use and development speed. Daemon-kit addresses this with some limited generators, making it easy to get going with a cron-style daemon, AMQP & XMPP bots as well as the newest addition, Ruote remote participants.

All this is fine and well, and people seem to like the project. The mailing list is getting some noise, we’re over 200 watchers strong on Github and the IRC channel has some folks popping in to say hi.

However, daemon-kit quickly made me lazy and realize there is a couple of things it can do much better. If you’ve used a generator before, you’ll notice the generated code is very much stuck to that type of daemon. Changing from XMPP to AMQP (for illustrative purposes) would be best accomplished by generating a new project and copying over the lib/ folder only. This sucks.

Another problem is people, myself included, would like to have Sinatra-style daemons (i.e. one file) for smaller tasks. Currently this is not possible at all. Another thing I know people are doing behind my back is generating daemons inside Rails projects, which may or may not work, depending on whether you load Rails’ environment.rb.

So, what happens now?

My thoughts are to implement privilege dropping support and tag a 0.2 release. This gives us a feature complete framework, albeit not as good as it can be. I’ll maintain 0.2 as a stable while undertaking the rewrite. A rewrite? Read on.

I posted to the daemon-kit list a suggestion for stackable daemon environments. I’ve discussed this in IRC with a few folks as well. Jordan Ritter gave me an exceptional breakdown of the dangers of doing something because it is “neat” and I’ve taken his warning to heart. However, I cannot seem to argue against stackable daemon environments, it sounds too good.

The idea is pretty simple. Stack-entries can be compared to Rack applications, with two significant differences. The first is that they will be called only once. They have the opportunity to change the environment in which the final code runs. The second is that they can be called at four different stages of the daemon lifecycle: argument processing, pre-daemonization, post-daemonization and shutdown. This differs from Rack’s single call() method.

The stackable nature also gives the stack members easier ways to set conventions and can dramatically minimize configuration. It paves the way for plugins, sinatra-style daemons, rails-based daemons and easier packaging of distributable daemons, and so much more. Looking at the internals of daemon-kit, it would greatly help simplify the existing code as well as help separate utility classes from stack members.

The more I think about it, the more obvious this becomes, and the more possibilities unfold for the framework. This will definitely make daemon-kit a force to be reckoned with, and hopefully I can persuade other library developers to offload their daemonizing code to daemon-kit, just like rack developers offload their HTTP handling to rack.

flattr this!

  • http://jmettraux.wordpress.com/ John Mettraux

    Hello Kenneth,

    excellent news :) Maybe the rack analogy is hard to live with, people (me the first) think run time but you mean configuration time… Sorry, can’t think of a better image/comparison to suggest.

    Keep up the great work !

  • http://blog.grayproductions.net/ James Edward Gray II

    I think a Sintra-esque DSL for building daemons sounds super fun.

    I also think trying things “because they are neat” is actually very healthy. What’s the worse that could happen? You could fail and realize it was a terrible idea that you wasted an insane amount of time on? Right! So what are you worried about? :D Seriously, I think playing around is good. It keeps the brain from getting all cobwebby.

    Random semi-unrelated thought: I saw another daemon library just the other day that would cause me fits. When I build a daemon I often need to build my own CLI interface. I don’t think that’s an unreasonable request. What if I want to be able to queue emails to me mail sender from the CLI, for example. Thus a library/framework that forces me into its own interface limits my ability to use it.

  • Kenneth Kalmer

    @John good point, I think I should almost dare call it “build” time since it does exactly that, build the environment for the daemonized code. I’ll still refine the terminology over time, feedback like this helps a lot to cement the concept.

    Thanks !

  • Kenneth Kalmer

    @James, I agree with you on trying stuff for learning. Jordan pointed out the dangers of re-implementing existing code in Ruby based purely on the neatness factor of doing it. Keeping the brain free of cobwebs is definitely a healthy exercise.

    As for the unrelated thought, your spot on. I’m hoping that daemon-kit will always be friendly to its users so they never end up fighting the framework. This is a big motivator for the rewrite, the easier it is to “build” the environments, the easier it will be to shape it to your exact requirements. It should be, like Jamis put it some time ago, like “play dough”.