Open Sourcery

I wrote daemon-kit to solve two big issues with writing daemon processes in Ruby:

1. Everyone is re-inventing the daemons gem
2. Individual daemons share a lot common code, apart from the daemonizing bits

As for #1, daemon-kit at first wrapped the daemons gem, and later ripped it out completely as it was difficult to wrap up the worst of the daemons gem properly.

#2 seemed to be a twofold problem that daemon-kit has also addressed with great success. The first was addressing the all too common issues of logging, pid file management, umasks, signal traps, config files, exception handling and other “low-level” issues. Everyone was implementing these things to a limited extent in their daemons and this could cause a lot of frustration when done wrong. The second part was the need, even if undiscovered, for higher level re-use and development speed. Daemon-kit addresses this with some limited generators, making it easy to get going with a cron-style daemon, AMQP & XMPP bots as well as the newest addition, Ruote remote participants.

All this is fine and well, and people seem to like the project. The mailing list is getting some noise, we’re over 200 watchers strong on Github and the IRC channel has some folks popping in to say hi.

However, daemon-kit quickly made me lazy and realize there is a couple of things it can do much better. If you’ve used a generator before, you’ll notice the generated code is very much stuck to that type of daemon. Changing from XMPP to AMQP (for illustrative purposes) would be best accomplished by generating a new project and copying over the lib/ folder only. This sucks.

Another problem is people, myself included, would like to have Sinatra-style daemons (i.e. one file) for smaller tasks. Currently this is not possible at all. Another thing I know people are doing behind my back is generating daemons inside Rails projects, which may or may not work, depending on whether you load Rails’ environment.rb.

So, what happens now?

My thoughts are to implement privilege dropping support and tag a 0.2 release. This gives us a feature complete framework, albeit not as good as it can be. I’ll maintain 0.2 as a stable while undertaking the rewrite. A rewrite? Read on.

I posted to the daemon-kit list a suggestion for stackable daemon environments. I’ve discussed this in IRC with a few folks as well. Jordan Ritter gave me an exceptional breakdown of the dangers of doing something because it is “neat” and I’ve taken his warning to heart. However, I cannot seem to argue against stackable daemon environments, it sounds too good.

The idea is pretty simple. Stack-entries can be compared to Rack applications, with two significant differences. The first is that they will be called only once. They have the opportunity to change the environment in which the final code runs. The second is that they can be called at four different stages of the daemon lifecycle: argument processing, pre-daemonization, post-daemonization and shutdown. This differs from Rack’s single call() method.

The stackable nature also gives the stack members easier ways to set conventions and can dramatically minimize configuration. It paves the way for plugins, sinatra-style daemons, rails-based daemons and easier packaging of distributable daemons, and so much more. Looking at the internals of daemon-kit, it would greatly help simplify the existing code as well as help separate utility classes from stack members.

The more I think about it, the more obvious this becomes, and the more possibilities unfold for the framework. This will definitely make daemon-kit a force to be reckoned with, and hopefully I can persuade other library developers to offload their daemonizing code to daemon-kit, just like rack developers offload their HTTP handling to rack.

Undoubtedly my worst time on stage ever! Ubuntu gave me issues with the projector, so I actually had to leave the stage to go to another conference hall to get the setup working again and run back up to the main hall… Once there, I was a mumbling mess, but made it through without the buzzer beating me.

Presenting a lightning talk at a Japanese conference is surely something else, the audience expects fireworks and the presenters deliver. It was truly an amazing experience, humilation and all, and I’m looking forward to the last day of the conference.

Since I last announced Capistrano support for daemon-kit, a few other things have happened that is steadily moving the project forward. Here is a synopsis of them all:

No more daemons

Sounds contradictory, but I’ve stripped out the daemons gem as a depedency. Everything the daemons gem offered is now handled inhouse, and works much better. For more information please read the release announcement.

Better command line argument handling

A lot has been done to smooth out command line argument handling, and daemons can now tap into the command line processing very easily. Details in the release announcement and the RDoc.

Error mails now handled by TMail

Makes them look better and be better formed, the initial code didn’t really produce meaningful mails. A lot of work still needs to be done in this arena.

Log with vigor!

Logging got a lot of attention with the new AbstractLogger interface. Log rotation is now dead simple and logging to Syslog is possible thanks to the SysLogLogger gem. See the RDoc for more information on logging.

Cucumber

Still a lot needs to be done to simplify testing daemon processes, but the inclusion of preliminary cucumber support is one step in the right direction. I’m hoping cucumber will be a great fit for daemon-kit, and help us as daemon developers define the difficult contexts in which our code runs.

Looking forward

Plenty still needs to be done, the TODO file in the repo is some insight into what the future holds. Mostly daemon-kit evolves as I need it to for my production environments. On Github we’re close to 200 watchers, and the Google Group does have a little activity.

As time goes on I’m hoping more people get involved and bring suggestions to the table for the framework. So please, do tell me how daemon-kit can make your life simpler for your, and your daemons.

Nothing like some possitive feedback to keep you looking after your own projects, and the feedback on daemon-kit has been great so far. My TODO list is growing every day with more features that I need, and others need as well. I mostly add features to daemon-kit as I need them right away, for this reason the docs is still lacking and the support site hasn’t been setup yet.

However, I’ve setup the following channels for daemon-kit communication (and updated the README):

• daemon-kit@googlegroups.com
• #daemon-kit on Freenode

I’ll be doing my best to stay on top of these channels and garner more feedback from the community at large.

Today’s sprint was one of the very first things I planned, support for capistrano.

When generating a daemon, use the ‘-d’ flag to specify which deployer to use. Currently only capistrano is supported, but I’ve left the door open for anyone willing to add support for Vlad the Deployer.

$ daemon_kit /path/to/project -d capistrano

This will leave you with a Capfile, which you don’t need to edit, as well as config/deploy.rb and config/deploy/*.rb. The environment specific files in config/deploy is used by the capistrano-ext gem to support multistage deployments for testing your daemons. Shared configuration details go into config/deploy.rb. You need the following gems installed to leverage the deployment options:

• capistrano
• capistrano-ext

Capistrano is historically built around Rails applications, but I lifted the capistrano ‘deploy’ recipe from capistrano-2.5.5 and customized it to fit in with the requirements of daemons. Some steps have been removed, and some added. Here is the tasks available:

$ cap -T
cap deploy                     # Deploys your project.
cap deploy:check               # Test deployment dependencies.
cap deploy:cleanup             # Clean up old releases.
cap deploy:cold                # Deploys and starts a `cold' application.
cap deploy:copy_configs        # Copies any shared configuration files from :...
cap deploy:get_current_version # Get the current revision of the deployed code
cap deploy:pending             # Displays the commits since your last deploy.
cap deploy:pending:diff        # Displays the `diff' since your last deploy.
cap deploy:restart             # Restarts your application.
cap deploy:rollback            # Rolls back to a previous version and restarts.
cap deploy:rollback:code       # Rolls back to the previously deployed version.
cap deploy:setup               # Prepares one or more servers for deployment.
cap deploy:start               # Start the daemon processes.
cap deploy:stop                # Stop the daemon processes.
cap deploy:symlink             # Updates the symlink to the most recently dep...
cap deploy:update              # Copies your project and updates the symlink.
cap deploy:update_code         # Copies your project to the remote servers.
cap deploy:upload              # Copy files to the currently deployed version.
cap invoke                     # Invoke a single command on the remote servers.
cap multistage:prepare         # Stub out the staging config files.
cap production                 # Set the target stage to `production'.
cap shell                      # Begin an interactive Capistrano session.
cap staging                    # Set the target stage to `staging'.

Very close to the original recipes. I’ve still got to clean this up even further, and add some documentation on how to leverage capistrano for daemons, but for the time being this works very well. The recipe can be found in lib/daemon_kit/deployment/capistrano.rb if you have the urge to review it.

Heading towards 0.2, here are the main highlights:

• god & monit config file generation
• Improved rdoc’s and supplementary documentation files
• Finishing off some issues with error handlers
• Better logging