Safety nets for your Ruby daemons
by Kenneth Kalmer on May 13, 2009
Daemon-kit has been getting a lot of TLC from me lately, and it’s not going to stop anytime soon. As I wander deeper and deeper into AMQP territory, I need to extend daemon-kit to cope with all kinds of new scenarios. One of those being unhandled exceptions.
The second thing I put on the TODO list was Rails-style exception handling. With version 0.1.6 there has been some progress made in that regard. Daemon-kit now sports a configurable safety net for dangerous code. By wrapping blocks of code in a “safety net”, unhandled exceptions are caught and logged, and optionally sent via email or to Hoptoad for review.
Hoptoad? In a Ruby daemon? Sure, inspiration came via these tweets.
And it only makes sense to do it. Now for some code:
safely do # do something silly silly.action! end
safely is mixed into Object and can be used freely. It is important to note that you have to handle your daemon-specific applications on your own and rely on safely as a fall over mechanism.
To configure the safety net, you can edit your config/environment.rb file and add the following lines to the configure block:
# for email notifications config.safety_net.handler = :mail config.safety_net.mail.recipients = ['you@gmail.com'] # for hoptoad config.safety_net.handler = :hoptoad config.safety_net.hoptoad.api_key = 'your-hoptoad-key'
The documentation is very rough at the moment, but the files you want to explore are lib/daemon_kit/safety.rb and the error handlers in lib/daemon_kit/error_handlers.
NOTE: If you are upgrading from an earlier daemon-kit, please upgrade your daemons as well by running the following rake task in the root of your daemon projects:
$ rake daemon_kit:upgrade
In the coming days/weeks you can look forward to the following enhancements as well:
- Improved logging
- Improved backtrace cleanups
- Improved rdoc’s
- rack application generator (with rack-mount)
I’m patching things up as I go along, adding features as I need them (and stuff I recall from my first daemons). There is still a lot of things that need attention, but they’ll be addressed and hopefully daemon-kit grows to becoming the premier framework for writing daemon processes in our beloved Ruby.
10 comments
Hey Ken — this is cool.
Exactly the sort of integration we had in mind for non-http-processing hoptoad error logging.
by Matt Jankowski on May 13, 2009 at 5:57 pm. #
Hey Matt
Thanks for the kind words. I’ll still improve the functionality a bit more, I could re-use the ‘request’ or ‘session’ features of Hoptoad to provide even more context when the errors are logged, but we’ll see in due course what is required specifically by daemon processes…
by Kenneth Kalmer on May 13, 2009 at 9:56 pm. #
Hm, upon setting out to start a new AMQP queue daemon I decided to try daemon-kit. I already have working code, I just need to daemonise it! So anyway I grabbed the latest and did daemon_kit testd -i amqp, then started it. Errors galore!
I don’t even know where to start with some of these errors, eg:
E, [2009-05-20T02:15:20.967626 #73649] ERROR — : AMQP::Buffer::Overflow (AMQP::Buffer::Overflow)
Uh-huh. Another few hundred lines follow, looks like some daemon-kit/AMQP interaction problem. Great. So .. does it work on your end? Maybe not the right approach for something I’d like to start today?
Looks great though, something like this is really needed!
by Sho on May 19, 2009 at 6:23 pm. #
Uh, scratch that. I was doing something wrong. Sorry for the noise. Seems to work great. Thanks, awesome!
by Sho on May 19, 2009 at 7:48 pm. #
@Sho – Glad it turned out good for you, looking forward to any feedback
by Kenneth Kalmer on May 19, 2009 at 9:53 pm. #
Turning out great thanks! But I am running into issues with logging/writing PID files when running as a gem. Have you tried that yet? Any configuration guidance as to how to make DK write logs into a common area? Or is this virgin territory : )
by Sho on May 25, 2009 at 4:23 pm. #
@Sho glad it’s working out for you !
OK, I have to admit I haven’t tried that yet, so it’s virgin territory. How about slapping a ‘hello world’ daemon-kit/gem project on github with some intended use and the failures? That would give me a point to start from and figure out a clean fix to the problem.
by Kenneth Kalmer on May 25, 2009 at 4:28 pm. #
Upon further reflection, I think that trying to write out-of-working-directory log files from a gem’d daemon is probably a waste of effort. I mean, you’d have to elevate permissions to write to /var anyway, then presumably drop permissions to actually run .. sounds like a lot of complexity for not much gain. I can’t think of any other library that does that. Mongrel, etc, rely on a naive approach where you either supply the log and pid locations as args or it just writes in the current WD.
As an unprivileged user you could write a pidfile into /tmp easily enough, but that doesn’t solve the logging problem. I suspect the “solution” would be that if you don’t supply a writable log file as an arg to the start command (or in config.yml), logging is disabled. But really, I’m not even sure that is worth the trouble. The whole thing is really designed to be run in place. Trying to make it gem-able just sounds like a recipe for complexity, fragility and disaster. Methinks deployment strategies, like you said, will revolve around cap et al rather than rubygems.
Meditate on this I will, but I suspect I’ve answered my own question …
by Sho on May 25, 2009 at 8:57 pm. #
I’ve made notes about double checking log file & pid file configurations. Busy with capistrano integration right now…
Thanks for the great feedback so far!
by Kenneth Kalmer on May 25, 2009 at 10:20 pm. #
I’m running into an issue with my daemons dying with the AMQP::Buffer::Overflow exceptions. I’m wondering if there is something I can use to test what might be causing it since it’s not an every time thing.
Backtrace:
/opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.7/lib/amqp/buffer.rb:252:in `_read’
/opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.7/lib/amqp/buffer.rb:95:in `read’
/opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.7/lib/amqp/buffer.rb:88:in `map’
/opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.7/lib/amqp/buffer.rb:88:in `read’
/opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.7/lib/amqp/buffer.rb:102:in `read’
/opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.7/lib/amqp/buffer.rb:88:in `map’
/opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.7/lib/amqp/buffer.rb:88:in `read’
/opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.7/lib/amqp/frame.rb:61:in `parse’
/opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.7/lib/amqp/buffer.rb:243:in `extract’
/opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.7/lib/amqp/frame.rb:60:in `parse’
/opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.7/lib/amqp/client.rb:115:in `receive_data’
/opt/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run_machine’
/opt/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run’
/opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.7/lib/amqp.rb:79:in `start’
/opt/local/lib/ruby/gems/1.8/gems/daemon-kit-0.1.8pre/lib/daemon_kit/amqp.rb:35:in `run’
/opt/local/lib/ruby/gems/1.8/gems/daemon-kit-0.1.8pre/lib/daemon_kit/amqp.rb:20:in `run’
by JGeiger on March 11, 2010 at 8:54 pm. #