Comments on: Safety nets for your Ruby daemons

By: JGeiger

JGeiger — Thu, 11 Mar 2010 18:54:20 +0000

I’m running into an issue with my daemons dying with the AMQP::Buffer::Overflow exceptions. I’m wondering if there is something I can use to test what might be causing it since it’s not an every time thing.

Backtrace:

/opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.7/lib/amqp/buffer.rb:252:in `_read’
/opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.7/lib/amqp/buffer.rb:95:in `read’
/opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.7/lib/amqp/buffer.rb:88:in `map’
/opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.7/lib/amqp/buffer.rb:88:in `read’
/opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.7/lib/amqp/buffer.rb:102:in `read’
/opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.7/lib/amqp/buffer.rb:88:in `map’
/opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.7/lib/amqp/buffer.rb:88:in `read’
/opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.7/lib/amqp/frame.rb:61:in `parse’
/opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.7/lib/amqp/buffer.rb:243:in `extract’
/opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.7/lib/amqp/frame.rb:60:in `parse’
/opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.7/lib/amqp/client.rb:115:in `receive_data’
/opt/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run_machine’
/opt/local/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run’
/opt/local/lib/ruby/gems/1.8/gems/amqp-0.6.7/lib/amqp.rb:79:in `start’
/opt/local/lib/ruby/gems/1.8/gems/daemon-kit-0.1.8pre/lib/daemon_kit/amqp.rb:35:in `run’
/opt/local/lib/ruby/gems/1.8/gems/daemon-kit-0.1.8pre/lib/daemon_kit/amqp.rb:20:in `run’

By: Kenneth Kalmer

Kenneth Kalmer — Mon, 25 May 2009 20:20:37 +0000

I’ve made notes about double checking log file & pid file configurations. Busy with capistrano integration right now…

Thanks for the great feedback so far!

By: Sho

Sho — Mon, 25 May 2009 18:57:58 +0000

Upon further reflection, I think that trying to write out-of-working-directory log files from a gem’d daemon is probably a waste of effort. I mean, you’d have to elevate permissions to write to /var anyway, then presumably drop permissions to actually run .. sounds like a lot of complexity for not much gain. I can’t think of any other library that does that. Mongrel, etc, rely on a naive approach where you either supply the log and pid locations as args or it just writes in the current WD.

As an unprivileged user you could write a pidfile into /tmp easily enough, but that doesn’t solve the logging problem. I suspect the “solution” would be that if you don’t supply a writable log file as an arg to the start command (or in config.yml), logging is disabled. But really, I’m not even sure that is worth the trouble. The whole thing is really designed to be run in place. Trying to make it gem-able just sounds like a recipe for complexity, fragility and disaster. Methinks deployment strategies, like you said, will revolve around cap et al rather than rubygems.

Meditate on this I will, but I suspect I’ve answered my own question …

By: Kenneth Kalmer

Kenneth Kalmer — Mon, 25 May 2009 14:28:40 +0000

@Sho glad it’s working out for you !

OK, I have to admit I haven’t tried that yet, so it’s virgin territory. How about slapping a ‘hello world’ daemon-kit/gem project on github with some intended use and the failures? That would give me a point to start from and figure out a clean fix to the problem.

By: Sho

Sho — Mon, 25 May 2009 14:23:25 +0000

Turning out great thanks! But I am running into issues with logging/writing PID files when running as a gem. Have you tried that yet? Any configuration guidance as to how to make DK write logs into a common area? Or is this virgin territory : )

By: Kenneth Kalmer

Kenneth Kalmer — Tue, 19 May 2009 19:53:28 +0000

@Sho – Glad it turned out good for you, looking forward to any feedback

By: Sho

Sho — Tue, 19 May 2009 17:48:44 +0000

Uh, scratch that. I was doing something wrong. Sorry for the noise. Seems to work great. Thanks, awesome!

By: Sho

Sho — Tue, 19 May 2009 16:23:59 +0000

Hm, upon setting out to start a new AMQP queue daemon I decided to try daemon-kit. I already have working code, I just need to daemonise it! So anyway I grabbed the latest and did daemon_kit testd -i amqp, then started it. Errors galore!

I don’t even know where to start with some of these errors, eg:

E, [2009-05-20T02:15:20.967626 #73649] ERROR — : AMQP::Buffer::Overflow (AMQP::Buffer::Overflow)

Uh-huh. Another few hundred lines follow, looks like some daemon-kit/AMQP interaction problem. Great. So .. does it work on your end? Maybe not the right approach for something I’d like to start today?

Looks great though, something like this is really needed!

By: Kenneth Kalmer

Kenneth Kalmer — Wed, 13 May 2009 19:56:46 +0000

Hey Matt

Thanks for the kind words. I’ll still improve the functionality a bit more, I could re-use the ‘request’ or ’session’ features of Hoptoad to provide even more context when the errors are logged, but we’ll see in due course what is required specifically by daemon processes…

By: Matt Jankowski

Matt Jankowski — Wed, 13 May 2009 15:57:59 +0000

Hey Ken — this is cool.

Exactly the sort of integration we had in mind for non-http-processing hoptoad error logging.