To AMQP or to XMPP, that is the question

Some time ago I started a series of blog posts [1,2] on Ruby and messaging. Most of the series is still in draft format and has to be completed. It’s a topic I’ve been very interested for some time, and have monitored the space quite closely looking for new implementations and collecting presentations and links as I go along. I’ve been a fan of Jabber (XMPP) for some time now, but it has some smells, to me at least…

First, some background. I’m the chief utopian at a South African ISP, inX, and most of my current work focusses on finishing off the fourth generation of our flagship project ISP in a Box. ISP in a Box is essentially a Rails frontend to a variety of services we procure and reprovision to resellers, which in turn sell to their clients. Apart from coping with a lot of models and extremely complex business processes, we also have to cope with procuring and provisioning services.

In this post I’ll aim to give some insight into our provisioning processes, and more specifically how I plan to implement our hosting provisioning over the coming days.

We exclusively use the excellent ruote Worflow Engine by John Mettraux to drive our procurement/provisioning processes, instead of overly complex state machines. As part of this I’ve implemented a Jabber Participant/Listener pair of classes for ruote, allowing any part of the workflow to be shelled out to a Jabber bot for processing, and continuing the workflow when done. This has worked great so far, but the requirements of our hosting services are somewhat more. This made me sit down and think, and I announced today that I’ll be doing a “pen and paper” SWOT on XMPP, AMQP and REST. What I came up with, is not a textbook SWOT, but at least a look at AMQP from an XMPP point of view. And it was interesting indeed, so much so that I’ve made up my mind to move to AMQP from XMPP. Here’s what happened…

The smells of XMPP

For the record, I really love XMPP as a protocol and the possibilities made possible by it. I’ve written a good couple of clients, for fun and for serious production use at our company. In my mind, here is a “SWOT” on XMPP from someone who works with it frequently.

Strengths Weaknesses
  • Presence – Know if your bots are running
  • Quick – Easy to get going (emerge ejabberd; gem install xmpp4-simple)
  • Queue like – ejabberd will store “offline chats”
  • Feedback and control via pidgin
  • Explicit identities (JID)
  • Roster – Managing relationships is a nightmare
  • Roster – High number of presence updates can be large overhead
  • PubSub – Feels like an afterthought to me
  • REST bridge (hence ratpack)
  • File transfers are an “extension” to protocol and daemons

NOTE: I personally feel that most of the times we implement XMPP where we should be implementing a real queueing protocol. This holds especially true, if you like me, counts on ejabberd’s persisting of messages while the client is down.

For me personally the biggest smell in XMPP is the roster, at least for automated clients. This shows in other implementations too, Vertebra implemented a gatekeeper called Herault. I assume that with Herault in place you don’t care about roster management, and all bots will freely accept invitations from each other, and ask Herault whether the commands should be accepted or not. This is a perfectly sound plan, permitted your XMPP server is well secured and doesn’t allow inter-domain communications, ie you’re running inside a wall of trust.

The promised land of AMQP

I’m typing this without ever having written a single piece of AMQP code (be it producer or consumer). The “SWOT” below is what I’ve extracted from various online sources over the last couple of weeks, and especially today. I’ve got a number of requirements that needs to be fulfilled first:

  • Peer to peer implementation of “bots” or “agents”
  • Identity driven commands -> “Server 1: Create new site”, “Server 2: Suspend site”
  • Unrestricted “bot” to “bot” communication
  • File transfers (big files)
  • No more roster management

Now the scary part is that all the above requirements can be fulfilled with XMPP. The implementation would smell however.

Requirement XMPP AMQP
Peer to peer communication Roster management (-1) Unique named queues (+1)
Identity driven commands Roster management (-1) Unique named queues (+1)
File transfers File transfer proxy configuration (-1) Supported on the wire (+1)
No roster management Implement copy of Vertebra’s Herault (-1) Unique named queues (+1)

I hope the table above shows how blurred the lines are, and how difficult it can be to make a decision between the two protocols. Below is my non-textbook “SWOT” of AMQP.

Strengths Weaknesses
  • No rosters to manage – Use queues
  • Security managed inside the broker (at least with RabbitMQ)
  • One to one (private) communication
  • One to many (fan out) communication
  • File transfer on the wire
  • REST interfaces in the making (RestMS)
  • Presence – Know if your producers/consumers are running
  • Explicit Identity – (JID)
  • Not so quick yet (RabbitMQ ebuild only came into portage today)

In Summary

Both protocols are great, and the differences are not obvious at first. XMPP is easily mistaken as a message queue, especially because of the “offline chat” features in Jabber daemons and the inherently “targetted messages”. AMQP supports all of this, and so much more, without the need for roster management or Heraults. The only thing that I’ll miss from XMPP is the presence features. All my bots currently “talk to me and members of our support team”. I’ll implement this in my AMQP producers/consumers irrespective.

XMPP is built for one-on-one communication, with “broadcasting” supported by a protocol extension called PubSub. AMQP inherently supports one-on-one and one-to-many communication through it’s clever message/exchange system.

Going forward, I’m starting with the excellent nanite project from Ezra. I’ll probably work on bringing AMQP comsumers/producers into ruote this week, as well as implementing a nanite actor/master combo for ruote as well.

A great and timely post on RubyInside higlights RabbitMQ and some presentations to watch.

Please chime in on the comments, admittedly I might have buttered up AMQP due to my recent excitement, but I have a “gut feeling” it will payoff better in the future than XMPP. XMPP still rocks, I’ve just been using it as the wrong tool for the job, and I fear others have done the same.

flattr this!

  • http://seanohalpin.github.com/ Sean O’Halpin

    It’s a case of horses for courses. While there is a lot of apparent overlap between XMPP and AMQP, they are really addressing different use-cases.

    I agree with your conclusion that XMPP is not suitable as a general purpose message queuing system. It has its own strengths though, particularly in its original domain of instant messaging (presence, rosters, federation, scaling, etc.). However, even though its PubSub protocol was first described in 1999 (http://xmpp.org/extensions/xep-0060.html), it has not seen much take up and in terms of implementation is still a poor cousin – hardly any end-user clients support it and even though server-side it is reasonably well-supported, there is no real consistency between implementations. Having said that, I still see XMPP PubSub as a viable way of delivering subscriptions over the net mainly due to the fact that some implementations can scale to many hundreds of thousands of simultaneous connections.

    My own experience of RabbitMQ so far has got me excited too: it’s the most flexible messaging platform I’ve ever used. AMQP provides you with all the primitives you need to construct any kind of messaging system. One of the downsides of that is that AMQP can come across as a bit complex (I wouldn’t say that XMPP is much simpler though). Well-designed client libraries (e.g. ruby-amqp) take care of a lot of the complexity for you.

    Also, what do you mean by ‘not so quick yet’? RabbitMQ is ~fast~.

  • http://melgray.org Mel Gray

    It should be noted that when using Ruby with either of these protocols you will have a far easier time scaling with AMQP.

    A typical XMPP session simply has too much going on at the same time for the ruby interpreter to handle (see: concurrency is a myth in Ruby). If you have a large roster, the number of presence and status updates add a significant amount of overhead.

    With AMQP you can bind multiple consumers to the same queue, so they can behave in a first come first serve fashion.

    I’ve tried using nanite and wasn’t really happy with it, because it seemed to do more than what I needed. I would recommend starting by either jumping head first into the AMQP gem or by using Lizzy to create small consumers.

  • Kenneth Kalmer

    @Sean – Thanks for the great feedback, regarding the “fast” I meant to get started, at least on gentoo. I’m a lazy developer, I want a simple “emerge -av1 rabbitmq && gem install tmm1-amqp” to get going. But as of yesterday, the RabbitMQ ebuild is available in gentoo’s portage, so my statement no longer holds true.

  • Kenneth Kalmer

    @Mel – Thanks for the insight on nanite. I need most of it’s functionality for my current use case, but I’ll be exploring the other options in due course as well.

  • http://foobarbazqux.blogspot.com Dimitar Dimitrov

    You might also want to check ZeroMQ low-latency messaging core, the RestMS spec and specifically the Zyre server (implementing RestMS using ZeroMQ). Looks like RestMS would give you almost everything you need.

    The projects are led by iMatix, creators of the OpenAMQ broker and participants in the AMQP consortium.

    Also, the ZMQ website has some quite interesting whitepaper if you are looking for comparison of messaging styles.

    [1] http://www.zeromq.org/
    [2] http://www.restms.org/
    [3] http://www.zyre.com/

  • http://www.restms.org Pieter Hintjens

    It would be interesting to see how you view RestMS (http://www.restms.org) as a messaging protocol that offers abstract message routing like AMQP, over RESTful HTTP.

  • Chris

    I’d like to know what all this AMQP, XMPP and workflow stuff is that you talk about and how I could use it with benefit to create applications, and what uses/advantages it has over my normal MVC approach to creating web applications.

  • Kenneth Kalmer

    @Pieter – RestMS is definitely on my rader, will get to that in a couple of days after I’ve familiarized myself with AMQP & nanite in general

  • Kenneth Kalmer

    @Chris – It extends the normal MVC approach (or any other approach) by offloading a) complicated business processes into a specialized workflow engine (see an older post I have on introducing Ruote in 20 minutes), and b) running long running tasks outside of the normal request-response cycle.

    For another take on XMPP/AMQP uses may I recommend you have a look at this excellent post by Ilya Grigorik http://www.igvita.com/2009/04/06/henry-ford-event-driven-architecture/

  • Kenneth Kalmer

    @Dimitar – Thanks for the links, RestMS is on my radar. As far as low-latency messaging goes, in our environment it wouldn’t make a substantial difference since the processes are long lived by nature (configuring DNS, configuring *nix servers for mail & hosting). However, I’m sure the need will arise, and I’ve bookmarked the resources.

  • Chris

    So you could say you’re following the WOP style ? Workflow Orientated Programming

  • https://stpeter.im/ stpeter

    Why do you need XMPP rosters in order to do plain old messaging? Rosters are an IM concept and not required for non-IM applications of XMPP.

  • Kenneth Kalmer

    @stpeter – As I interpret it, XMPP is IM, but thats beside the point. In my experiences with xmpp4r-simple you need to handle all messages coming in, presence updates, contact requests and normal messages. If you don’t you end up with processes unexpectedly consuming 100% CPU after some time. I can only speculate to the way xmpp4r-simple uses threading to queue the messages internally, and eventually this queue just blows up.

    If you’re blindly accept incoming friend requests, ie not care about the roster, you have to implement an application specific “whitelist” to only respond to certain “masters”. Or you have to manage the rosters directly and only use some other way to manage every bots relationships. Or even worse, you somehow disable rosters all together and circumvent one of XMPP’s core features, making it a set of glorified TCP connections…

    Ultimately, I’m not here to diss XMPP, although it seems like it. I’m dissing how I’ve been using it, and how many others have been using it. AMQP fits the bill perfectly, where XMPP needs some bending. This bending has been abstracted away by libraries like xmpp4r-simple and jabber-bot. I think the problems solved by XMPP are extremely interesting and can lead to some amazing projects in the future, by developers leveraging it’s strength better than I could. But for sending messages around between autonomous systems, AMQP just “feels better”.

  • Kenneth Kalmer

    @Chris – I guess so, maybe Event Driven Programming/Architecture sounds more impressive?

  • http://www.restms.org Pieter Hintjens

    @Kenneth,

    A very interesting thread, the first I’ve seen that dissects the different ways XMPP and AMQP work. One of the keys to AMQP is precisely that a publisher does not know (or care) who is at the other end of the routing & queuing system. The recipients for a message are abstracted into an address (routing key, headers).

    We’ve found in practice is that this works very nicely.

    But there are some cases where presence matters. For example, when requesting a particular service, it’s important to know if the service is absent (i.e. no subscribers for that particular address). AMQP does this using the same hack as email does (“if the message cannot be routed, send it back to me”).

    RestMS does it by equating a feed with a service, and letting clients test the presence / absence of a feed. Like AMQP, it hides recipients from senders behind an address abstraction.

    I suspect that XMPP is actually a higher-level protocol, which if we rolled the clock back some years, might be implementable on top of a generic messaging protocol like AMQP or RestMS.

  • Kenneth Kalmer

    @Pieter

    Great input, thanks!

    As far as I can tell nanite implements “presence”, I still have to dissect how they do it. I *assume* they use other queues where the agents push “pings” back to the “masters”… RestMS also looks promising, just gimme time to build up my AMQP skills from the ground up first :)

    In my environment I just require the broker to persist messages until the consumer comes back up again, so on the short term it isn’t a problem. But I can conjure up a few scenarios where presence does become important.

    As for “XMPP on top of AMQP”, I actually suggested to friend that they look into using a queuing protocol for building a “facebook chat” for another social network, instead of wrapping XMPP up in an overly complex BOSH/Comet scenario. This was several weeks ago, and now that you mention it, it might be a great way to implement chat…

  • http://jmettraux.wordpress.com John Mettraux

    @Chris

    the MVC model is fine, but sometimes the ‘work’ can’t simply be contained in a request/response cycle (generating a big pdf for instance). Pushing the work towards workers via a queue and replying immediately (“we’re working on it”) is a common scenario.

    Now a worker may be a workflow engine orchestrating more complex units of work among specialized workers (and it can leverage a queue system to communicate with those sub-workers).

    Ilya wrote someting related at http://www.igvita.com/2009/04/06/henry-ford-event-driven-architecture/ worth a read.

    Best regards,

  • http://www.opensourceconnections.com Scott Stults

    Whenever I see a new message queuing system, I always wonder why nobody chooses SMTP and NNTP. Those protocols have been around forever. They have solid, scalable implementations, support guaranteed delivery, and together support one-to-one and one-to-many transmission. The writers of SOAP realized this, but I’ve yet to see anyone use SMTP to deliver SOAP calls (and nobody wants to do SOAP nowadays anyway.)

    That said, I agree that ruote and XMPP are excellent, and I look forward to seeing how your AMQP implementation works out.

  • http://blog.thecapacity.org jay

    Good thoughts, and good luck. I read this article yesterday that maybe helpful with AMQP. It’s python but I don’t think that should stall anyone;

    http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/comment-page-19/#comment-2376

  • Kyle Schaffrick

    @Scott I think the secret sauce AMQP has that SMTP/NNTP lack is that the routing is configurable “on the wire,” in the sense that clients declare what exchanges, queues, and bindings they intend to use and how they relate to each other, as part of the ordinary course of using the broker’s services. These meta-level features are defined by the AMQP spec, both in protocol and semantics.

    On the other hand, a message queueing system implemented in SMTP/NNTP would have to A) define these meta-level features in an ad-hoc way and then implement them itself, separately from or as an extension to the actual broker/MTA, or B) give up this dynamic configurability, in which case the broker would have to know a priori (via configuration) what the distribution patterns should be.

    The fact that (for instance) RabbitMQ had essentially *no* static configuration files to edit (in stark contrast with most MTAs, to be sure) baffled me for a time until I realized that it worked this way. After some consideration, I believe it is one of the great strengths of AMQP’s approach.

  • http://singpolyma.net Stephen Paul Weber

    I think stpeter’s point was that you don’t even need a concept of friend requests in XMPP. You should be able to safely ignore those packets, or just in general not use them.

    XMPP is a stateful messaging protocol that happens to be really good at IM… it is not IM specific (although some XEP’s are)

  • Pingback: Driving Business Processes in Ruby | Open Sourcery

  • http://game.net.au Rj

    So, how have you gone with AMQP? Has your love affair persisted or are you back with XMPP?

  • Kenneth Kalmer

    Love affair has persisted. AMQP is part of my everyday work and has really nestled itself quite deeply into my toolbox. I still have my eyes on XMPP though, libraries like Blather makes XMPP a dream to work with. But like the post stresses, best tool for the job.

  • prashanth

    Wouldn’t it be better to use AMQP and XMPP for different purpose?
    Use AMQP for message queuing for background tasks and XMPP for real time publishing to webclients i.e instead of polling use BOSH along with XMPP?

  • Pingback: Advanced Messaging & Routing with AMQP

  • Pingback: Advanced Messaging & Routing with AMQP « Yet another wordpress blog

  • Pingback: AMQP, une alternative à JMS ? | Blog Xebia France