First, some background. I’m the chief utopian at a South African ISP, inX, and most of my current work focusses on finishing off the fourth generation of our flagship project ISP in a Box. ISP in a Box is essentially a Rails frontend to a variety of services we procure and reprovision to resellers, which in turn sell to their clients. Apart from coping with a lot of models and extremely complex business processes, we also have to cope with procuring and provisioning services.

In this post I’ll aim to give some insight into our provisioning processes, and more specifically how I plan to implement our hosting provisioning over the coming days.

We exclusively use the excellent ruote Worflow Engine by John Mettraux to drive our procurement/provisioning processes, instead of overly complex state machines. As part of this I’ve implemented a Jabber Participant/Listener pair of classes for ruote, allowing any part of the workflow to be shelled out to a Jabber bot for processing, and continuing the workflow when done. This has worked great so far, but the requirements of our hosting services are somewhat more. This made me sit down and think, and I announced today that I’ll be doing a “pen and paper” SWOT on XMPP, AMQP and REST. What I came up with, is not a textbook SWOT, but at least a look at AMQP from an XMPP point of view. And it was interesting indeed, so much so that I’ve made up my mind to move to AMQP from XMPP. Here’s what happened…

The smells of XMPP

For the record, I really love XMPP as a protocol and the possibilities made possible by it. I’ve written a good couple of clients, for fun and for serious production use at our company. In my mind, here is a “SWOT” on XMPP from someone who works with it frequently.

NOTE: I personally feel that most of the times we implement XMPP where we should be implementing a real queueing protocol. This holds especially true, if you like me, counts on ejabberd’s persisting of messages while the client is down.

For me personally the biggest smell in XMPP is the roster, at least for automated clients. This shows in other implementations too, Vertebra implemented a gatekeeper called Herault. I assume that with Herault in place you don’t care about roster management, and all bots will freely accept invitations from each other, and ask Herault whether the commands should be accepted or not. This is a perfectly sound plan, permitted your XMPP server is well secured and doesn’t allow inter-domain communications, ie you’re running inside a wall of trust.

The promised land of AMQP

I’m typing this without ever having written a single piece of AMQP code (be it producer or consumer). The “SWOT” below is what I’ve extracted from various online sources over the last couple of weeks, and especially today. I’ve got a number of requirements that needs to be fulfilled first:

• Peer to peer implementation of “bots” or “agents”
• Identity driven commands -> “Server 1: Create new site”, “Server 2: Suspend site”
• Unrestricted “bot” to “bot” communication
• File transfers (big files)
• No more roster management

Now the scary part is that all the above requirements can be fulfilled with XMPP. The implementation would smell however.

I hope the table above shows how blurred the lines are, and how difficult it can be to make a decision between the two protocols. Below is my non-textbook “SWOT” of AMQP.

In Summary

Both protocols are great, and the differences are not obvious at first. XMPP is easily mistaken as a message queue, especially because of the “offline chat” features in Jabber daemons and the inherently “targetted messages”. AMQP supports all of this, and so much more, without the need for roster management or Heraults. The only thing that I’ll miss from XMPP is the presence features. All my bots currently “talk to me and members of our support team”. I’ll implement this in my AMQP producers/consumers irrespective.

XMPP is built for one-on-one communication, with “broadcasting” supported by a protocol extension called PubSub. AMQP inherently supports one-on-one and one-to-many communication through it’s clever message/exchange system.

Going forward, I’m starting with the excellent nanite project from Ezra. I’ll probably work on bringing AMQP comsumers/producers into ruote this week, as well as implementing a nanite actor/master combo for ruote as well.

A great and timely post on RubyInside higlights RabbitMQ and some presentations to watch.

Please chime in on the comments, admittedly I might have buttered up AMQP due to my recent excitement, but I have a “gut feeling” it will payoff better in the future than XMPP. XMPP still rocks, I’ve just been using it as the wrong tool for the job, and I fear others have done the same.

Like all good bloggers out there, it seems plenty of days between episodes in a series is quite reasonable. Nevermind, thats just wrong. Anyone following this blog would have noticed the interesting rise and fall of BIND DLZ on Rails, and its successor PowerDNS on Rails. This does seem to take a good chunk out of valuable writing time, especially when the likes of John Mettraux are waiting for the next installments… So without further adue, here is Ruby and Messaging (Part 2) – Giving Rails a Break

Introduction and ramblings

Undoubtedly the topic of messaging in Rails has generated a lot of interest lately, and I’m sad that I missed my chance to be ahead of the curve in terms of coverage. Why is messaging such a hot topic? Well, Ruby (Rails and other frameworks) allows you to build extremely popular and powerful sites in record time. This does however bring some unforeseen problems, scaling being number one.

Oh no, not another scaling post! On the contrary, I’d like to highlight how developers hit stardom without ever realizing what it takes. The folks over at Google, Yahoo! and other popular properties had years to figure out how to build reliable, distributed and scalable systems. Amazon is one of the stars of the new Cloud Computing genre, with folks like EngineYard following hot on their tales. These cloud computing providers capitalize on the fact that everyday developers don’t know a thing about scaling systems. A timely post on the Google Blog proves this quite well:

Back then, we did everything in batches: one workstation could compute the PageRank graph on 26 million pages in a couple of hours, and that set of pages would be used as Google’s index for a fixed period of time.

They processed 26 million records in a couple of hours on a workstation. I doubt that workstation compared to even a small AMI. See my point?

Now before we get to batches and distributed systems, lets take a step back and look at the simplest of Ruby messaging systems: the ones that help Rails “scale”. I’ll just be touching on them briefly, you’ll have to consult Google for more information and real examples.

Giving Rails a break

Now some purists might flame me for this, since it covers messaging at a conceptual level. The real implementations cannot be seen as pure messaging, but I believe in crawling before you walk (it’s just how nature intended).

So how why does Rails need a break? Some tasks just take too long. Bold, I know. “Too long” is extremely relative, considering Flickr keeps there page generation times well below 250 miliseconds. In that case, 300 milliseconds is a 20% increase in processing time! That, my dear reader, is taking “too long”.

I’ll be focusing the remainder of the post on methods that resemble messaging systems, not on ways to make Ruby any faster, or bypass expensive ActiveRecord operations. Other people have covered those topics already, and this is a build up to some more serious articles on messaging, not scaling.

In process messaging

I’ll be using the classical sending email is slow example, since its relevant to almost every single Rails application in existence.

Let me take a step back and define “in process messaging” as I see it. In process messaging is the act of sending an instruction from one part of a running application, to another. This may, or may not, involve the use of threading, but it remains inside a single process. This differs from inter-process (and mostly asynchronous) messaging, which involves message passing between different running systems.

Back to sending email. It’s slow because a mail server is a different beast all together. It takes more than 250 ms for a mail to get queued in a SMTP server, especially if the mail server is not on the same server as the Rails application. And besides, why can’t we run that “in the background” and tell the user the message has been sent? This means the user gets the confirmation page before Rails is even done with sending the message.

Using something like PHP would make this a snap. You’ll generate your confirmation message to the user, flush PHP’s buffers and ask Apache to close off the request. Then you’ll send the email afterwards, out of sight and out of mind. Rails doesn’t sport this option, but all is not lost.

Enter two plugins that caught my eye.

Spawn

Though it can run in fork mode (making it asynchronous), it does support threading mode (making it in process).

A simple code example looks like this:

spawn do
  Mailer.deliver_registration_confirmation( @subscriber )
end

I’ve had the pleasure to use spawn on a couple of projects and it works very nicely. It’s especially rewarding to watch your log files and see the spawned process took a couple of seconds and you know that you didn’t keep the request on hold for that time.

You can also use spawn outside of Rails with minimal effort. Open up spwan.rb and have a look for yourself, it is a beautful little file that is very good at doing one thing.

background

The aptly named “background” plugin also does a great job of handling these long running tasks. I’ve used this one in an application or two and love it. It also covers various operation modes, but at its simplest it will also run in process.

background do
  Mailer.deliver_registration_confirmation( @subscriber )
end

It’s as simple as that. It offers slightly more than spawn, but I’ll cover more advanced topics later in the series. It also supports out of process messaging, but thats covered next…

Inter-process messaging

Not to move on too quickly, but inter-process processing follows the same lines as the in-process examples. I’m using inter-process messaging loosely here to refer to Ruby-to-Ruby communication within the same project, and where you as the developer control both ends of the conversation.The later parts of the series will address different queuing implementations and protocols for performing tasks larger than just sending mail.

A player I really want to mention here is BackgroundRB. It steps things up a bit by running your workers outside of the Rails process. Jobs are queued & scheduled and handled by worker processes. It seems tightly coupled with Rails, so I don’t know how easy it would be to use it outside of Rails.

Using Distributed Ruby, or DRb, is another great way to have asynchronous processing. DRb is more of a roll-your-own solution, but really helps if you’ve planned properly and understand the DRb implementation. DRb uses its own format to pass messages around between servers. Did I just mention servers, and not client-server? Yes. In the DRb world the client is also a server and is asked to do processing the “server” can’t. This has bitten me on the arse before, and thats why I said you need proper planning. If you pass complicated objects over the wire, the processing will be directed back to the client, effectively rendering DRb useless. I’ll cover this at some other time.

Another approach I’ve used where the communication needs to be synchronous, but between different processes is to build my own REST services. I’ve favored Halcyon so far, for no other reason than getting familiar with JSON. You can use your imagination when you control both ends of the conversation, but keep your messages in a format thats preferably standard and open, like XML/JSON.

Synchronous and asynchronous processing

Just to avoid confusion I need to make this clear. No matter how your messages travel, be it in-process or inter-process, it doesn’t make it asynchronous. My examples of using the spawn and background allow for asynchronous processing using variables as in-process messages (or instructions). Furthermore the REST API example does the opposite, it uses inter-process messaging as part of a synchronous process.

Keep the distinction clear in your design. Messaging is a form of communication and not of processing. Various people use the terms almost interchangeably and this makes it seemingly difficult to grasp the concepts correctly.

Using a clear, concise and open messaging implementation will build you applications that can grow almost indefinitely. Look at the web as a prime example of this, with billions simple HTTP messages getting passed around every couple of minutes.

Looking forward

There are definitely better approaches available to us, including message queuing, broadcasting and message federation, to name but a few. The next parts of the series will cover these topics in more details. Looking back at the first part of the series I made a proposition that we model our systems after existing ecosystems found in nature. In such a ecosystem each component can live on it’s own, evolve separate and flourish.

The topics discussed in the post doesn’t really encourage the building of an ecosystem, but rather an abomination if used incorrectly. Its from my own experiences of dealing with these abominations that I’ve modeled my ecosystems on nature and continue to see them flourish. These approaches are also tied to Rails, with the exception of DRb and REST, and create a lock-in of sorts. Not that I don’t like Rails, I love having options even more.

The next two articles will focus on using Jabber/XMPP and AMQP, and all their friends. Two very powerful technologies that compete head on for developer mindshare, although I sometimes believe the mindshare should be 50/50. After that I might change the direction to practical uses and implementations of messaging (ruote comes to mind).

NOTE: This post has been in draft for a couple of months now, and might seem a bit dated because of that. I need to complete the series and have decided to publish it for a sense of completeness.

Now that I have your attention, lets forget scaling all together and instead start focusing on how you can give Rails a break using messaging or ‘backgrounding’. This is the first in a series of articles where I’m combining my own research and experience together to a) prevent me from making the same mistakes in future, and b) hopefully help someone else to not make the same mistakes I did.

Prelude

Ever since I’ve read Distributed programming with Jabber and EventMachine I’ve been baffled by the almost endless possibilities this offers. I’ve shared this knowledge with several people, who got excited too, but not as excited as I did. Then I read about Vertebra on Ezra’s blog and almost lost my mind, shared this knowledge again and got a more enthusiastic response. Since then I’ve been scouring the net for more information and examples on using Jabber for distributed programming, as well as alternatives, since something was not entirely right in the equation.

Up until now I’ve used some insane combinations of Distributed Ruby, SQS, “Roll your own ActiveRecord queue”, REST and who even knows what else to achieve some levels of ’scalability’ and ‘asynchronous processing’.

Each project I work on have varying demands, from being ready to survive a Slashdotting all the way down to integrating with several desperate systems. Each one has its own expectations in terms of processes, business rules, and reporting on “background tasks”. It is quite a colorful affair I assure you. Now on each iteration I try something else, something new, something better. And they do get better, so I’m proud of the work I’ve done, and even more proud of my team members keeping up with the frantic evolution of the code everywhere.

However, this roll-your-own messaging systems are not scalable or reliable. You need the wisdom of crowds at your disposal. You need to draw on the knowledge of guys who’ve built systems for Fortune 500’s, built systems for telco’s, built systems for financial markets, built systems for medical care, built systems for aviation, etc.

Why? These guys have faced obstacles we can never imagine ,and they’ve overcome them. Through the years their knowledge has become available to us, and we must apply it. I’m sure if Twitter thought like a telco, and built Twitter the way a telco would, it would have never broken a sweat handling all those tweats.  But I’m not one to criticise, on with the story.

What is messaging, and why is it important?

Good question! Wikipedia sums it up like this:

There are two main senses of the word “message” in computer science: messages passed within software, which may or may not be human-readable, and human-readable messages delivered via computer software for person-to-person communication.

We’ll be focussing on the first sense, “messages passed within software”.

Why is messaging important?

It allows us to build systems that have the following characteristics:

1. Snappy websites that don’t block while performing expensive operations (sending mail, resizing photo’s, encoding videos, etc).
2. Horizontal scalability, just throw in more instances of your program to speed up processing
3. Loosely coupled components that merely ‘talk’ to each other
4. Ability to easily replace Ruby with something else for specific tasks

By passing simple messages around between different processes, or systems, we can build an ecosystem from smaller, more specialized components and then allow users to interact with the ecosystem as whole, as opposed to a monolithic compound.

Nature has proven over and over again how effective ecosystems are at surviving by evolving. Having loosely coupled components that exist independently, but talk to each other via messages, allows you to build extremely powerful systems, that would by virtue, scale indefinitely and evolve constantly.

Awesome! How do I do it?

It’s quite simple, you need this gem, that app, several servers, big dollars. You first need to understand a couple of concepts. This is very important, and this was my biggest mistake. There is no shortage of ways to perform distributed programming or to run expensive operations asynchronously, and this list is not even a chip off the iceberg.

In my experience as a Rails and Ruby developer, I’ve dabbled in a few different ways to solve the asynchronous and distributed programming problems, each with their own merits (and demerits). Each of these problems and proposed solutions are covered this series, so I’m only gonna touch on them briefly now.

Asynchronous processing in Ruby on Rails

Here I’m specifically referring to expensive tasks that ‘lock up the browser’ without any real reason at all. I can probably build a list from here to the moon based on the experiences of others and myself, but I won’t. Simple examples will suffice, and these include sending mails, resizing images, interacting with other services (RSS, REST) and cron-like processing. Just like I said in the opening statements, the expectations on today’s websites are enormous…

Distributed processing with Ruby

Leveraging cloud computing to perform an array of expensive or time consuming tasks. This is a wide open description, but I’d like to restrict the scope to tasks like video encoding, photo manipulation, sending newsletters to thousands of users, crawler activities, and other general tasks related to servicing millions of users. I’m not referring to cracking the human genome, or even solving the Rubik’s cube in the smallest number of steps. Tasks of those caliber require something much bigger than what my mind can conjure up.

Cron-like processing of tasks

The newsletter example above will fit in here too, since we need to determine at a specific time who will receive which newsletters. The trigger to some crawling activities might also be scheduled. I’ll make the article is more clear, I’m a but stumped now…

Workflow-like systems

In this artcle I’ll probably get crucified for not discussing awesome libraries like ruote and other BPM engines, but will instead focus on using messaging to drive sequential processes through loosely coupled components, building the ecosystem example used earlier in this article.

For the impatient and the eager

I’ll be discussing examples built on top of DRb, SQS, Jabber (XMPP), Advanced Messaging and Queueing Protocol (AMQP) and others. I’ll also be looking at various clients, servers and services that help facilitate message sending, consumption and brokering with Ruby.

Most important of all is not the technology it self, but the lessons learned from using it. These are lessons I learned while getting baptized by fire, or learned from others who show the same scars. I want to help others to not have the same scars, and instead focus on the next generation of problems so I can learn from them again.

Looking forward

At the time of writing I have the other articles in some combination of notes, taboo’s, del.icio.us-ness and draft posts, and am looking forward to completing them eagerly for publishing, hence this appetizer.

Please add the feed to your reader of choice so you can grab them as they are released. I’d also appreciate some pre-emptive feedback and pointers from people who’ve wondered down the same path so I can include more info in the coming articles.