Open Sourcery

CouchDB is hands down my favorite of the NoSQL variants and offers some pretty spectacular features, none of which I will bore you with in this post. I will however jot down how I (fairly easily) achieved pagination with couchrest & will_paginate in a fairly large Rails application recently.

John P Wood discussed some issues they faced with will_paginate and couchrest during the migration of TextMe to CouchDB, but l left out some code to work with. Couchrest itself had some pagination support that got pulled to some extent… This left me wanting, and wondering, since it was my turn to walk down this path.

CouchDB is a different beast, its aggressive use of indexes means that occasionally you loose some functionality that you’ve been accustomed to having in other persistence mechanisms, like the number of rows matching a query. Jan Lenhardt explains on the CouchDB issue tracker in more detail, but it boils down that you need a reduce function to calculate the number of rows. Sounds difficult? Not at all!

In my case I had a collection of announcements to deal with, and the announcement archives is a paginating collection of documents. Standard will_paginate stuff, nothing special.

Those reading in a reader would want to click through to the post to view the embedded gists further down, or view them directly at Github.

Below is a condensed version of the model from our paginating system:

I’ve included only one view and a corresponding class method, as it is enough to proof the principle. Lets dissect.

The map/reduce functions are extremely simplistic, they simply emit the announcer and the date the announcement was created. This allows for easy scoping and ordering of the announcements. The reduce simply counts our returned records. The magic is in the class method that sets up our WillPaginate::Collection with data from our views.

Line 30 creates a new WillPaginate::Collection instance, passing it the page number and total per page as parameters, it gives us back a pager that we can manipulate.

Lines 31 through 38 uses couchrest’s pagination support to pull out data from our view. The most important things to note here are that the page and per_page options are sent to the paginator and we skip the reduce step.

Once we have our records loaded, we ‘replace’  the pager’s collection with our results from our view (line 40).

The final step is to determine the total number of documents available to us, and for this we need the reduce function. On line 42 we call the same view, with the same arguments, except for requiring the reduce step to happen. We use the results (lines 43 to 47) to inform the pager (will_paginate’s pager) how many rows there are in total.

The controller and the views might look something like this:

It worked, and it shows that we don’t loose as much as we might think when moving away from ActiveRecord and the ton of plugins surrounding it.

Thanks to John for documenting the migration of TextMe, just knowing that it was possible to combine couchrest & will_paginate gave me the push I needed to figure this out.

Disclaimer: This code is extracted “as is” from a real life system and might contain idioms/phrases, and even code, that doesn’t make 100% when viewed in a gist. Please wear your thinking cap when applying this lesson to your own projects.

I’m in the very fortunate position, two actually, of being able to 1) migrate my biggest production application from MySQL to CouchDB, and 2) build a stunning new system for a multinational welfare organization on top of CouchDB.

I’ve been lurking in the CouchDB world for quite some time and have spent a lot of time wrestling with how to loosely relate documents in CouchDB to each other. A big part of learning to use CouchDB successfully is to break away from the shackles of the relational world. Relationships between documents is one such a shackle that seems hard to break.

It is unavoidable that data has to be correlated, and I wanted to rethink how to do it. After plenty of discussions in #ruote with John Mettraux we came up with a model based on how the web works. Since CouchDB is “off the web”, the approach feels quite fitting to me and hopefully to you too.

First some insight into my thinking at this stage.

The web has been successful in loosely expressing relationships between documents. Take two examples:

and

For those of you reading this through a reader, click through to see the gist’s above.

Simple as it seems, in both cases we have a document that is somehow related to the page. The nature of the relationship is expressed via the rel attribute, and the target specified via the href attribute. This got me thinking. Since CouchDB is made off the web, can’t these same principles be applied?

Yes, they can. And here is how:

Currently you might be tempted to express relationships link this in your JSON:

Where changing it to this holds the key:

If anything this format, albeit more verbose, expresses the relationships more clearly and in a format that is web friendly. We’ve broken the shackles of relational thinking.

Enter Correlate

Correlate is an experiment in this line of thinking. It is a mixin for CouchRest’s extended documents that allows you to express these relationships:

Correlate generates getter and setter methods for working with your relationships and lot more (review the README). It also includes a compatibility layer for ActiveRecord to help when you’re migrating from ActiveRecord to CouchRest or building a system on CouchDB that needs to access legacy data via ActiveRecord.

The project is still pretty much a moving target, but I’d love to hear how others address the same issues. Correlate does a great job at maintaining relationship information in a web friendly manner and providing you with some convenience around the verbose data structure. Correlate also has a lot of room for improvement, but that will hopefully change over the coming days as I continue integrating it deeply into my existing projects.

Please fork the project on github and join the experiment with me.

Handsoap is a new, fresh Ruby library for creating SOAP clients. Why am I excited about soap? I’m not, unless they’re beautiful. Yet, somewhere along the line you are going to be faced with writing a client for a SOAP service. Whether it is from your suppliers (4 different crap API’s in my case), or just for mashing up data from a rest-less source.

Ruby has very little to offer in terms of SOAP, afaik it only had soap4r until handsoap came along…

BIG DISCLAIMER: This is not a flame war against soap4r, it’s against SOAP.

The authors and contributors have spent a lot of time and effort over the years to build soap4r up to where it is currently. It is a remarkable piece of work, but the cracks are showing. I’ve spent countless hours digging through the code trying to figure out why my mappings and registeries aren’t working as expected. You find code working around bugs in Ruby 1.8.2 and 1.8.4, and we’re all trying to move to 1.9. That is a lot of skeletons to hide…

It’s not all doom for soap4r, I’ve used it to play around with some public API’s for weather and currency data, and if the API is well defined and leverages SOAP as a protocol, soap4r really rocks. The results of wsdl2ruby is bit tricky to navigate, but mostly you can leave that alone and focus on your code.

But, and there always has to be a but…

A lot of API’s are written by people who have no idea what it is like to be on the consuming side of their mess. This is where soap4r falls flat on its face. Take the case of Postini and the postini gem for instance. The gem was my first gem ever and as far as I’m considered it was my worst code ever (open source at least). It was an absolute mess, barely worked more than what our systems required, and was not expandable in any way. The root of the problem was not soap4r, but the worst combination of WSDL and PDF docs to land on any developers screen. The documentation and the WSDL don’t match, not by a long shot, and you have to debug minor issues with wiredumps. I’ll stop my rant now.

Enter handsoap

As development of our premier product comes closer to launch, our email systems came under scrutiny and I needed to update the way we interact with Postini through our gem. I couldn’t, I could barely figure out what I did in the first place. If there was any proper use of the gem outside our company, I would have probably been summoned to criminal court for the injustice I had done in the first release.

So I started over, and I learned about handsoap a couple of weeks ago already.

This was the result in 24 hours:

$ git log --stat 2f91d4d6eaa8bd76c188f20929a19e456d1bb52e..HEAD | grep changed

2 files changed, 15 insertions(+), 4 deletions(-)
 1 files changed, 38 insertions(+), 0 deletions(-)
 2 files changed, 282 insertions(+), 1 deletions(-)
 1 files changed, 1 insertions(+), 1 deletions(-)
 12 files changed, 9 insertions(+), 682 deletions(-)
 1 files changed, 1 insertions(+), 0 deletions(-)
 8 files changed, 184 insertions(+), 3883 deletions(-)
 6 files changed, 154 insertions(+), 18 deletions(-)

That is over 4000 lines of code, gone! Talk about beauty soap…

At the moment the library is a super thin layer on top of the raw API, but it works surprisingly well and easier to comprehend.

Handsoap’s beauty comes that it leaves you to do all the parsing and prepping of the SOAP requests/responses. At first this sounds daunting, but as you start using it the benefits become very clear and you are in full control of your client. The author published a video tutorial which shows how to build a weather server client in Rails. It is a very fast paced video, but download a copy and step through it piece by piece.

Handsoap does an excellent job at bringing together curb as the HTTP client, and Nokogiri for the XML parsing. It delivers, blazingly fast.

Code Teaser

The EndpointResolverService in the postini gem is a nice example of a single method service that is used to return another endpoint for subsequent API calls.

At the very least you need a skeleton class that looks something like this:

EXAMPLE_SERVICE_ENDPOINT = {
  :uri => 'http://example.org/ws/service',
  :version => 2
}

class SomeService < Handsoap::Service
  endpoint EXAMPLE_SERVICE_ENDPOINT
  on_create_document do |doc|
    doc.alias 'end', 'https://api-meta.postini.com/api2/endpointresolver'
  end
end

After that, you’re pretty much on your own (which turns out to be great).  Troels shows in the video tutorial how to use a clever Java application called soapUI to analyze the WSDL and prepare your request markup and response parsing accordingly. It works like I charm.

I have three other internal soap4r-based gems that work on the worst API’s you’ve seen in your life (can you say nusoap?). One of them even lacks a WSDL, and returns serialized PHP data. They were a nightmare to consume, and I’m looking forward to cleaning them up with handsoap as well.

I’m hoping this blog post serves as a motivator for you to clean up your (SOAP) act too. I’ve got two more posts lined up on handsoap, one about mocking your way through writing a SOAP client, and another on using soapUI’s mock service capabilities.

Until then, please share in the comments your worst Ruby SOAP stories (and go easy on soap4r, it’s not their fault).

So it’s been pretty quiet around these parts lately, my apologies. But I’ve not been keeping quiet on other fronts, I’ve been busy implementing ruote, by John Mettraux and contributing some patches back to this amazing project. Looking at my github profile you’ll see a lot of things have been happening, and twitter also has some noise…

Today though, I thought I’ll take a break and help Michelle with some XML and ActionScript issues and in the process we needed some Lorem Ipsum’s to test big chunks of text. I remember reading about the lorem gem (by John Nunemaker) and recently Shoes’ Raisins release, and decided to combine them.

The result, lorem-app, my fitting of shoes. I must admit that the fitting went better than expected. Shoes is brilliant, and this release runs much better than the previous one (at least on my Ubuntu amd64 laptop). It takes a bit used to, but rocks. I can just imagine some really solid apps coming from this little framework as more people adopt it and start contributing back.

Hope you enjoy, and watch this space for some other great things coming in the near future.