Correlating documents in CouchDB

by Kenneth Kalmer on December 13, 2009

I’m in the very fortunate position, two actually, of being able to 1) migrate my biggest production application from MySQL to CouchDB, and 2) build a stunning new system for a multinational welfare organization on top of CouchDB.

I’ve been lurking in the CouchDB world for quite some time and have spent a lot of time wrestling with how to loosely relate documents in CouchDB to each other. A big part of learning to use CouchDB successfully is to break away from the shackles of the relational world. Relationships between documents is one such a shackle that seems hard to break.

It is unavoidable that data has to be correlated, and I wanted to rethink how to do it. After plenty of discussions in #ruote with John Mettraux we came up with a model based on how the web works. Since CouchDB is “off the web”, the approach feels quite fitting to me and hopefully to you too.

First some insight into my thinking at this stage.

The web has been successful in loosely expressing relationships between documents. Take two examples:

and

For those of you reading this through a reader, click through to see the gist’s above.

Simple as it seems, in both cases we have a document that is somehow related to the page. The nature of the relationship is expressed via the rel attribute, and the target specified via the href attribute. This got me thinking. Since CouchDB is made off the web, can’t these same principles be applied?

Yes, they can. And here is how:

Currently you might be tempted to express relationships link this in your JSON:

Where changing it to this holds the key:

If anything this format, albeit more verbose, expresses the relationships more clearly and in a format that is web friendly. We’ve broken the shackles of relational thinking.

Enter Correlate

Correlate is an experiment in this line of thinking. It is a mixin for CouchRest’s extended documents that allows you to express these relationships:

Correlate generates getter and setter methods for working with your relationships and lot more (review the README). It also includes a compatibility layer for ActiveRecord to help when you’re migrating from ActiveRecord to CouchRest or building a system on CouchDB that needs to access legacy data via ActiveRecord.

The project is still pretty much a moving target, but I’d love to hear how others address the same issues. Correlate does a great job at maintaining relationship information in a web friendly manner and providing you with some convenience around the verbose data structure. Correlate also has a lot of room for improvement, but that will hopefully change over the coming days as I continue integrating it deeply into my existing projects.

Please fork the project on github and join the experiment with me.

VirtualBox, Gentoo and serial consoles

by Kenneth Kalmer on September 30, 2009

More as a reminder to myself for when I need this again, but I’m sure everyone needs this at least once.

Having screwed up my kernel configs for my VirtualBox Gentoo image, I needed a serial console to catch the boot messages scrolling past in order to see if all the required hardware was being loaded by the kernel. I’ve never done this on a physical machine before but I am converted now and will acquire a USB to serial port converter in the near future…

Using this article as a base you need to do the following:

  1. Enable serial ports for your virtual machine
  2. Select “Host Pipe”
  3. Enter /tmp/vboxconsole as the filename
  4. Use netcat to read the console: nc -U /tmp/vboxconsole

When booting you need to amend your grub boot line to have the following at the end:

console=ttyS0,38400

Making it look something like this:

kernel=/kernel-2.6.30-r6 root=/dev/sda3 console=ttyS0,38400

Proceed to boot and look at netcat to see the entire boot output scroll past without disappearing into thin air when the kernel panics.

Man, I love virtualization. I tested this on Mac OS X 10.5 with VirtualBox 3.0.6, but it should work on any *nix platform. Some more Gentoo serial console madness can be found on the old Gentoo Wiki.

Rails specs not running under Ruby 1.9 ?

by Kenneth Kalmer on September 27, 2009

I spent some time getting PowerDNS on Rails to run on Ruby 1.9.1, which ended up being very easy due to the small amount of plugins & gems used by the project. The only change I had to make myself was to the acts_as_audited plugin, where the one-line fix got merged upstream.

The worst part of the process was getting the specs to run with rake spec. Using ./script/spec it worked on individual specs and on all the specs worked as advertised, but rake spec didn’t do anything.

After a lot of time spent in the debugger I wasn’t any wiser. The only difference was that in Ruby 1.8 the example groups were fully loaded, and in Ruby 1.9 they were empty. I gave up and started searching relentlessly for some information on the issue. I couldn’t find anything, until I found an indirect solution on the rspec wiki. It seems that if you have any versions of the test-unit gem after the 1.2.3 release installed, your Rails specs will simply not run. For me, removing test-unit 2.0.3 made the difference and the specs ran properly. PowerDNS on Rails has now joined the ranks of my Ruby 1.9 compatible projects.

Ruote in 20 minutes video

by Kenneth Kalmer on August 25, 2009

Kudos to my brother and the rest of the guys at Monopost for cleaning up the bad audio as best they can.

Ruote in 20 minutes from Kenneth Kalmer on Vimeo.

The original post of the presentation is over here.