Open Sourcery

After a misrable production implementation of BIND on a MySQL backend, we were forced to re-evaluate our use of PowerDNS, and what happens to the BIND DLZ on Rails project.

I’m glad to announce that PowerDNS on Rails will be taking over where BIND DLZ on Rails left off.

It’s been a crazy three days of refactoring, but the code is now fully operational and we have our first production implementation (complete with clients using the REST interface). It’s an exciting time for the project, over the next couple of weeks I’ll be ironing out some grey areas of PowerDNS with its users and I’ll be improving the UI significantly (as well as sneaking in new features).

This time I’ll hold back on promises of release candidates, instead I’ll just tag them and announce them afterwards.

I’ll also be posting some interesting Rails tips, especially since I had to bend ActiveRecord in ways I didn’t thought possible to cope with the PowerDNS schema. Thanks to everyone who made RSpec, without it this refactoring job would have been a disaster from the word go.

Here is to the future!

It is with great excitement, and sadness that I announce the tag of BIND DLZ on Rails RC1.

We were very motivated as a team to get this product, and the accompanying infrastructure in place so would could continue to enhance and expand our DNS infrastructure. We made two fatal mistakes in trying to achieve this goal:

Understand your existing infrastructure

We used the easy way out and blamed PowerDNS for some of our DNS does, where it ended up being out woes with TUPA, not PowerDNS. Typical I guess, since everyone else uses BIND, we’ll use BIND as well. We never went out to fully understand the problems not the solutions. We just decided to blindly drop an entire stack of services for a new one. Thats bad.

Understand your new infrastructure

Same goes for this. We checked out the BIND-DLZ patches, heard it was accepted into BIND itsef, and got excited. We could run everyone’s dream DNS server with a flexible MySQL 5 backend. Boy, what a mistake. We should have evaluated BIND DLZ first, before building an entire UI for it and then only testing it.

The whole of last week was spent trying to get BIND to behave. It would random crash without warning. I discussed this with the bind-dlz-testers list over at SourceForge, who argued I should downgrade the MySQL client libraries to MySQL 4. For us this was easily possible since the MySQL slaves and DNS servers were different boxes, for others this might not be the case. As part of this excercise I had to learn how to update Gentoo ebuild’s, so I could submit a fix to Gentoo as well for their net-dns/bind-9.5.1-p1 ebuild.

Who’s to blame? Well, us. Not BIND, or the guys who developed the DLZ patches. There are plenty reports out there of issues with the MySQL client libs, but some very clever people have found ingenious ways of working around it. I personally think we have an odd combination of Hardened Gentoo & BIND issues.

What happens next?

Well, we’ll be sticking to PowerDNS for the time being, or maybe permanently. We’ll be planning our DNS offerings out in full, and then start to see how PowerDNS can accomodate us. If, and only if, it cannot, we’ll dive into the alternatives.

All the work is not lost, I’ve basically made a copy (not a fork) of the git repo and modified the entire application to run on the PowerDNS schema. So keep an eye out for PowerDNS on Rails.

I won’t try to juggle branches, the differences are too big. However, I’ll be porting changes implemented in PowerDNS on Rails back to BIND DLZ on Rails. My hope is that someone picks up BIND DLZ on Rails and runs with it further.

Thanks to everyone for their interest in the project.

UPDATE (Jul 22, 2008): I don’t have connectivity at home at the moment, so I’ll roll our production systems over to BIND (and BIND DLZ on Rails) from PowerDNS and Tupa in the morning. Based on the initial feedback from our support team I’ll either address small bugs we overlooked, or tag RC1, whichever comes first.

Just a quick update, we’re closing the gap rapidly on pushing RC1 to Github. Hopefully that happens by the weekend. We’ll also be rolling it into production use to iron out any last remaining issues that didn’t surface from our lab tests/reviews.

We just have one issue to overcome with will_paginate and our custom scoped finders for Zones, then I’ll push to Github.

In preparation (and celebration) I’ve added the project to Ohloh as well for some added metrics and publicity. See the badge below, and on the updated project page.

Strangely enough this started as a request from Tyler Bird after reviewing my Capistrano presentation. He was interested in learning how I use Capistrano to update external site dependencies, like a local cached copy of the Google Analytics JavaScript files.

Originally all I did was have this task in my Capfile, and hooked it in after the “deploy:symlink” task:

desc "Update local urchin.js file"
task :update_urchin, :role => :web do
  run "wget -O #{current_path}/public/urchin.js http://www.google-analytics.com/urchin.js"
end

I then simply changed the configurations of the Google Analytics plugin to point to local urchin.js file.

Wow, thats simple enough, but what’s the catch?

Two things bothered me, support for the newer ga.js file, and not having access to Rails’ timestamping of assets for far future caching.

I pointed out to Tyler that I’m revisiting a lot of things we do with Capistrano and will post them here as I go along. The Analytics issue was one that bothered me constantly, so I set out to correct it.

I made my own fork of the Google Analytics plugin over at GitHub, and added support for having local copies of both the legacy and new Analytics JavaScript files, complete with Rails timestamping in the mix. I also added a rake task to help out with keeping the files up to date.

So the capistrano task I mentioned earlier would be changed to look like this now:

desc "Update local Analytics cached files"
task :update_analytics, :role => :web do
  run "cd #{current_path} && rake google_analytics:update RAILS_ENV=#{rails_env}"
end

Now you can neatly update your cached copies on each deployment.

Please visit the GitHub fork for more information on the plugin itself. I’ll set up a local project page on this blog tomorrow. The whole excercise was aimed at showing how to use Capistrano for updating external dependencies, although I have to admit it got a little lost in the noise as I carried on.