Macros allow a DNS administrator to setup a predefined sequence of steps that will later be applied to any domain they choose, via the web interface or API access.

Where is this used? Well, lets take a simple example related around web hosting. Lets say you have zone templates configured for three data centers, one in South Africa, one in Europe and one in the US. By default you’ll probably use one of these templates when new domains are added to the system. The template might contain everything you need for web hosting and email to work. Later the client asks to be shifted to another continent, for whatever reason, and you’ll have to update a load of records for that domain.

Macros solves this by allowing you to predefine several actions which either creates, removes or updates records for the domain. Macros can then be applied on any domain to have these changes take affect on that domain.

Just like zone templates, macros are available to owners and administrators alike.

PowerDNS on Rails Macro Screen (click for full version)

I’ve also started creating a new official website for the PowerDNS on Rails project, and it lives over here.

Some background

Almost all applications these days are expected to provide some kind of REST interface, over and above HTML. It is just part of the whole programmable web paradigm, and makes applications just so much more useful.

PowerDNS on Rails does sport a limited REST interface for adding new domains, and I’ll extend it over time to provide a full ActiveResource compatible interface to the system. The REST paradigm works great, make no mistake, but in a lot of cases it falls flat…

Don’t re-invent the wheel

You spend a couple of hours getting PowerDNS on Rails installed and tweaked, you even get new domain additions plugged in via the REST interface. This works great. For some of the bigger clients you provide full access to manage their own domains. They’re happy, you’re happy.

For some time this works, but then smaller clients start bothering you with requests for CNAME’s and other misc A records. Initially it doesn’t matter much, until they start realizing it takes you a minute to make the changes, so they start requesting more often, and more often, and more…

Now you sit with some tough decisions…

Rebuild PowerDNS into your own app, and leverage a full REST API.

This makes sense, since most ISP’s (us included) have some other form of management system that the clients use to add & remove domains. In a lot of cases we completely hide the complexities of the DNS system from the users, they only know about their domains, nothing else.

But why rebuild an already complete interface? Why bother when the open source community will maintain your interfaces for you?

Sold? Thought so, now you might have one option left…

Manage users & domain ownerships

This reinvents another wheel all together. Some folks might argue that PowerDNS on Rails could leverage external authentication mechanisms like LDAP… I don’t believe this will solve any problems, I believe it would only make it worse.

And once you start giving people access to the system you have to deal with forgotten password requests, amongst other things. Enabling the forgotten password request functionality for a system that performs core backbone functions is not really an option, the damage done would really hurt domain owners and users dependent on DNS…

So what is a man to do?

One possible solution

So we agreed we’re not building our own interface on top of PowerDNS on Rails’ existing one, we alse agreed that we’re not going into the external authentication debate.

So we need a mechanism that will provide the user with pre-defined access to a specific domain so they can do their own updates, without bothering us.

The mechanism should ideally have the following characteristics:

• Linked to a user that issued the access, gives accountability
• Linked to a specific domain, prevents snooping around the system
• Default policy of deny everything
• Restrict alterations to the RR’s
  • We host you mail, you can’t change the MX records
  • We host your site, you can’t change some A records
  • You cannot alter the SOA or NS records, ever
  • You can add a CNAME, then remove it later
  • Add/remove additional A records that won’t affect other hosted services
  • Protect RR’s by type (MX, A, TXT, CNAME, etc…)
• Prevent adding RR’s
• Prevent removing RR’s
• An expiry mechanism, only valid until X
• Hook into the audit trails

This is more or less the conversation our support guys would have with a person before they would login for the first time. Why have the conversation when you could restrict their movements completely? Since you provide them with the DNS and other services, its up to specify how they can enhance their naming system, or destroy it totally.

Enter authentication tokens!

Currently authentication tokens can only be created via HTTP post, and only by token_user role members. The web interface is forth coming. Authentication tokens restrict user movements to a specific domain, and specific RR’s. They have all the characteristics mentioned earlier, and the interfaces adapt dynamically to show the user only what they can do.

You can now easily hook PowerDNS on Rails into your existing systems by providing users with a ‘advanced DNS’ link. User selects, your systems asks PowerDNS on Rails for a token with the said restrictions in place, get a snippet of XML back with contains the token and full URL to the system. Then you just redirect your client to PowerDNS on Rails, et voila.

To get an idea of how to create authentication tokens, look at  spec/models/auth_token_spec.rb for complete examples. I know documentation is still an issue with the project…

And the disclaimers

In the entire history of the project this has been the first push to a public repo that hasn’t been battle tested in our production or staging environments. I’m busy setting up our staging environment for testing, and I’m sure the users will come up with some issues that will beed addressing.

I wanted to get the code out there for review as well, since I’m not an expert in application security there might be an issue or two that I’m not aware off. The code is extensively spec’ed, and if you don’t use the tokens the rest of the application still works as espected. A few minor bugs we’re squashed inadvertedly in the process, and I’ll update the tracker to reflect this.

Call for comments!

Please scrutinize the code, and give feedback. Hopefully the idea and implementation is proven solid and people can extract it for later use in other projects where the DRY principle of tokens really make a difference.

PowerDNS on Rails uses the cookie store, and until b2ff9410de[...] had both the session key and session secret hard coded in the environment.rb file.

It came down on my like a ton of bricks when I was moderating a comment by JGeiger on my previous post (Using hoptoad in open source project deployments). I quickly jumped to fix the code using a solution put out by Trevor Turk which requires users to set both the session key and session secret in their database.yml files.

I know this is not an optimal solution, but works until the Core team (and us) have found a reasonable solution to the problem.

As part of stating the obvious (you gotta love hindsight), I recommend anyone who is running PowerDNS on Rails in a production environment to run it over SSL. Apart from session hijacking woes, having your zone data readable as plain text is just as bad as allowing zone transfers to any DNS client…

Next on the radar is authentication tokens, allowing to give users one-time access to a specific domain for performing updates.

We’ll keep you posted!

Auditing in Rails is not for the faint hearted. The Rails Recipes book has an example on how to do this with sweepers, and using the new ActiveRecord dirty object tracking can help ease the problem too. So after some investigation I found the brilliant acts_as_audited plugin by Brendon Keepers.

After playing with it, I realised one short coming. This is very much a problem specifically for PowerDNS on Rails, but after some thought I realized it could be applied to other auditing challenges as well.

Enter parent record tracking

I forked the original project on github, and started hacking. Not too long after I came up with this:

class Author < ActiveRecord::Base
  has_many :books
end
class Book < ActiveRecord::Base
  belongs_to :author
  acts_as_audited :parent => :author
end

Using the sweepers is also possible, like this:

class Application < ApplicationController::Base
  audit Author, Book, :parents => { Book= > :author }
end

As contrived as it can be, it allows us to track all the changes to books by a specific author. In the DNS world this allows us to easily access all changes made to the records of a particular domain, and it becomes very valuable. It becomes even more valuable when you need to start tracking deleted records.

Pull requests were sent to the network, hopefully they get accepted and I can remove my repo again.

Looking forward?

This was a relatively small step forward for the plugin, but opens a lot of possiblities for the host applications, like PowerDNS on Rails. In any ISP environment audits are extremely important, especially when clients start gaining limited access to the backend systems…

I rolled out MySQL replication with SSL enabled, you can Google for some good howto’s on the topic and I’ll give some posts below. Basically we have secure replication, and each DNS server is isolated in case of a disaster.

The gotcha then?

The MySQL docs, and some other howto’s indicate you should set the SSL client certificate details in your my.cnf file, under the [client] section. It makes sense, the slave is a client of the master. Appears not so with Gentoo’s mysql-5.0.60-r1 ebuild. It appears even less so with PowerDNS, who rightfully reads the my.cnf file as client.

Cause and effect

PowerDNS now tries to connect to the slave using the SSL details specified in the [client] section of the my.cnf file. This breaks, since you probably never configured your MySQL slave to have PowerDNS connect via SSL.

Secondly, it appears MySQL blatantly ignores the settings when used with replication, and you actually need to specify the client certificates in the CHANGE MASTER TO statement.

Aftermath

Nothing serious, was quick to piece together what was going on. Now I’ll have double digit DNS servers scattered around the globe near pearing point, with SSL encryption for the replication data. Brilliant, that really is resillient DNS!

More reading

• MySQL Manual – Using SSL for secure connections
• MySQL Manual – Setting up replication using SSL
• Be your own CA – Not MySQL specific
• MySQL Replication with SSL
  
• MySQL Replication with SSL

Hope this prevents any future missery for someone else.

I’m glad to announce that PowerDNS on Rails will be taking over where BIND DLZ on Rails left off.

It’s been a crazy three days of refactoring, but the code is now fully operational and we have our first production implementation (complete with clients using the REST interface). It’s an exciting time for the project, over the next couple of weeks I’ll be ironing out some grey areas of PowerDNS with its users and I’ll be improving the UI significantly (as well as sneaking in new features).

This time I’ll hold back on promises of release candidates, instead I’ll just tag them and announce them afterwards.

I’ll also be posting some interesting Rails tips, especially since I had to bend ActiveRecord in ways I didn’t thought possible to cope with the PowerDNS schema. Thanks to everyone who made RSpec, without it this refactoring job would have been a disaster from the word go.

Here is to the future!

We were very motivated as a team to get this product, and the accompanying infrastructure in place so would could continue to enhance and expand our DNS infrastructure. We made two fatal mistakes in trying to achieve this goal:

Understand your existing infrastructure

We used the easy way out and blamed PowerDNS for some of our DNS does, where it ended up being out woes with TUPA, not PowerDNS. Typical I guess, since everyone else uses BIND, we’ll use BIND as well. We never went out to fully understand the problems not the solutions. We just decided to blindly drop an entire stack of services for a new one. Thats bad.

Understand your new infrastructure

Same goes for this. We checked out the BIND-DLZ patches, heard it was accepted into BIND itsef, and got excited. We could run everyone’s dream DNS server with a flexible MySQL 5 backend. Boy, what a mistake. We should have evaluated BIND DLZ first, before building an entire UI for it and then only testing it.

The whole of last week was spent trying to get BIND to behave. It would random crash without warning. I discussed this with the bind-dlz-testers list over at SourceForge, who argued I should downgrade the MySQL client libraries to MySQL 4. For us this was easily possible since the MySQL slaves and DNS servers were different boxes, for others this might not be the case. As part of this excercise I had to learn how to update Gentoo ebuild’s, so I could submit a fix to Gentoo as well for their net-dns/bind-9.5.1-p1 ebuild.

Who’s to blame? Well, us. Not BIND, or the guys who developed the DLZ patches. There are plenty reports out there of issues with the MySQL client libs, but some very clever people have found ingenious ways of working around it. I personally think we have an odd combination of Hardened Gentoo & BIND issues.

What happens next?

Well, we’ll be sticking to PowerDNS for the time being, or maybe permanently. We’ll be planning our DNS offerings out in full, and then start to see how PowerDNS can accomodate us. If, and only if, it cannot, we’ll dive into the alternatives.

All the work is not lost, I’ve basically made a copy (not a fork) of the git repo and modified the entire application to run on the PowerDNS schema. So keep an eye out for PowerDNS on Rails.

I won’t try to juggle branches, the differences are too big. However, I’ll be porting changes implemented in PowerDNS on Rails back to BIND DLZ on Rails. My hope is that someone picks up BIND DLZ on Rails and runs with it further.

Thanks to everyone for their interest in the project.