Just a quickie to tell you about my (first) latest gist: http://gist.github.com/19021
Basically search through the postfix maillog for a pattern, extract the postfix message id from each matched line, and then search through the log file for each corresponding line.
This rebuilds your original search with some more context, and makes it a snap to trace a message through its lifecycle…
webby, for those who don’t know, is a brilliant little gem by Time Pease for creating and managing static HTML sites with Ruby. Built as extension to Rake, it allows you to manage your content seperately from your layout, and use a host of filters on your content (textile, haml, erb, etc).
I’ve been loving it so far, and have a host of ideas for extending webby (which will start in due time). We’ve been rebuilding the entire network of inX websites using all kinds of random combinations of Ruby. Between Merb, Rails and Webby we’ve got all our bases covered for the different websites.
Sitemaps in webby
Part of any site optimization experience is providing a sitemap for Google/Yahoo! indexing. This helps the search providers to better understand your content.
Creating a sitemap in Rails & Co is extremely simple, but how do you tackle it if webby?
Well, webby gives us access to a @pages instance variable, which is its own internal “resource database”. Note that I used the term resource database loosely. This allows us to run through all the pages in our site and extract some of their meta-data for use in a generated sitemap.
Create a file called sitemap.xml in the root of your content folder, and paste the following into it:
---
filter: erb
extension: xml
layout: nil
---
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<% @pages.find( :all, :sitemap => true ).each do |page| %>
<url>
<loc><%= page.url %></loc>
<lastmod><%= page.mtime.strftime("%Y-%m-%d") %></lastmod>
<% if page.changefreq %>
<changefreq><%= page.changefreq %></changefreq>
<% end %>
<% if page.priority %>
<priority><%= page.priority %></priority>
<% end %>
</url>
<% end %>
</urlset>The sitemap has its own meta, telling webby not to use a layout and keep the xml file extension. For the rest the file uses the @pages resource database to find pages that have been flagged for inclusion in the sitemap.
To add a file to the sitemap you extend the meta information of the page like this:
--- sitemap: true ---
You can also provide a priority to the sitemap like this:
--- sitemap: true priority: 0.9 ---
And even a change frequency:
--- sitemap: true priority: 0.9 changefeq: weekly ---
If you run a blog you can probably extend the idea for generating RSS or ATOM feeds.
Robots.txt extension
You also add a robots.txt file to the root of your content directory to help the crawlers find the sitemap, here is my copy:
--- filter: erb extension: txt layout: --- Sitemap: /sitemap.xml
Further reading
Yesterday I pushed a slew of changes to the PowerDNS on Rails repo. The sole focus of the work was to extend the authentication system to have temporary authentication tokens. What are authentication tokens? Allow me to explain…
Some background
Almost all applications these days are expected to provide some kind of REST interface, over and above HTML. It is just part of the whole programmable web paradigm, and makes applications just so much more useful.
PowerDNS on Rails does sport a limited REST interface for adding new domains, and I’ll extend it over time to provide a full ActiveResource compatible interface to the system. The REST paradigm works great, make no mistake, but in a lot of cases it falls flat…
Don’t re-invent the wheel
You spend a couple of hours getting PowerDNS on Rails installed and tweaked, you even get new domain additions plugged in via the REST interface. This works great. For some of the bigger clients you provide full access to manage their own domains. They’re happy, you’re happy.
For some time this works, but then smaller clients start bothering you with requests for CNAME’s and other misc A records. Initially it doesn’t matter much, until they start realizing it takes you a minute to make the changes, so they start requesting more often, and more often, and more…
Now you sit with some tough decisions…
Rebuild PowerDNS into your own app, and leverage a full REST API.
This makes sense, since most ISP’s (us included) have some other form of management system that the clients use to add & remove domains. In a lot of cases we completely hide the complexities of the DNS system from the users, they only know about their domains, nothing else.
But why rebuild an already complete interface? Why bother when the open source community will maintain your interfaces for you?
Sold? Thought so, now you might have one option left…
Manage users & domain ownerships
This reinvents another wheel all together. Some folks might argue that PowerDNS on Rails could leverage external authentication mechanisms like LDAP… I don’t believe this will solve any problems, I believe it would only make it worse.
And once you start giving people access to the system you have to deal with forgotten password requests, amongst other things. Enabling the forgotten password request functionality for a system that performs core backbone functions is not really an option, the damage done would really hurt domain owners and users dependent on DNS…
So what is a man to do?
One possible solution
So we agreed we’re not building our own interface on top of PowerDNS on Rails’ existing one, we alse agreed that we’re not going into the external authentication debate.
So we need a mechanism that will provide the user with pre-defined access to a specific domain so they can do their own updates, without bothering us.
The mechanism should ideally have the following characteristics:
This is more or less the conversation our support guys would have with a person before they would login for the first time. Why have the conversation when you could restrict their movements completely? Since you provide them with the DNS and other services, its up to specify how they can enhance their naming system, or destroy it totally.
Enter authentication tokens!
Currently authentication tokens can only be created via HTTP post, and only by token_user role members. The web interface is forth coming. Authentication tokens restrict user movements to a specific domain, and specific RR’s. They have all the characteristics mentioned earlier, and the interfaces adapt dynamically to show the user only what they can do.
You can now easily hook PowerDNS on Rails into your existing systems by providing users with a ‘advanced DNS’ link. User selects, your systems asks PowerDNS on Rails for a token with the said restrictions in place, get a snippet of XML back with contains the token and full URL to the system. Then you just redirect your client to PowerDNS on Rails, et voila.
To get an idea of how to create authentication tokens, look at spec/models/auth_token_spec.rb for complete examples. I know documentation is still an issue with the project…
And the disclaimers
In the entire history of the project this has been the first push to a public repo that hasn’t been battle tested in our production or staging environments. I’m busy setting up our staging environment for testing, and I’m sure the users will come up with some issues that will beed addressing.
I wanted to get the code out there for review as well, since I’m not an expert in application security there might be an issue or two that I’m not aware off. The code is extensively spec’ed, and if you don’t use the tokens the rest of the application still works as espected. A few minor bugs we’re squashed inadvertedly in the process, and I’ll update the tracker to reflect this.
Call for comments!
Please scrutinize the code, and give feedback. Hopefully the idea and implementation is proven solid and people can extract it for later use in other projects where the DRY principle of tokens really make a difference.
