inkdroid

So on March 1st lucene v1.9 was released and the *next day* pylucene v1.9 is released. Nice work!

I guess there are a bunch of methods that are deprecated in 1.9 which will dissappear entirely in v2.0. Now would be a good time to update usage…

Bruce D’Arcus has started putting his XBib project into subversion at sourceforge. The code is going to include ruby, python and styling libraries. I didn’t realize that sourceforge was offering up svn now…which is a welcome change. I want to set aside some time to get familiar with Bruce’s model and code.

Ian Bicking has some interesting notes about competing web development technologies–mainly in response to some posts from Ivan Krystic. The discussion is definitely recommended, especially if you find yourself looking at web application frameworks for Python and Ruby. I found the pivot point of the discussion to be around a new term (for me) — the “frankenweb”.

My understanding is that like Frankenstein (a being created by stitching together random body parts from dead humans) the frankenweb is an unholy mixture of MVC components pulled from different projects, when put together result in an ugly partially functional whole. I think this characterization of Ian’s work is really unpleasant, but strangely compelling. I think that this is mainly because of Ian’s response:

The “Frankenweb” is a feature, and it describes the web we have, the software we have, and the future that is inevitable. The world was never all J2EE, or ASP(.NET), or PHP, and it won’t be all Rails either.

I think Ian is right on about this: “frankenweb” does describe the web we have, and hopefully the web we will continue to have–and the degree to which we can all interoperate is the degree to which the web will succeed. Perhaps I’m seeing the frankenweb through Weinberger-Colored-Glasses, having just finished Small Pieces Loosely Joined (which I thoroughly enjoyed and plan to write about later if there is time). Weinberger does an excellent job of distilling the essence of the web, and how its architecture enabled it to pull itself up from it’s own bootstraps, grow and adapt:

In the real world, I can’t just put in a door from my apartment to my neighbor’s so that anyone can go through. But that’s exactly how the web was built. Tim Berners-Lee orginally created the web so that scientists could link to the work of other scientists without having to ask their permission. If I put a page into the public Web, you can link to it without having to ask to do anything special, without asking me if it’s alright with me, and without even letting me know that you’ve done it…The web couldn’t have been built if everyone had to ask permission first.

Of course I’m conflating links between pages, and API links between software components…but what Ian says about embracing the frankenweb seems to resonate with this somehow.

It’s also quite disorienting to hear Ivan and others lauding tight coupling:

You don’t see the Ruby on Rails guys modularizing Rails to the point of pain. You see them delivering a single, high-polish, tightly coupled product that does its job well.

Given the various pluggable modules that make up Rails I think “tightly coupled” is largely an overstatement. Granted they are available in the same code base, and I haven’t tried to use one of them in isolation–but I imagine it could be done if someone wanted to say, use a activerecord model in a script or something. The Pragmatic Programmer has a really nice chapter on decoupling, and the authors are actually heavily involved in the Ruby/Rails community. The chapter starts out with a nice quote from Robert Frost’s poem The Mending Wall:

Good fences make good neighbors.

It seems to me that Ian is doing the hard work of patching some of these fences, and building a few and deserves a lot of credit for the effort and cat herding.

I just noticed that a9 has released a second draft of opensearch v1.1. This draft includes details on opensearch autodiscovery for providing a reference to the opensearch description file in an HTML page. This could have a lot of potential for browser plugins. Also, they’ve added a Query element that can be used for echoing back the query that was used to generate results…kinda like the echoedRequest in SRU. These are the things that popped out at me. Of course the big news in the first draft was that Atom can now be used in responses.

At any rate it was nice to see that they link to my opensearch python library from their tools page. Once 1.1 moves from draft I’m going to work on upgrading it from 1.0 right away.

I imagine this is old hat to long time gmail users but I just noticed that my gmail is available via atom…and very easily at that with Mark Pilgrim’s Universal Feed Parser.

    import feedparser
 
    feed = feedparser.parse(
        'https://username:password' + 
        '@gmail.google.com/gmail/feed/atom')
 
    for entry in feed.entries:
        print entry.title

Umm, wow! Adrian Holovaty announced the Washington Post Congressional Votes Database. This site is of important for at least two reasons:

• it offers RSS feeds for tracking the voting of your house/senate representative
• it is powered by Python and the Django web framework.

As far as the RSS goes I just pulled up Dick Durbin’s recent votes and there were over 20 events since 11/17/2005, whereas the comparable service from GovTrack had only one event since then.

After the election I daydreamed about somehow getting involved in the political process in a technical way…which is how I found my way to GovTrack, who are essentially doing very elaborate screen scraping of the Thomas database at the Library of Congress. One thing I really like about the GovTrack project is they are making their data available as RDF, for downstream applications. Adrian’s work seems to draw on a richer data source, as I imagine is the case at a place like the Post. All I can say is well done, and damn…you’ve only been there for a couple months right? Talk about hitting the ground running.

At the recent Snakes and Rubies Adrian indicated that there was going to be some huge Django related news. When the voting db hit my Instant Messenger, IRC client and RSS aggregator I thought that this was it. But according to Adrian there’s something bigger in the works…

I managed to attend Snakes and Rubies yesterday where Adrian Holovaty and David Heinemeier Hansson talked about their respective web frameworks: Django and Rails.

The event started at 2PM and went to 6:30PM or so, and was attended by over 100 people! I watched this little event take shape out of the mists of the local python and ruby mailing lists and was just amazed to see how vibrant the Chicago software development scene is, or has become in the 3 years I’ve lived in the area.

What’s more Adrian and David did a great job promoting both of their projects, while remaining amiable and respectful of the other camp. It’s hard to imagine a similar event between two commercial frameworks. Both were given about 45 minutes or so to talk about their software in any way they wanted. They both had extremely different yet effective presentation styles, and their projects had one important thing in common: disillusionment with PHP.

Rather than talking technical details Adrian spent most of his time focused on how Django came to be down in Kansas at lawrence.com. lawrence.com began it’s life as a PHP application which served as a community site for all sorts of goings on in Lawrence, Kansas. The site interleaved all sorts of local entertainment content: music, dining, art, movies…and it encouraged user participation. For example you can listen to mp3s from local musicians, but here’s the twist, you listen to them as they are playing in town…so if you like a song you can jump over to the venue later that week to see them live. Another example was a full on sports site for the local little leagues which posted details of games, scores, weather conditions, etc. All of this was detailed to show how deeply intertwined all the data was.

The really interesting stuff for me was when Adrian described how journalism informed his software development practices…and how Django fed into this process. In the same way that journalists work against the clock to get news stories out quickly and accurately Adrian and his team worked to get software projects done often on similar deadlines (sometimes like 4 hours). They quickly found that their PHP infrastructure wasn’t allowing them to respond quickly enough without introducing side effects, and decided that they needed new tools…which is how Django was born. In fact the little league application mentioned above was the first Django application.

Adrian has since moved on to the Washington Post, where he is their resident web technology mad scientist. Apparently they are using Django in some form at the Post, or are planning to since he mentioned Django’s caching can scale to the 9 million odd requests the Post gets in a single day.

Unfortunately my lead pencil ran out of lead just a bit of the way into David’s talk, so I don’t have as much written down from the Rails presentation. David dropped some wonderful one liners that I wish I could have written down. Much unlike Adrian, David let actual code do most of the talking for him.

Early on he had a screen with a quote from Richard Feynman on the importance of finding beautiful solutions to problems (if you remember the quote please let me know). This quote kind of guided the rest of the talk where David showed off beautiful Model, View and Controller code from RubyOnRails…and it really was beautiful stuff. David’s thesis was that beautiful things make you happy, and happiness makes you more productive…so beautiful code will make for happy, productive programmers. Much of this comes back to the essential philosophy of Ruby–to give joy to programmers. At any rate, the lights were dimmed and David gave us a tour of what RubyOnRails code looks like, while highlighting some of the strengths of the project and the Ruby language. On one of the pages there was some code to set a cookie expiration, and the date was created like so:

   20.years.from_now

How cool is that! I wasn’t sure if this was part of Ruby proper until I fired up my ruby interpreter to check:

biblio:~ ed$ irb
irb(main):001:0> 20.years.from_now
NoMethodError: undefined method `years' for 20:Fixnum
        from (irb):1

Whereas from the Rails console it works fine:

biblio:~/Projects/cheap ed$ script/console 
Loading development environment.
>> 20.years.from_now
=> Thu Dec 04 15:38:48 CST 2025

So Rails decorates the Fixnum class with the years method. Pretty awesome :-) Another thing David highlighted was that Ruby is used everywhere, from configuration, to writing XML, to writing JavaScript. I was even surprised to hear him argue for full on Ruby in view templates. His argument is that even when a framework offers only a limited set of tags, it’s still offering logic, and rather than creating some bastardized tag language why not just use tried and true Ruby.

The two presentations were followed by a few (not many) moderated questions, and some questions from the audience. The highlight for me was when Why the Lucky Stiff’s question was asked:

Looking a bit beyond web frameworks, how do you envision the world coming to an end?

David responded by “scoping” the world to mean the world of software development and said that this world would come to an end if the layers of Java “sedimentation” continue to accrue. He went on to predict that we’re at a crossroads in software development, and that a paradigm shift is underway…intentionally provocative, and pretty much right on as far as web development goes if you ask me. Adrian responded “Yoko Ono”.

So, as you can tell I’m still digesting the presentations and discussion. There was so much good stuff, and I was really struck by the collegiality between the two guys: open source software development at its finest. The two main things I took away were embracing the boundaries between software development and a particular industry like Journalism, or in my case Libraries; and always trying to strive for the beautiful in software, “boiling down” a thorny problem into its most simple and elegant expression.

While playing around with the Java JDOM library, I found myself resorting to jython to experiment with the API. It’s just so much easier this way for me:

#!/usr/bin/env jython
 
from java.io import StringReader
 
from org.jdom import Document
from org.jdom.input import SAXBuilder
from org.jdom.xpath import XPath
 
xml = '<foo><bar>foobar</bar></foo>'
 
builder = SAXBuilder()
document = builder.build(StringReader(xml))
xpath = XPath.newInstance('//foo/bar')
node = xpath.selectSingleNode(document)
print node.getText()

In case it’s of interest I’ve got a little jython startup script which automatically makes .jar files I drop in a particular directory available to the interpreter. So when testing jdom all i had to do was drop jdom.jar in my /usr/local/jython/jars and it’s immediately available the next time I start up jython.

#!/bin/bash
 
JYTHON_HOME=/usr/local/jython
 
for jar in $JYTHON_HOME/jars/*.jar
do
    jars="$jars:$jar"
done
 
CLASSPATH="$JYTHON_HOME/dist/jython.jar$jars"
 
java -cp $CLASSPATH \
	$JYTHON_JAVA_ARGS -Dpython.home=$JYTHON_HOME \
	org.python.util.jython "$@"

Pretty handy, especially for interactive sessions.

It’s great to see that our ChiPy sprint bore some fruit for the PyPI service. There’s now decent XMLRPC support in PyPI for querying the packages. This will hopefully open up the door for lots of PyPI utilities that abound in the Perl/CPAN world…like this very simple client for listing packages:

#!/usr/bin/env python
 
import xmlrpclib
import sys
 
pypi = xmlrpclib.ServerProxy('http://python.org/pypi')
for hit in pypi.search( { 'name':sys.argv[1] } ):
    urls = pypi.package_urls( hit['name'], hit['version'] )
    if urls:
        hit['url'] = urls[0]['url']
    else:
        hit['url'] = 'None'
    print 'NAME: %(name)s [%(version)s]\nDESC: %(summary)s\nURL: %(url)s' % hit
    print

which if you put in your PATH allows you to:

biblio:~ ed$ pypi-search sql
NAME: dbsql [0.7.1]
DESC: A tool to sync pilot-db databases with an SQL database.
URL: None

NAME: ll-orasql [0.6]
DESC: Utilities for working with cx_Oracle
URL: http://cheeseshop.python.org/packages/source/l/ll-orasql/ll-orasql-0.6.tar.bz2

NAME: ll-sql [0.7.1]
DESC: A module for generating SQL queries
URL: http://cheeseshop.python.org/packages/source/l/ll-sql/ll-sql-0.7.1.tar.gz

NAME: MySQL-python [1.2.0]
DESC: Python interface to MySQL
URL: None

NAME: pSQL [0.9.2]
DESC: SQL statement wrapper class
URL: None

NAME: pymssql [0.5.2]
DESC: A simple database interface to MS-SQL for Python.
URL: None

NAME: pymssql [0.6.0]
DESC: A simple database interface to MS-SQL for Python.
URL: None

NAME: pymssql [0.7.1]
DESC: A simple database interface to MS-SQL for Python.
URL: None

NAME: pyPgSQL [2.3]
DESC: pyPgSQL - A Python DB-API 2.0 compliant interface to PostgreSQL.
URL: None

NAME: pysqlite [2.0.3]
DESC: DB-API 2.0 interface for SQLite 3.x
URL: None

NAME: SnakeSQL [0.5.2 Alpha]
DESC: Pure Python SQL database supporting NULLs, foreign keys and simple joins
URL: None

NAME: SQLEdit [1.0]
DESC: An sql table editor and query viewer.
URL: None

NAME: SQLObject [0.6.1]
DESC: Object-Relational Manager, aka database wrapper
URL: None

As you can see there is something odd going on with the package URL…it requires an extra trip to the XMLRPC service to get it, and it is rarely there.

I mentioned Pascal’s Triangle in the previous post, and after typing in the Oz code decided to make a Pascal’s Triangle pretty printer in python.

from sys import argv
 
def pascal(n):
    if n == 1:
        return [ [1] ]
    else:
        result = pascal(n-1)
        lastRow = result[-1]
        result.append( [ (a+b) for a,b in zip([0]+lastRow, lastRow+[0]) ] )
        return result
 
def pretty(tree):
    if len(tree) == 0: return ''
    line = '  ' * len(tree)
    for cell in tree[0]:
        line += '  %2i' % cell
    return line + "\n" + pretty(tree[1:])
 
if __name__ == '__main__':
    print pretty( pascal( int(argv[1]) ) )

Which, when run with can generate something like this:

biblio:~/Projects/bookclub ed$ python pascal.py 9
                     1
                   1   1
                 1   2   1
               1   3   3   1
             1   4   6   4   1
           1   5  10  10   5   1
         1   6  15  20  15   6   1
       1   7  21  35  35  21   7   1
     1   8  28  56  70  56  28   8   1

It’s been fun reading up on the uses for Pascal’s triangle, although I imagine this is old hat for people more familiar with math than I. Still I think getting through this tome will be time well spent in the long run.