inkdroid » publishing

federal register embraces the web and opensource

ed — Tue, 27 Jul 2010 15:05:52 +0000

Tom Lee of the Sunlight Foundation blogged yesterday about the new Federal Register website. The facelift was also announced a few days earlier by the Archivist of the United States, David Ferriero. If you aren’t familiar with it already, the Federal Register is basically the daily newspaper of the United States Federal Government, which details all the rules and regulations of the federal agencies. It is compiled by the Office of the Federal Register located in the National Archives, and printed by the Government Printing Office. As the video describing the new site points out, the Federal Register began publication in 1936 in the depths of the Great Depression as a way to communicate in one place all that the agencies were doing to try to jump start the economy. So it seems like a fitting time to be rethinking the role of the Federal Register.

I’m no usability expert, but just a few minutes browsing the new site and comparing it to the old one make it clear what a leap forward this is. Hopefully the legal status of the new site will be ironed out shortly.

Most of all it’s great to see that the Federal Register is now a single web application. The service it provides to the American public is important enough to deserve its own dedicated web presence. As the developers point out in their video describing the effort, they wanted to make the Federal Register a “first class citizen of the web”…and I think they are certainly helping do that. This might seem obvious, but often there is a temptation to jam publications from the print world (like the Federal Register) into dumbed down monolithic repositories that treat all “objects” the same. Proponents of this approach tend to characterize one off websites like Federal Register 2.0 as “yet another silo”. But I think it’s important to remember that the web was really created to break down the silo walls, and that every well designed web site is actually the antithesis of a silo. In fact, monolithic repository systems that treat all publications as static documents to be uniformly managed are more like silos than these ‘one off’ dedicated web applications.

As a software developer working in the federal government there were a few things about the Federal Register 2.0 that I found really exciting:

Fruitful collaboration between federal employees and citizen activist/geeks initiated by a software development contest.
Extensive use of opensource technologies like Ruby, Ruby on Rails, MySQL, Sphinx, nginx, Varnish, Passenger, Apache2, Ubuntu Linux, Chef. Opensource technologies encourage collaboration by allowing citizen activists/technologists to participate without having to drop a princely sum.
Release of the source code for the website itself, using decentralized revision control (git) so that people can easily contribute changes, and see how the site was put together.
Extensive use of syndicated feeds to communicate how how content is being added to the site, ical feeds to keep on top of events going on in your area, and detailed XML for each entry.
The robots.txt file for the site makes the content fully crawlable by web indexers, except for search related portions of the website. Excluding dynamic search results is often important for performance reasons, but much of the article content can be discovered via links, see below about permalinks. They also have made a sitemap available for crawlers to efficiently discover URLs for the content.
Deployment of the web application to the cloud using Amazon’s EC2 and S3 services. Cloud computing allows computing resources to scale to meet demand. In effect this means that government IT shops don’t have to make big up front investments in infrastructure to make new services available. I guess the jury is still out, but I think this will eventually prove to greatly lower the barrier to innovation in the egov sector. It also lets the more progressive developers in government leap frog ancient technologies and bureaucracies to get things done in a timely manner.
And last, but certainly not least … now every entry in the Federal Register has a URL!. Permalinks for the Federal Register are incredibly important for citability reasons. I predict that we’ll quickly see more and more people referencing specific parts of the Federal Register in social media sites like Facebook, Twitter and out on the open web in blogs, and in collaborative applications like Wikipedia.

I would like to see more bulk access to XML data made available, for re-purposing on other websites–although I guess it might be able to walk from the syndicated feeds to the detailed XML. Also, the search functionality is so rich it would be useful to have an OpenSearch description that documents it, and perhaps provides some hooks for getting back JSON and/or XML representations. Perhaps even following the lead of the London Gazette and trying to make some of the structured metadata available in the the HTML using RDFa. It also looks like content is only available for 2008 on, so it might be interesting to see how easy it would be to make more of the historic content available.

But the great thing about what these folks have done is now I can fork the project on github, see how easy it is to add the changes, and let the developers know about my updates to see if they are worth merging back into the production website. This is an incredible leap forward for egov efforts–so hats off to everyone who helped make this happen.

a middle way for linked data at the bbc

ed — Tue, 02 Mar 2010 20:13:00 +0000

I got the chance to attend the 2nd London Linked Data Meetup that was co-located with dev8d last week, which turned out to be a whole lot of fun. I figured if I waited long enough other people would save me from having to write a good summary/discussion of the event…and they have: thanks Pete Johnston, Ben Summers, Sheila Macneill, Martin Belam and Frankie Roberto.

The main thing that I took away is how much good work the BBC is doing in this space. Given the recent news of cuts at the BBC, it seems like a good time to say publicly how important some of the work they are doing is to the web technology sector. As part of the Meetup Tom Scott gave a presentation on how the BBC are using Linked Data to integrate distinct web properties in the BBC enterprise, like their Programmes and the Wildlife Finder web sites.

The basic idea is that they categorize (dare I say catalog?) television and radio content using wikipedia/dbpedia as a controlled vocabulary. Just doing this relatively simple thing means that they can create another site like the Wildlife Finder that provides a topical guide to the natural world (and also happens to use wikipedia/dbpedia as a controlled vocabulary), that then links to their audio and video content. Since the two sites share a common topic vocabulary, they are able to automatically create links from the topic guides to all the radio and television content that are on a particular topic.

For a practical example take a look consider this page for the Great Basin Bristlecone Pine:

If you scroll down on the page you’ll see a link to a video clip from David Attenborough’s documentary Life on the Programmes portion of the website. Now take a step back and consider how these are two separate applications in the BBC enterprise that are able to build a rich network of links between each other. It’s the shared controlled vocabulary (in this case dbpedia derived from wikipedia) which allows them to do this.

If you take a peak in the html you’ll see the resource has an alternate RDF version:

/nature/species/Pinus_longaeva.rdf" />

The Resource Description Framework (RDF) is really just the best data model we have for describing stuff that’s on the Web, and the type of links between resources that are on (and off) the Web. Personally, I prefer to look at RDF as Turtle which is pretty easily done with Dave Beckett‘s handy rapper utility (`aptitude install raptor-utils` if you are following from home).

rapper -o turtle http://www.bbc.co.uk/nature/species/Pinus_longaeva

The key bits of the RDF are the description of the Great Basin bristlecone pine:


    rdfs:seeAlso  ;
    foaf:primaryTopic  .


    dc:description "Great Basin bristlecone pines are restricted to the mountain ranges of California, Nevada and Utah and have a remarkable ability to survive in this extremely harsh and challenging environment. They grow extremely slowly, and are some of the oldest living organisms in the world. With some aged at almost 5,000 years these amazing trees can reveal information about Earth's climate variations. Amazingly, the leaves, or needles, can remain green for over 45 years." ;
    wo:class  ;
    wo:family  ;
    wo:genus  ;
    wo:growsIn ,  ;
    wo:kingdom  ;
    wo:name  ;
    wo:order  ;
    wo:phylum  ;
    a wo:Species ;
    rdfs:label "Great Basin bristlecone pine" ;
    owl:sameAs  ;
    foaf:depiction  .

And then the description of the clip that is related to the topic of Great Basin bristlecone pine:


    dc:title "Ancient bristlecones" ;
    po:subject  ;
    a po:Clip .

And we can follow our nose and fetch a description of the Ancient bristelcones clip:

rapper -o turtle http://www.bbc.co.uk/programmes/p005fs5p

Which tells us lots of stuff, like that it’s a documentary part of the science and nature genre, gives us a synopsis, and even links the clip to the episode and series it is a part of:


    dc:title "Ancient bristlecones" ;
    po:format  ;
    po:genre ,  ;
    po:long_synopsis """Bristlecone pines live at the limit of life, above 3,000m in the mountains of  western America. Almost continuous freezing temperatures and savage winds make life so tough, that these bristlecones only grow for six weeks of the year.

Everything is about conserving energy.They hardly ever shed their needles which can last more than 30 years. After centuries of being blasted by storms a full grown tree still survives with only a strip of bark a few inches wide.

These trees live life at such a slow pace they can reach a great age. Some are over 5,000 years old. It has been said of the bristlecones that to live here is to take a very long time to die.""" ;
    po:medium_synopsis "Living above 3,000 metres, North America's bristlecones cope with freezing temperatures and battering winds by only growing for six weeks of the year. But seeing as they may live for more than 5,000 years, that's still a fair bit of growing in a single lifetime. Slowly but surely does it..." ;
    po:short_synopsis "The world's oldest trees have survived 5,000 years of harsh conditions." ;
    po:version  ;
    a po:Clip .


    po:clip  ;
    a po:Series .


    po:clip  ;
    a po:Episode .

Conspicuously missing from this description is something like:


    dcterms:subject  .

But presumably it’s hiding underneath the covers in the Programmes database, and that’s what lets them link stuff up?

Also very interesting was Georgi Kobilarov‘s description of Uberblic. Since Georgi helped create dbpedia and is now consulting with the BBC, it seems like uberblic is positioning itself to provide a platform for the BBC to have it’s own local cache of the world of Linked Data. Having a local curated view of the world of linked data is something Dan Chudnov identified as a real need at the first Linked Data workshop at code4lib 2009 for caching and proxying linked data…so it is really cool to see solutions starting to appear in this space…and for them to be adopted by institutions like the BBC.

Georgi demo’d how an edit on wikipedia would be immediately reflected in the structured data available from uberblic. It was a real time update, and extremely impressive. It looks like part of the uberblic strategy is to crawl BBC’s web site and other pockets of Linked Data to enable the sort of linking across web properties that Tom described. I’d also surmise given the realtime nature of this that Georgi is bypassing dbpedia dumps and using the Wikipedia changes atom feed in conjunction with extractors that were built as part of the dbpedia project. But I’d love to know more of the mechanics of the update. It also would be interesting to know if uberblic has a notion of versions.

The really powerful message that the BBC is helping promote is this idea that good websites are APIs. Tom mentioned Paul Downey’s notion that Web APIs Are Just Web Sites. It’s a subtle but extremely important point that I learned primarily working closely with Dan Krech for a year or so. It’s an unfortunate side effect of lots market driven talk about web2.0, web3.0 and Linked Data in general that this simple REST message gets lost. We took it seriously in the design of the “API” at the Library of Congress’ Chronicling America. It’s also something I tried to talk about later in the week at dev8d when I had to quickly put a presentation together:

The slides probably won’t make much sense on their own, but the basic message was that we often hear about Linked Data in terms of pushing all your data to some triple store so you can start querying it with SPARQL and doing inferencing, and suddenly you’re going to be sitting pretty, totally jacked up on the Semantic Web.

If you are like me, you’ve already got databases where things are modeled, and you’ve created little web apps that have extracted information from the databases and put them on the web as HTML docs for people around the world to read (queue some mid 1990s grunge music). Expecting people to chuck away the applications and technology stacks they have simply to say they do Linked Data is wishful thinking. What’s missing is a simple migration strategy that would allow web publishers to easily recognize the value in publishing the contents of their database as Linked Data, and how it complements the HTML (and XML, JSON) publishing they are currently doing. My advice to folks at dev8d boiled down to:

Keep modelling your stuff how you like
Identify your stuff with Cool URIs in your webapps
Link your stuff together in HTML
Link to machine friendly formats (RSS, Atom, JSON, etc)
Use RDF to make your database available on the web using vocabularies other people understand.
Start thinking about technologies like SPARQL that will let you query pools and aggregated views of your data.
Consider joining the public-lod discussion list and joining the conversation.

I got some nice comments afterwards from Graham Klyne, Hugh Glaser, Adrian Stevenson and Mark Phillips so I felt pretty happy…granted most of the hard line Linked Data folks had already left a couple days earlier.

So some really exciting stuff is going on at the BBC. They are using Linked Data in a practical way that benefits their enterprise in real ways. I’m crossing my fingers and hoping that the value of what is going on here is recognized, and the various cuts that are going on won’t affect any of the fine work they are doing on improving the Web.

For more information check out the Semantic Web Case Study they folks at the BBC wrote summarizing their approach for the W3C.

data.australia.gov.au and rdfa

ed — Fri, 29 Jan 2010 14:36:57 +0000

In my previous blog post I was trying to demonstrate the virtues of data.gov.uk making the descriptions of their datasets available as RDFa. Just this morning I learned from Mark Birbeck that the folks down under at data.australia.gov.au did this last October!

For example this page describing a dataset for public Internet locations has this RDF metadata inside it:

 cc:attributionName "http://www.centrelink.gov.au/"@en-au ;
     cc:attributionURL  ;
     dc:coverage.geospatial "Australia"@en-au ;
     dc:coverage.temporal "Not specified"@en-au ;
     dc:creator "Centrelink"@en-au ;
     dc:date.modified "2009-08-31"^^xsd:date ;
     dc:date.published "2009-08-31"^^xsd:date ;
     dc:description """Location of Centrelink Offices
"""^^rdf:XMLLiteral ;
     dc:identifier "80"@en-au ;
     dc:keywords "Social Security"^^rdf:XMLLiteral ;
     dc:license "Creative Commons - Attribution 2.5 Australia (CC-BY)"^^rdf:XMLLiteral ;
     dc:source ""^^rdf:XMLLiteral ;
     dc:subject "Community,  Employment,  Government"^^rdf:XMLLiteral ;
     dc:title "Location of Centrelink Offices"@en-au ;
     dc:type  ;
     agls:jurisdiction "[Commonwealth of] Australia (AU)"@en-au ;

 dc:format "CSV"@en-au .

Now this data isn’t without problems: notice the XML literals as objects in the assertions involving subject, keyword, license and source? But it’s a Beta after all, and lots of us are learning this as we go, so Australia deserves a ton of credit. One really nice thing they are doing is making assertions about the format and URL location of the dataset itself. It would be even better if the dataset description was linked up with the dataset files using oai-ore or some other vocabulary.

In about 5 minutes I adapted the simplistic data.gov.uk crawler to crawl the data.australia.gov.au data. There aren’t as many datasets, so the crawler only pulled down 1725 triples (minus the xhtml triples)…but perhaps I missed some in my simplistic crawl.

Seeing both the data.gov.uk and data.australia.gov.au efforts to make dataset descriptions available makes me wonder if it could be useful for the W3C eGov Working Group to provide some lightweight guidance on how to make dataset descriptions available: what sorts of vocabularies to use, the kinds of assertions that are important, etc. It’s hard not to daydream of trying to provide an aggregated view of both pools of data, which is kept in synch using the web, and which perhaps could pull down aggregated datasets and archive them, etc. Perhaps a little spot checking tool that took at look at your HTML and let you know if it can work as a dataset description would be useful too?

data.gov.uk and rdfa

ed — Tue, 26 Jan 2010 22:41:19 +0000

The recent public release of the UK Government’s data.gov.uk site got picked up by the press last week in articles at The Guardian, Prospect Magazine and elswhere. These have been supplemented by some more technical discussions at ReadWriteWeb, Open Knowledge Foundation, Talis, Jeni Tennison’s blog, and some helpful emails from Leigh Dodds (Talis) and Jonathan Gray (Open Knowledge Foundation) on the w3c egovernment discussion list.

One thing that I haven’t seen mentioned so far in public (which I just discovered today) is that data.gov.uk is using RDFa to expose metadata about the datasets in a machine readable way. What this means is that in an HTML page for a dataset like this there are some extra HTML attributes like about, property, rel that have been thoughtfully used to express some structured metadata about the dataset, which can be extracted from the HTML and expressed say as Turtle:

 dct:coverage "Great Britain (England, Scotland, Wales)"@en ;
     dct:created "2009-12-04"@en ;
     dct:creator "Department for Environment, Food and Rural Affairs"@en ;
     dct:isReferencedBy  ;
     dct:license "Crown Copyright"@en ;
     dct:source ,  ;
     dct:subject
         ,
         ,
         ,
         ,
         ,
          .

In fact since data.gov.uk has a nice paging mechanism that lists all the datasets it’s not hard to write a little script that scrapes all the metadata for the datasets (35,478 triples) right out of the web pages.

I also noticed via Stéphane Corlosquet today that data.gov.uk is using the Drupal open-source content management system. To what extent Drupal7′s new RDFa features are being used to layer in this RDFa isn’t clear to me. But it is an exciting development. It’s exciting because data.gov.uk is a great example of how to bubble up data that’s typically locked away in databases of some kind into the HTML that’s out on the web for people to interact with, and for crawlers to crawl and re-purpose.

For example I can now write a utility to check the status of the external dataset links, to make sure they are they are there (200 OK). The complete results by URL can be summarized by rolling up by status code:

Status Code	Number of Datasets
200	2977
404	106
502	23
503	14
[Errno socket error] [Errno -2] Name or service not known	8
500	3
nonnumeric port: ”	1
[Errno socket error] [Errno 110] Connection timed out	1
400	1

Or I can generate a list of dataset subjects (eventhough it’s already available I guess). Here’s the top 25:

Subject	Number of Datasets
health	645
care	427
child	398
population	341
children	295
school	273
health-and-social-care	271
health-well-being-and-care	205
economy	202
economics-and-finance	189
census	188
education	176
communities	154
benefit	153
road	144
children-education-and-skills	121
people-and-places	111
government-receipts-and-expenditure	110
education-and-skills	110
housing	108
environment	107
tax	107
life-in-the-community	106
employment	105
tax-credit	96

I realize it’s early days but here are a few things it would be fun to see at data.gov.uk:

add some RDFa and SKOS or CommonTag in tag pages like education: this would allow things to be hooked up a bit more explicitly, tags to be given nice labels, and encourage the reuse of the tagging vocabulary within and outside data.gov.uk
link the dataset descriptions to the dataset resources themselves (the pdfs, excel spreadsheets, etc) that are online using a vocabulary like the Open Archives Reuse and Exchange and/or POWDER. This would allow for the harvesting and aggregation not only of the metadata, but the datasets as well.

I imagine much of this sort of hacking around can be enabled by querying the data.gov.uk SPARQL endpoint. But it hasn’t been very clear to me exactly what data is behind there. And there is something comforting about being able to crawl the open web to find the information that’s there in open to view.

Hacking O’Reilly RDFa

ed — Tue, 22 Dec 2009 18:17:03 +0000

I recently learned from Ivan Herman‘s blog that O’Reilly has begun publishing RDFa in their online catalog of books. So if you go and install the RDFa Highlight bookmarklet and then visit a page like this and click on the bookmarklet you’ll see something like:

Those red boxes you see are graphical depictions of where metadata can be found interleaved in the HTML. In my screenshot you can maybe barely see an assertion involving the title being displayed:

 dc:title "Natural Language Processing with Python"

But there is actually quite a lot of metadata hiding in the page, which can be found by running the page through the RDFa Distiller (quickly skim over this if your eyes glaze over when you see Turtle):

@prefix dc:  .
@prefix foaf:  .
@prefix frbr:  .
@prefix gr:  .
@prefix rdf:  .
@prefix rdfs:  .
@prefix xml:  .
@prefix xsd:  .

 a frbr:Expression ;
     dc:creator , ,  ;
     dc:issued "2009-06-12"^^xsd:dateTime ;
     dc:publisher "O'Reilly Media"@en ;
     dc:title "Natural Language Processing with Python"@en ;
     frbr:embodiment , ,  . 

 a gr:Offering ;
     gr:includesObject
         [ a gr:TypeAndQuantityNode ;
             gr:ammountOfThisGood "1"^^xsd:float ;
             gr:hasPriceSpecification
                 [ a gr:UnitPriceSpecification ;
                     gr:hasCurrency "GBP"@en ;
                     gr:hasCurrencyValue "34.50"^^xsd:float
                 ] ;
             gr:typeOfGood 
         ] . 

 a gr:Offering ;
     gr:includesObject
         [ a gr:TypeAndQuantityNode ;
             gr:ammountOfThisGood "1"^^xsd:float ;
             gr:typeOfGood 
         ] . 

 a gr:Offering ;
     gr:includesObject
         [ a gr:TypeAndQuantityNode ;
             gr:ammountOfThisGood "1"^^xsd:float ;
             gr:includesObject
                 [ a gr:TypeAndQuantityNode ;
                     gr:ammountOfThisGood "1"^^xsd:float ;
                     gr:hasPriceSpecification
                         [ a gr:UnitPriceSpecification ;
                             gr:hasCurrency "None"@en ;
                             gr:hasCurrencyValue "49.49"^^xsd:float
                         ] ;
                     gr:typeOfGood 
                 ] ;
             gr:typeOfGood 
         ] . 

 a gr:Offering ;
     gr:includesObject
         [ a gr:TypeAndQuantityNode ;
             gr:ammountOfThisGood "1"^^xsd:float ;
             gr:hasPriceSpecification
                 [ a gr:UnitPriceSpecification ;
                     gr:hasCurrency "USD"@en ;
                     gr:hasCurrencyValue "44.99"^^xsd:float
                 ] ;
             gr:typeOfGood 
         ] . 

 a gr:Offering ;
     gr:includesObject
         [ a gr:TypeAndQuantityNode ;
             gr:ammountOfThisGood "1"^^xsd:float ;
             gr:hasPriceSpecification
                 [ a gr:UnitPriceSpecification ;
                     gr:hasCurrency "USD"@en ;
                     gr:hasCurrencyValue "35.99"^^xsd:float
                 ] ;
             gr:typeOfGood 
         ] . 

 a foaf:Person ;
     foaf:homepage  ;
     foaf:name "Steven Bird"@en . 

 a foaf:Person ;
     foaf:homepage  ;
     foaf:name "Ewan Klein"@en . 

 a foaf:Person ;
     foaf:homepage  ;
     foaf:name "Edward Loper"@en . 

 a frbr:Manifestation ;
     dc:type  . 

 a frbr:Manifestation ;
     dc:identifier  ;
     dc:issued "2009-06-12"^^xsd:dateTime ;
     dc:type  . 

 a frbr:Manifestation ;
     dc:extent """512"""@en ;
     dc:identifier  ;
     dc:issued "2009-06-19"^^xsd:dateTime ;
     dc:type  .

So that’s a lot of data. The nice thing about rdf is that you can look at the vocabularies that are being used to get an idea of the rough shape of the underlying data. Just looking at the namespace prefixes we can see that O’Reilly has chosen to use the following vocabularies:

Dublin Core Terms: for indicating the publisher, title, authors, issue date and identifiers for a book
Friend of a Friend (FOAF): for modeling authors as People
Functional Requirements for Bibliographic Records (FRBR): for relating a particular book (Expression) to its various Manifestations of the title: ebook, printed book
Good Relations: for making pricing information available

I was curious so I wrote a little crawler (41 lines of Python+rdflib) to collect all the metadata from the O’Reilly Catalog pages. Yes all the pages! It ended up pulling down 92,101 triples, which I’ve made available as rdf/xml and ntriples files.

A nice side effect of having the data as a big ntriples file is you can do unix pipe tricks with sort, cut, uniq like this to get some ballpark numbers on what types of resources are in the rdf graph:

ed@curry:~/Projects/oreilly-crawler$ rdfsum catalog.nt
   6803 
   5861 
   4564 
   4065 
   2100 
   2023

Another nice thing about pulling the RDFa down with rdflib is you end up with a little berkeleydb triple store which you can query with SPARQL, say to pull out all the authors and titles:

    SELECT ?title ?author
    WHERE {
      ?title_uri dct:title ?title .
      ?title_uri dct:creator ?author_uri .
      ?author_uri foaf:name ?author .
    }

And adding a little bit of networkx judo you can get an xmas-friendly graph of authors (the green dots are books and the red ones are authors ; I limited author labels to authors who had written more than 2 books).

Admittedly this is not very readable, but I imagine someone with more network visualization skillz could do something nicer in short order. There’s a lot that could be done with the data. This exercise was mainly just to demonstrate how layering some new stuff into your HTML can really open up doors for how people use your website. Clearly O’Reilly did some deep thinking about what data they had, and what vocabularies they wanted to model it with. But once they’d done that they probably just had to go add 50 lines to an HTML template somewhere, and it was published (props to David Brunton for this turn of phrase). It’s a really good sign that a tech publisher with the stature of O’Reilly is giving this method of data publishing a try.

My only suggestion (for anyone at O’Reilly who might be reading) would be that it would be nice if they used HTTP URLs instead of URNs for People, Works and Expressions. I understand why they did it: using URNs eases deployment somewhat since you don’t have to worry about httpRange-14 stuff. But I think they could easily use a hash URI instead of an URN, and make it easy for people to link to their stuff in other data. The Cool URIs For the Semantic Web has some other patterns they might want to consider, but simply adding a hash to their existing page URIs should do the trick. So for example, consider if OpenLibrary wanted to link their notion of of a book to O’Reilly’s notion of a book with owl:sameAs. If they used they URN they’d have:

 owl:sameAs  .

but if O’Reilly identified their expressions with a URL they would enable something like:

 owl:sameAs  .

This may seem like a minor point, but it’s really important to be able to follow your nose on the web–particularly in Linked Data. If a piece of software ran across the O’Reilly URL in a chunk of OpenLibrary RDF, the program could HTTP GET it, and learn more stuff about the book. But if it got the URN it wouldn’t really know how to fetch a representation for that resource without some special case logic that mapped the URN to a URL. There is a reason why Tim Berners-Lee included the following as the second of his design principles for Linked Data:

Use HTTP URIs so that people can look up those names.

Anyhow, hats off to O’Reilly for putting RDFa into practice. I hope the rest of the publishing (and library world) take note. If you are looking to learn more about RDFa Ben Adida and Mark Birbeck‘s RDFa Primer: Bridging the Human and Data Webs is a really nice intro.

New York Times Topics as SKOS

ed — Wed, 19 Aug 2009 04:46:59 +0000

Serves 23,376 SKOS Concepts

INGREDIENTS

Text editor: Vim, Emacs, TextMate, etc
Python
BeautifulSoup
rdflib
Internet connection

DIRECTIONS

Open a new file using your favorite text editor.
Instantiate an RDF graph with a dash of rdflib.
Use python’s urllib to extract the HTML for each of the Times Topics Index Pages, e.g. for A.
Parse HTML into a fine, queryable data structure using BeautifulSoup.
Locate topic names and their associated URLs, and gently add them to the graph with a pinch of SKOS.
Go back to step 3 to fetch the next batch of topics, until you’ve finished Z.
Bake the RDF graph as an rdf/xml file.

NOTES

If you don’t feel like cooking up the rdf/xml yourself you can download it from here (might want to right-click to download, some browsers might have trouble rendering the xml), or download the 68 line implementation and run it yourself.

The point of this exercise was mainly to show how thinking of the New York Times Topics as a controlled vocabulary, that can be serialized as a file, and still present on the Web, could be useful. Perhaps to someone writing an application that needs to integrate with the New York Times and who want to be able to tag content using the same controlled vocabulary. Or perhaps someone wants to be able to link your own content with similar content at the New York Times. These are all use cases for expressing the Topics as SKOS, and being able to ship it around with resolvable identifiers for the concepts.

Of course there is one slight wrinkle. Take a look at this Turtle snippet for the concept of Ray Bradbury:

@prefix rdf:  .
@Prefix skos:  .

 a skos:Concept;
    skos:prefLabel "Bradbury, Ray";
    skos:broader ;
    skos:inScheme 
    .

Notice the URI being used for the concept?


http://topics.nytimes.com/top/reference/timestopics/people/b/ray_bradbury#concept

The wrinkle is that there’s no way to get RDF back from this URI currently. But since NYT is already using XHTML, it wouldn’t be hard to sprinkle in some RDFa such that:


...
Ray Bradbury
...

And voila you’ve got Linked Data. I took the 5 minutes to mark up the HTML myself and put it here which you can run through the RDFa Distiller to get some Turtle. Of course if the NYT ever decided to alter their HTML to provide this markup this recipe would be simplified greatly: no more error prone scraping, the assertions could be pulled directly out of the HTML.

public.resource.org to liberate Code of Federal Regulations

ed — Wed, 17 Sep 2008 21:51:15 +0000

good news via the govtrack mailing list

Carl Malamud of public.resource.org, with funding from a bunch of places including a small bit from GovTrack’s ad profits, announced his intention to purchase from the Government Printing Office documents they produce in the course of their statutory obligations and then have the nerve to sell back to the public at prohibitive prices. The document to be purchased is the Code of Federal Regulations, the component of federal law created by executive branch agencies, in electronic form. Once obtained, it will be posted openly/freely online.

More here: http://public.resource.org/gpo.gov/index.html

And Carl’s letter to the GPO:
http://public.resource.org/gpo.gov/the_honorable.html

It’s pretty sad that it has to come to this…but it’s also pretty awesome that it’s happening.

provide and enable

ed — Wed, 18 Jun 2008 18:05:24 +0000

I got a chance to meet Jennifer Rigby of the National Archives UK at the LinkedDataPlanet Conference in New York City (thanks Ian). Jennifer is the Head of IT Strategy, and told me lots of interesting stuff related to a profound shift they’ve had in their online strategies to:

Provide and Enable

So rather than pouring all their energy into making applications to visualize archival resources, the National Archives have recognized that making machine readable resources available to the public (in formats like RDF and RDFa) is really important to their core mission. In addition to providing services and data, they are trying to enable an ecosystem of innovation around their assets–or in their words:

• We will allow others to harness the power of our information, leading to a far wider range of products and services than we could provide ourselves.
• We will continue to work with commercial partners to provide online access to millions of records.

Jennifer said we can look forward to an announcement around OpenTech2008 (July 5th) about a set of important publications that are going to made available by the Archives as RDF and RDFa. In addition I heard about how they work with website data harvested by Internet Archive to create a resolver service for transient publications on the web.

Hearing how a big organization like the National Archives can come to this realization of “Provide and Enable”, and then start to execute on it was really encouraging–and inspiring. It is also refreshing to see people recognize, in writing the importance of semantic web technologies:

We have started exploring new ideas and technologies, including using RDFa for publishing the Gazettes. The way we now publish legislation has a key role to play in the further development of the semantic web.

justify my links

ed — Thu, 29 May 2008 12:58:07 +0000

Thanks to a tip from Ian, I’m looking forward to (hopefully) attending the Linked Data Planet conference in New York City as a volunteer. The idea is that I just have to pay for my hotel, and the cost of admission is waived. It seems my travel money is a bit limited at the moment (sometimes it’s there, sometimes it isn’t), so I figured minimizing costs would be appreciated. But today I got a request to “justify” my attendance at the conference. It was actually kind of a good exercise to sit down and write why I think the conference and Linked Data in general is important to the Library of Congress.

One of the challenges of Digital Repository work is modeling the context for digital objects. The context for a digital object includes the set of relationships a particular digital object has with other objects in the repository. 30 years of relational database research and development have allowed us to do this modeling pretty effectively within the scope of a particular application.

Very often, particularly in institutions the size of the Library of Congress, the context for a digital object includes digital objects found elsewhere in the enterprise–in other applications, with their own databases. In addition some institutions (like LC) also need to make their digital resources available publicly for other organizations to reference. The challenge here is in making the objects found in silos or islands of application data (typically housed in databases) reference-able and resolvable, so that other applications inside and outside the enterprise can use them.

As a practical example, a picture of Dizzie Gilliespie found in the America Memory collection

is related to the book:

To be, or not–to bop: memoirs / Dizzy Gillespie, with Al Fraser.

which we have described in our online catalog. The person Dizzy Gillespie is also represented in LC’s name authority file with the Library of Congress Control Number n50033872, and the Linked Authority File at OCLC. And perhaps this picture of Dizzie Gillespie in American Memory will find it’s way into the World Digital Library application that is currently being built. How can we practically and explicitly identify and then represent the relationships between these resources? Is it even possible?

The Linked Data Planet conference is a two day workshop describing how to use traditional web technologies in conjunction with semantic web technologies (RDF, OWL, SPARQL, RDFa and GRDDL) to enable this sort of linking of resources inside particular applications, within the enterprise and around the world. My hope is that the conference will provide guidance on simple things LC can do with web technologies that have been in use for 20 years, to model the relationships between digital resources at the Library of Congress.

Hopefully that will convince them :-)

Apologies to Madonna for the blog post title…

permalinks reloaded

ed — Mon, 17 Dec 2007 21:17:07 +0000

The recently announced Zotero / InternetArchive partnership is exciting on a bunch of levels. The one that immediately struck me was the use of the Internet Archive URI. As you may have noticed before all the content in Internet Archive Wayback Machine can be referenced with a URL that looks something like:

http://web.archive.org/web/{yyyymmddhhmmss}/{url}

Where url is the document URL you want to look up in the archive at the given time. So for example:

http://web.archive.org/web/19981202230410/http://www.google.com/

is a URL for what http://google.com looked like on December 02, 1998 at 23:04:10. Perhaps this is documented somewhere prominent or is common knowledge, but it looks like you can play with the timestamp, and archive.org will adjust as needed, redirecting you to the closest snapshot it can find:

and even:

http://web.archive.org/web/http://www.google.com/

which redirects to the most recent content for a given URL. It’s just a good old 302 at work:

ed@curry:~$ curl -I http://web.archive.org/web/199812/http://www.google.com/
HTTP/1.1 302 Found
Date: Mon, 17 Dec 2007 21:11:12 GMT
Server: Apache/2.0.54 (Ubuntu) PHP/5.0.5-2ubuntu1.2 mod_ssl/2.0.54 OpenSSL/0.9.7g mod_perl/2.0.1 Perl/v5.8.7
Location: http://web.archive.org/web/19981202230410/www.google.com/
Content-Type: text/html; charset=iso-8859-1

So anyhow, pretty cool use of URIs and HTTP right? The addition of zotero to the mix will mean that scholars can cite the web as it appeared at a particular point in time:

… as scholars begin to use not only traditional primary sources that have been digitized but also “born digital” materials on the web (blogs, online essays, documents transcribed into HTML), the possibility arises for Zotero users to leverage the resources of IA to ensure a more reliable form of scholarly communication. One of the Internet Archive’s great strengths is that it has not only archived the web but also given each page a permanent URI that includes a time and date stamp in addition to the URL.

Currently when a scholar using Zotero wishes to save a web page for their research they simply store a local copy. For some, perhaps many, purposes this is fine. But for web documents that a scholar believes will be important to share, cite, or collaboratively annotate (e.g., among a group of coauthors of an article or book) we will provide a second option in the Zotero web save function to grab a permanent copy and URI from IA’s web archive. A scholar who shares this item in their library can then be sure that all others who choose to use it will be referring to the exact same document.

This is pretty fundamental to scholarship on the web. Of course when generating a time anchored permalink with zotero one can well expect that archive.org will on occasion not have a snapshot of said content, resulting in a 404. It would be great if archive.org could leverage these requests for snapshots as requests to go out and archive the page. One could imagine a blocking and nonblocking request: the former which would spawn a request to fetch a particular URI, stash content away, and return the permalink; and the latter which would just quickly return the best match its already got (which may be a 404).

Anyhow, it’s really good to see these two outfits working together. Nice work!

ps. dear lazyweb is there a documented archive.org api available?