I was also asked how easy it would be to generate the .gov and .edu graphs, as well as top level domains. I already had the hostnames exported, so it was just a few greps, sorts and uniqs away. I’ve included the graphs and the full data files below. My friend Dan Chudnov suggested that plotting this data on a logarithmic scale would probably look better. I think he’s probably right, but I just haven’t gotten around to doing that yet. It’s definitely something I will keep in mind for an app that allowed this slicing and dicing of the Wikipedia external links data.

top 100 .edu hosts in en.wikipedia external links

download

top 100 .gov hosts in en.wikipedia external links

download

top 100 top level domains in en.wikipedia external links

download

Now I might be greedy, but when I saw that 1 million epubs are available my mind immediately jumps to thinking of getting them, indexing them and whatnot. Then I guiltily justified my greedy thoughts by pondering the conventional digital preservation wisdom that Lots of Copies Keeps Stuff Safe (LOCKSS). The books are in the public domain, so …. why not?

Google Books has a really nice API, which lets you get back search results as Atom, with lots of links to things like thumbnails, annotations, item views, etc. You also get a nice amount of Dublin Core metadata. And you can limit your search to books published before 1923. For example here’s a search for pre-1923 books that mention “Stevenson” (disclaimer: I don’t think the 1923 limit is actually working):

curl 'http://books.google.com/books/feeds/volumes?tbs=cd_max:Jan%2001_2%201923&q=Stevenson' | xmllint --format -

which yields:

< ?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:openSearch="http://a9.com/-/spec/opensearchrss/1.0/" xmlns:gbs="http://schemas.google.com/books/2008" xmlns:dc="http://purl.org/dc/terms" xmlns:batch="http://schemas.google.com/gdata/batch" xmlns:gd="http://schemas.google.com/g/2005">
  <id>http://www.google.com/books/feeds/volumes</id>
  <updated>2010-08-30T20:37:27.000Z</updated>
  <category scheme="http://schemas.google.com/g/2005#kind" term="http://schemas.google.com/books/2008#volume"/>
  <title type="text">Search results for Stevenson</title>
  <link rel="alternate" type="text/html" href="http://www.google.com"/>
  <link rel="http://schemas.google.com/g/2005#feed" type="application/atom+xml" href="http://www.google.com/books/feeds/volumes"/>
  <link rel="self" type="application/atom+xml" href="http://www.google.com/books/feeds/volumes?q=Stevenson"/>
  <link rel="next" type="application/atom+xml" href="http://www.google.com/books/feeds/volumes?q=Stevenson&amp;start-index=11&amp;max-results=10"/>
  <author>
    <name>Google Books Search</name>
    <uri>http://www.google.com</uri>
  </author>
  <generator version="beta">Google Book Search data API</generator>
  <opensearch:totalresults>206</opensearch:totalresults>
  <opensearch:startindex>1</opensearch:startindex>
  <opensearch:itemsperpage>10</opensearch:itemsperpage>
  <entry>
    <id>http://www.google.com/books/feeds/volumes/ENMWAAAAYAAJ</id>
    <updated>2010-08-30T20:37:27.000Z</updated>
    <category scheme="http://schemas.google.com/g/2005#kind" term="http://schemas.google.com/books/2008#volume"/>
    <title type="text">Kidnapped</title>
    <link rel="http://schemas.google.com/books/2008/thumbnail" type="image/x-unknown" href="http://bks4.books.google.com/books?id=ENMWAAAAYAAJ&amp;printsec=frontcover&amp;img=1&amp;zoom=5&amp;edge=curl&amp;sig=ACfU3U0dHqcPyQYmX-QBDY7DqGofAlUvvw&amp;source=gbs_gdata"/>
    <link rel="http://schemas.google.com/books/2008/info" type="text/html" href="http://books.google.com/books?id=ENMWAAAAYAAJ&amp;dq=Stevenson&amp;ie=ISO-8859-1&amp;source=gbs_gdata"/>
    <link rel="http://schemas.google.com/books/2008/preview" type="text/html" href="http://books.google.com/books?id=ENMWAAAAYAAJ&amp;printsec=frontcover&amp;dq=Stevenson&amp;ie=ISO-8859-1&amp;cd=1&amp;source=gbs_gdata"/>
    <link rel="http://schemas.google.com/books/2008/annotation" type="application/atom+xml" href="http://www.google.com/books/feeds/users/me/volumes"/>
    <link rel="http://schemas.google.com/books/2008/acsepubfulfillmenttoken" type="application/vnd.adobe.adept+xml" href="http://books.google.com/books/download/Kidnapped.acsm?id=ENMWAAAAYAAJ&amp;format=epub&amp;output=acs4_fulfillment_token"/>
    <link rel="alternate" type="text/html" href="http://books.google.com/books?id=ENMWAAAAYAAJ&amp;dq=Stevenson&amp;ie=ISO-8859-1"/>
    <link rel="self" type="application/atom+xml" href="http://www.google.com/books/feeds/volumes/ENMWAAAAYAAJ"/>
    <gbs:embeddability value="http://schemas.google.com/books/2008#embeddable"/>
    <gbs:openaccess value="http://schemas.google.com/books/2008#enabled"/>
    <gbs:viewability value="http://schemas.google.com/books/2008#view_all_pages"/>
    <dc:creator>Robert Louis Stevenson</dc:creator>
    <dc:date>1909</dc:date>
    <dc:format>308 pages</dc:format>
    <dc:format>book</dc:format>
    <dc:identifier>ENMWAAAAYAAJ</dc:identifier>
    <dc:identifier>HARVARD:HN1JZ9</dc:identifier>
    <dc:title>Kidnapped</dc:title>
    <dc:title>being memoirs of the adventures of David Balfour in the year 1851 ...</dc:title>
  </entry>
  <entry>
    <id>http://www.google.com/books/feeds/volumes/WZ0vAAAAMAAJ</id>
    <updated>2010-08-30T20:37:27.000Z</updated>
    <category scheme="http://schemas.google.com/g/2005#kind" term="http://schemas.google.com/books/2008#volume"/>
    <title type="text">Treasure Island</title>
    <link rel="http://schemas.google.com/books/2008/thumbnail" type="image/x-unknown" href="http://bks8.books.google.com/books?id=WZ0vAAAAMAAJ&amp;printsec=frontcover&amp;img=1&amp;zoom=5&amp;edge=curl&amp;sig=ACfU3U3AusJAerfzUTAoo8iLZkeOL755rg&amp;source=gbs_gdata"/>
    <link rel="http://schemas.google.com/books/2008/info" type="text/html" href="http://books.google.com/books?id=WZ0vAAAAMAAJ&amp;dq=Stevenson&amp;ie=ISO-8859-1&amp;source=gbs_gdata"/>
    <link rel="http://schemas.google.com/books/2008/preview" type="text/html" href="http://books.google.com/books?id=WZ0vAAAAMAAJ&amp;printsec=frontcover&amp;dq=Stevenson&amp;ie=ISO-8859-1&amp;cd=2&amp;source=gbs_gdata"/>
    <link rel="http://schemas.google.com/books/2008/annotation" type="application/atom+xml" href="http://www.google.com/books/feeds/users/me/volumes"/>
    <link rel="http://schemas.google.com/books/2008/acsepubfulfillmenttoken" type="application/vnd.adobe.adept+xml" href="http://books.google.com/books/download/Treasure_Island.acsm?id=WZ0vAAAAMAAJ&amp;format=epub&amp;output=acs4_fulfillment_token"/>
    <link rel="alternate" type="text/html" href="http://books.google.com/books?id=WZ0vAAAAMAAJ&amp;dq=Stevenson&amp;ie=ISO-8859-1"/>
    <link rel="self" type="application/atom+xml" href="http://www.google.com/books/feeds/volumes/WZ0vAAAAMAAJ"/>
    <gbs:embeddability value="http://schemas.google.com/books/2008#embeddable"/>
    <gbs:openaccess value="http://schemas.google.com/books/2008#enabled"/>
    <gbs:viewability value="http://schemas.google.com/books/2008#view_all_pages"/>
    <dc:creator>Robert Louis Stevenson</dc:creator>
    <dc:creator>George Edmund Varian</dc:creator>
    <dc:date>1918</dc:date>
    <dc:description>CHAPTER I THE OLD SEA DOG AT THE &amp;quot;ADMIRAL BENBOW&amp;quot; SQUIRE Trelawney, Dr. Livesey, 
and the rest of these gentlemen having asked me to write down the whole ...</dc:description>
    <dc:format>306 pages</dc:format>
    <dc:format>book</dc:format>
    <dc:identifier>WZ0vAAAAMAAJ</dc:identifier>
    <dc:identifier>NYPL:33433075793830</dc:identifier>
    <dc:subject>Fiction</dc:subject>
    <dc:title>Treasure Island</dc:title>
  </entry>
  <entry>
    <id>http://www.google.com/books/feeds/volumes/REUrAQAAIAAJ</id>
    <updated>2010-08-30T20:37:27.000Z</updated>
    <category scheme="http://schemas.google.com/g/2005#kind" term="http://schemas.google.com/books/2008#volume"/>
    <title type="text">Stevenson</title>
    <link rel="http://schemas.google.com/books/2008/thumbnail" type="image/x-unknown" href="http://bks6.books.google.com/books?id=REUrAQAAIAAJ&amp;printsec=frontcover&amp;img=1&amp;zoom=5&amp;sig=ACfU3U2pRkGJW1MpjNMZgSc6jDIp3HuT0Q&amp;source=gbs_gdata"/>
    <link rel="http://schemas.google.com/books/2008/info" type="text/html" href="http://books.google.com/books?id=REUrAQAAIAAJ&amp;dq=Stevenson&amp;ie=ISO-8859-1&amp;source=gbs_gdata"/>
    <link rel="http://schemas.google.com/books/2008/preview" type="text/html" href="http://books.google.com/books?id=REUrAQAAIAAJ&amp;q=Stevenson&amp;dq=Stevenson&amp;ie=ISO-8859-1&amp;cd=3&amp;source=gbs_gdata"/>
    <link rel="http://schemas.google.com/books/2008/annotation" type="application/atom+xml" href="http://www.google.com/books/feeds/users/me/volumes"/>
    <link rel="http://schemas.google.com/books/2008/acsepubfulfillmenttoken" type="application/vnd.adobe.adept+xml" href="http://books.google.com/books/download/Stevenson.acsm?id=REUrAQAAIAAJ&amp;format=epub&amp;output=acs4_fulfillment_token"/>
    <link rel="alternate" type="text/html" href="http://books.google.com/books?id=REUrAQAAIAAJ&amp;dq=Stevenson&amp;ie=ISO-8859-1"/>
    <link rel="self" type="application/atom+xml" href="http://www.google.com/books/feeds/volumes/REUrAQAAIAAJ"/>
    <gbs:embeddability value="http://schemas.google.com/books/2008#not_embeddable"/>
    <gbs:openaccess value="http://schemas.google.com/books/2008#disabled"/>
    <gbs:viewability value="http://schemas.google.com/books/2008#view_no_pages"/>
    <dc:creator>Adlai Ewing Stevenson</dc:creator>
    <dc:creator>Grace Darling</dc:creator>
    <dc:creator>David Darling</dc:creator>
    <dc:date>1977-10</dc:date>
    <dc:format>127 pages</dc:format>
    <dc:format>book</dc:format>
    <dc:identifier>REUrAQAAIAAJ</dc:identifier>
    <dc:identifier>STANFORD:36105037014342</dc:identifier>
    <dc:publisher>McGraw-Hill/Contemporary</dc:publisher>
    <dc:subject>Biography &amp; Autobiography</dc:subject>
    <dc:title>Stevenson</dc:title>
  </entry>
  <entry>
    <id>http://www.google.com/books/feeds/volumes/3ibdGgAACAAJ</id>
    <updated>2010-08-30T20:37:27.000Z</updated>
    <category scheme="http://schemas.google.com/g/2005#kind" term="http://schemas.google.com/books/2008#volume"/>
    <title type="text">Stevenson</title>
    <link rel="http://schemas.google.com/books/2008/thumbnail" type="image/x-unknown" href="http://bks1.books.google.com/books?id=3ibdGgAACAAJ&amp;printsec=frontcover&amp;img=1&amp;zoom=5&amp;sig=ACfU3U2ryyj2yW3FOO9-8WPQtihovQqQOA&amp;source=gbs_gdata"/>
    <link rel="http://schemas.google.com/books/2008/info" type="text/html" href="http://books.google.com/books?id=3ibdGgAACAAJ&amp;dq=Stevenson&amp;ie=ISO-8859-1&amp;source=gbs_gdata"/>
    <link rel="http://schemas.google.com/books/2008/preview" type="text/html" href="http://books.google.com/books?id=3ibdGgAACAAJ&amp;dq=Stevenson&amp;ie=ISO-8859-1&amp;cd=4&amp;source=gbs_gdata"/>
    <link rel="http://schemas.google.com/books/2008/annotation" type="application/atom+xml" href="http://www.google.com/books/feeds/users/me/volumes"/>
    <link rel="http://schemas.google.com/books/2008/acsepubfulfillmenttoken" type="application/vnd.adobe.adept+xml" href="http://books.google.com/books/download/Stevenson.acsm?id=3ibdGgAACAAJ&amp;format=epub&amp;output=acs4_fulfillment_token"/>
    <link rel="alternate" type="text/html" href="http://books.google.com/books?id=3ibdGgAACAAJ&amp;dq=Stevenson&amp;ie=ISO-8859-1"/>
    <link rel="self" type="application/atom+xml" href="http://www.google.com/books/feeds/volumes/3ibdGgAACAAJ"/>
    <gbs:embeddability value="http://schemas.google.com/books/2008#not_embeddable"/>
    <gbs:openaccess value="http://schemas.google.com/books/2008#disabled"/>
    <gbs:viewability value="http://schemas.google.com/books/2008#view_no_pages"/>
    <dc:creator>Robert Louis Stevenson</dc:creator>
    <dc:date>2007-01-17</dc:date>
    <dc:description>This scarce antiquarian book is included in our special Legacy Reprint Series.</dc:description>
    <dc:format>128 pages</dc:format>
    <dc:format>book</dc:format>
    <dc:identifier>3ibdGgAACAAJ</dc:identifier>
    <dc:identifier>ISBN:1430495375</dc:identifier>
    <dc:identifier>ISBN:9781430495376</dc:identifier>
    <dc:publisher>Kessinger Pub Co</dc:publisher>
    <dc:subject>Poetry</dc:subject>
    <dc:title>Stevenson</dc:title>
    <dc:title>Day by Day</dc:title>
  </entry>
  <entry>
    <id>http://www.google.com/books/feeds/volumes/3QI-AAAAYAAJ</id>
    <updated>2010-08-30T20:37:27.000Z</updated>
    <category scheme="http://schemas.google.com/g/2005#kind" term="http://schemas.google.com/books/2008#volume"/>
    <title type="text">A child's garden of verses</title>
    <link rel="http://schemas.google.com/books/2008/thumbnail" type="image/x-unknown" href="http://bks0.books.google.com/books?id=3QI-AAAAYAAJ&amp;printsec=frontcover&amp;img=1&amp;zoom=5&amp;edge=curl&amp;sig=ACfU3U1KKV9Y2E1jnnc83b4NA_GNAouwbA&amp;source=gbs_gdata"/>
    <link rel="http://schemas.google.com/books/2008/info" type="text/html" href="http://books.google.com/books?id=3QI-AAAAYAAJ&amp;dq=Stevenson&amp;ie=ISO-8859-1&amp;source=gbs_gdata"/>
    <link rel="http://schemas.google.com/books/2008/preview" type="text/html" href="http://books.google.com/books?id=3QI-AAAAYAAJ&amp;printsec=frontcover&amp;dq=Stevenson&amp;ie=ISO-8859-1&amp;cd=5&amp;source=gbs_gdata"/>
    <link rel="http://schemas.google.com/books/2008/annotation" type="application/atom+xml" href="http://www.google.com/books/feeds/users/me/volumes"/>
    <link rel="http://schemas.google.com/books/2008/acsepubfulfillmenttoken" type="application/vnd.adobe.adept+xml" href="http://books.google.com/books/download/A_child_s_garden_of_verses.acsm?id=3QI-AAAAYAAJ&amp;format=epub&amp;output=acs4_fulfillment_token"/>
    <link rel="alternate" type="text/html" href="http://books.google.com/books?id=3QI-AAAAYAAJ&amp;dq=Stevenson&amp;ie=ISO-8859-1"/>
    <link rel="self" type="application/atom+xml" href="http://www.google.com/books/feeds/volumes/3QI-AAAAYAAJ"/>
    <gbs:embeddability value="http://schemas.google.com/books/2008#embeddable"/>
    <gbs:openaccess value="http://schemas.google.com/books/2008#enabled"/>
    <gbs:viewability value="http://schemas.google.com/books/2008#view_all_pages"/>
    <dc:creator>Robert Louis Stevenson</dc:creator>
    <dc:date>1914</dc:date>
    <dc:description>IN winter I get up at night And dress by yellow candle-light. In summer, quite 
the other way, I have to go to bed by day. I have to go to bed and see The ...</dc:description>
    <dc:format>136 pages</dc:format>
    <dc:format>book</dc:format>
    <dc:identifier>3QI-AAAAYAAJ</dc:identifier>
    <dc:identifier>CORNELL:31924052752262</dc:identifier>
    <dc:subject>Children's poetry, Scottish</dc:subject>
    <dc:title>A child's garden of verses</dc:title>
    <dc:title>by Robert Louis Stevenson; illustrated by Charles Robinson</dc:title>
  </entry>
  <entry>
    <id>http://www.google.com/books/feeds/volumes/Gmk-AAAAYAAJ</id>
    <updated>2010-08-30T20:37:27.000Z</updated>
    <category scheme="http://schemas.google.com/g/2005#kind" term="http://schemas.google.com/books/2008#volume"/>
    <title type="text">Travels with a donkey in the Cevennes</title>
    <link rel="http://schemas.google.com/books/2008/thumbnail" type="image/x-unknown" href="http://bks5.books.google.com/books?id=Gmk-AAAAYAAJ&amp;printsec=frontcover&amp;img=1&amp;zoom=5&amp;edge=curl&amp;sig=ACfU3U1CM_jn5vHlfF0va-U32zu2Rr2QIg&amp;source=gbs_gdata"/>
    <link rel="http://schemas.google.com/books/2008/info" type="text/html" href="http://books.google.com/books?id=Gmk-AAAAYAAJ&amp;dq=Stevenson&amp;ie=ISO-8859-1&amp;source=gbs_gdata"/>
    <link rel="http://schemas.google.com/books/2008/preview" type="text/html" href="http://books.google.com/books?id=Gmk-AAAAYAAJ&amp;printsec=frontcover&amp;dq=Stevenson&amp;ie=ISO-8859-1&amp;cd=6&amp;source=gbs_gdata"/>
    <link rel="http://schemas.google.com/books/2008/annotation" type="application/atom+xml" href="http://www.google.com/books/feeds/users/me/volumes"/>
    <link rel="http://schemas.google.com/books/2008/acsepubfulfillmenttoken" type="application/vnd.adobe.adept+xml" href="http://books.google.com/books/download/Travels_with_a_donkey_in_the_Cevennes.acsm?id=Gmk-AAAAYAAJ&amp;format=epub&amp;output=acs4_fulfillment_token"/>
    <link rel="alternate" type="text/html" href="http://books.google.com/books?id=Gmk-AAAAYAAJ&amp;dq=Stevenson&amp;ie=ISO-8859-1"/>
    <link rel="self" type="application/atom+xml" href="http://www.google.com/books/feeds/volumes/Gmk-AAAAYAAJ"/>
    <gbs:embeddability value="http://schemas.google.com/books/2008#embeddable"/>
    <gbs:openaccess value="http://schemas.google.com/books/2008#enabled"/>
    <gbs:viewability value="http://schemas.google.com/books/2008#view_all_pages"/>
    <dc:creator>Robert Louis Stevenson</dc:creator>
    <dc:date>1916</dc:date>
    <dc:description>THE DONKEY, THE PACK, AND THE PACK - SADDLE IN a little place called Le 
Monastier, in a pleasant highland valley fifteen miles from Le Puy, I spent 
about a ...</dc:description>
    <dc:format>287 pages</dc:format>
    <dc:format>book</dc:format>
    <dc:identifier>Gmk-AAAAYAAJ</dc:identifier>
    <dc:identifier>HARVARD:HWP541</dc:identifier>
    <dc:subject>Cévennes Mountains (France)</dc:subject>
    <dc:title>Travels with a donkey in the Cevennes</dc:title>
    <dc:title>An inland voyage</dc:title>
  </entry>
  <entry>
    <id>http://www.google.com/books/feeds/volumes/f3A-AAAAYAAJ</id>
    <updated>2010-08-30T20:37:27.000Z</updated>
    <category scheme="http://schemas.google.com/g/2005#kind" term="http://schemas.google.com/books/2008#volume"/>
    <title type="text">St. Ives</title>
    <link rel="http://schemas.google.com/books/2008/thumbnail" type="image/x-unknown" href="http://bks6.books.google.com/books?id=f3A-AAAAYAAJ&amp;printsec=frontcover&amp;img=1&amp;zoom=5&amp;edge=curl&amp;sig=ACfU3U0HtuxZqcqgneZTtM0LH5rk4F2E4w&amp;source=gbs_gdata"/>
    <link rel="http://schemas.google.com/books/2008/info" type="text/html" href="http://books.google.com/books?id=f3A-AAAAYAAJ&amp;dq=Stevenson&amp;ie=ISO-8859-1&amp;source=gbs_gdata"/>
    <link rel="http://schemas.google.com/books/2008/preview" type="text/html" href="http://books.google.com/books?id=f3A-AAAAYAAJ&amp;printsec=frontcover&amp;dq=Stevenson&amp;ie=ISO-8859-1&amp;cd=7&amp;source=gbs_gdata"/>
    <link rel="http://schemas.google.com/books/2008/annotation" type="application/atom+xml" href="http://www.google.com/books/feeds/users/me/volumes"/>
    <link rel="http://schemas.google.com/books/2008/acsepubfulfillmenttoken" type="application/vnd.adobe.adept+xml" href="http://books.google.com/books/download/St__Ives.acsm?id=f3A-AAAAYAAJ&amp;format=epub&amp;output=acs4_fulfillment_token"/>
    <link rel="alternate" type="text/html" href="http://books.google.com/books?id=f3A-AAAAYAAJ&amp;dq=Stevenson&amp;ie=ISO-8859-1"/>
    <link rel="self" type="application/atom+xml" href="http://www.google.com/books/feeds/volumes/f3A-AAAAYAAJ"/>
    <gbs:embeddability value="http://schemas.google.com/books/2008#embeddable"/>
    <gbs:openaccess value="http://schemas.google.com/books/2008#enabled"/>
    <gbs:viewability value="http://schemas.google.com/books/2008#view_all_pages"/>
    <dc:creator>Robert Louis Stevenson</dc:creator>
    <dc:date>1906</dc:date>
    <dc:description>IVES CHAPTER IA TALE OF A LION RAMPANT IT was in the month of May,, that I was 
so unlucky as to fall at last into the hands of the enemy. ...</dc:description>
    <dc:format>528 pages</dc:format>
    <dc:format>book</dc:format>
    <dc:identifier>f3A-AAAAYAAJ</dc:identifier>
    <dc:identifier>HARVARD:HWP61W</dc:identifier>
    <dc:title>St. Ives</dc:title>
    <dc:title>being the adventures of a French prisoner in England</dc:title>
  </entry>
  <entry>
    <id>http://www.google.com/books/feeds/volumes/4mb8LuKKwocC</id>
    <updated>2010-08-30T20:37:27.000Z</updated>
    <category scheme="http://schemas.google.com/g/2005#kind" term="http://schemas.google.com/books/2008#volume"/>
    <title type="text">Cruising with Robert Louis Stevenson</title>
    <link rel="http://schemas.google.com/books/2008/thumbnail" type="image/x-unknown" href="http://bks6.books.google.com/books?id=4mb8LuKKwocC&amp;printsec=frontcover&amp;img=1&amp;zoom=5&amp;edge=curl&amp;sig=ACfU3U3VC0jTy4rEC4rjtFjLofSY7bAf2Q&amp;source=gbs_gdata"/>
    <link rel="http://schemas.google.com/books/2008/info" type="text/html" href="http://books.google.com/books?id=4mb8LuKKwocC&amp;dq=Stevenson&amp;ie=ISO-8859-1&amp;source=gbs_gdata"/>
    <link rel="http://schemas.google.com/books/2008/preview" type="text/html" href="http://books.google.com/books?id=4mb8LuKKwocC&amp;printsec=frontcover&amp;dq=Stevenson&amp;ie=ISO-8859-1&amp;cd=8&amp;source=gbs_gdata"/>
    <link rel="http://schemas.google.com/books/2008/annotation" type="application/atom+xml" href="http://www.google.com/books/feeds/users/me/volumes"/>
    <link rel="http://schemas.google.com/books/2008/acsepubfulfillmenttoken" type="application/vnd.adobe.adept+xml" href="http://books.google.com/books/download/Cruising_with_Robert_Louis_Stevenson.acsm?id=4mb8LuKKwocC&amp;format=epub&amp;output=acs4_fulfillment_token"/>
    <link rel="alternate" type="text/html" href="http://books.google.com/books?id=4mb8LuKKwocC&amp;dq=Stevenson&amp;ie=ISO-8859-1"/>
    <link rel="self" type="application/atom+xml" href="http://www.google.com/books/feeds/volumes/4mb8LuKKwocC"/>
    <gbs:embeddability value="http://schemas.google.com/books/2008#embeddable"/>
    <gbs:openaccess value="http://schemas.google.com/books/2008#disabled"/>
    <gbs:viewability value="http://schemas.google.com/books/2008#view_partial"/>
    <dc:creator>Oliver S. Buckton</dc:creator>
    <dc:date>2007</dc:date>
    <dc:description>Cruising with Robert Louis Stevenson: Travel, Narrative, and the Colonial Body is the first book-length study about the influence of travel on Robert Louis ...</dc:description>
    <dc:format>344 pages</dc:format>
    <dc:format>book</dc:format>
    <dc:identifier>4mb8LuKKwocC</dc:identifier>
    <dc:identifier>ISBN:0821417568</dc:identifier>
    <dc:identifier>ISBN:9780821417560</dc:identifier>
    <dc:publisher>Ohio Univ Pr</dc:publisher>
    <dc:subject>Literary Criticism</dc:subject>
    <dc:title>Cruising with Robert Louis Stevenson</dc:title>
    <dc:title>travel, narrative, and the colonial body</dc:title>
  </entry>
  <entry>
    <id>http://www.google.com/books/feeds/volumes/4yo9AAAAYAAJ</id>
    <updated>2010-08-30T20:37:27.000Z</updated>
    <category scheme="http://schemas.google.com/g/2005#kind" term="http://schemas.google.com/books/2008#volume"/>
    <title type="text">New Arabian nights</title>
    <link rel="http://schemas.google.com/books/2008/thumbnail" type="image/x-unknown" href="http://bks0.books.google.com/books?id=4yo9AAAAYAAJ&amp;printsec=frontcover&amp;img=1&amp;zoom=5&amp;edge=curl&amp;sig=ACfU3U1_NeiJKS1J279NoYk98NzRZURI3Q&amp;source=gbs_gdata"/>
    <link rel="http://schemas.google.com/books/2008/info" type="text/html" href="http://books.google.com/books?id=4yo9AAAAYAAJ&amp;dq=Stevenson&amp;ie=ISO-8859-1&amp;source=gbs_gdata"/>
    <link rel="http://schemas.google.com/books/2008/preview" type="text/html" href="http://books.google.com/books?id=4yo9AAAAYAAJ&amp;printsec=frontcover&amp;dq=Stevenson&amp;ie=ISO-8859-1&amp;cd=9&amp;source=gbs_gdata"/>
    <link rel="http://schemas.google.com/books/2008/annotation" type="application/atom+xml" href="http://www.google.com/books/feeds/users/me/volumes"/>
    <link rel="http://schemas.google.com/books/2008/acsepubfulfillmenttoken" type="application/vnd.adobe.adept+xml" href="http://books.google.com/books/download/New_Arabian_nights.acsm?id=4yo9AAAAYAAJ&amp;format=epub&amp;output=acs4_fulfillment_token"/>
    <link rel="alternate" type="text/html" href="http://books.google.com/books?id=4yo9AAAAYAAJ&amp;dq=Stevenson&amp;ie=ISO-8859-1"/>
    <link rel="self" type="application/atom+xml" href="http://www.google.com/books/feeds/volumes/4yo9AAAAYAAJ"/>
    <gbs:embeddability value="http://schemas.google.com/books/2008#embeddable"/>
    <gbs:openaccess value="http://schemas.google.com/books/2008#enabled"/>
    <gbs:viewability value="http://schemas.google.com/books/2008#view_all_pages"/>
    <dc:creator>Robert Louis Stevenson</dc:creator>
    <dc:date>1922</dc:date>
    <dc:description>THE SUICIDE CLUB STORY OF THE YOUNG MAN WITH THE CREAM TARTS DURING his 
residence in London, the accomplished Prince Florizel of Bohemia gained the ...</dc:description>
    <dc:format>386 pages</dc:format>
    <dc:format>book</dc:format>
    <dc:identifier>4yo9AAAAYAAJ</dc:identifier>
    <dc:identifier>HARVARD:HWP51H</dc:identifier>
    <dc:subject>Fiction</dc:subject>
    <dc:title>New Arabian nights</dc:title>
  </entry>
  <entry>
    <id>http://www.google.com/books/feeds/volumes/z2Yf1FX02EkC</id>
    <updated>2010-08-30T20:37:27.000Z</updated>
    <category scheme="http://schemas.google.com/g/2005#kind" term="http://schemas.google.com/books/2008#volume"/>
    <title type="text">Robert Louis Stevenson</title>
    <link rel="http://schemas.google.com/books/2008/thumbnail" type="image/x-unknown" href="http://bks2.books.google.com/books?id=z2Yf1FX02EkC&amp;printsec=frontcover&amp;img=1&amp;zoom=5&amp;edge=curl&amp;sig=ACfU3U0gOHJkzL_2WLHxXwdINCOTt03bMw&amp;source=gbs_gdata"/>
    <link rel="http://schemas.google.com/books/2008/info" type="text/html" href="http://books.google.com/books?id=z2Yf1FX02EkC&amp;dq=Stevenson&amp;ie=ISO-8859-1&amp;source=gbs_gdata"/>
    <link rel="http://schemas.google.com/books/2008/preview" type="text/html" href="http://books.google.com/books?id=z2Yf1FX02EkC&amp;printsec=frontcover&amp;dq=Stevenson&amp;ie=ISO-8859-1&amp;cd=10&amp;source=gbs_gdata"/>
    <link rel="http://schemas.google.com/books/2008/annotation" type="application/atom+xml" href="http://www.google.com/books/feeds/users/me/volumes"/>
    <link rel="http://schemas.google.com/books/2008/acsepubfulfillmenttoken" type="application/vnd.adobe.adept+xml" href="http://books.google.com/books/download/Robert_Louis_Stevenson.acsm?id=z2Yf1FX02EkC&amp;format=epub&amp;output=acs4_fulfillment_token"/>
    <link rel="alternate" type="text/html" href="http://books.google.com/books?id=z2Yf1FX02EkC&amp;dq=Stevenson&amp;ie=ISO-8859-1"/>
    <link rel="self" type="application/atom+xml" href="http://www.google.com/books/feeds/volumes/z2Yf1FX02EkC"/>
    <gbs:embeddability value="http://schemas.google.com/books/2008#embeddable"/>
    <gbs:openaccess value="http://schemas.google.com/books/2008#disabled"/>
    <gbs:viewability value="http://schemas.google.com/books/2008#view_partial"/>
    <dc:creator>Richard Ambrosini</dc:creator>
    <dc:creator>Richard Dury</dc:creator>
    <dc:date>2006</dc:date>
    <dc:description>As the editors point out in their Introduction, Stevenson reinvented the “personal essay” and the “walking tour essay,” in texts of ironic stylistic ...</dc:description>
    <dc:format>377 pages</dc:format>
    <dc:format>book</dc:format>
    <dc:identifier>z2Yf1FX02EkC</dc:identifier>
    <dc:identifier>ISBN:0299212246</dc:identifier>
    <dc:identifier>ISBN:9780299212247</dc:identifier>
    <dc:publisher>Univ of Wisconsin Pr</dc:publisher>
    <dc:subject>Literary Criticism</dc:subject>
    <dc:title>Robert Louis Stevenson</dc:title>
    <dc:title>writer of boundaries</dc:title>
  </entry>
</feed>

Now it would be nice if the Atom included <link> elements for the epubs themselves. Perhaps the feed could even use the recently released “acquisition” link relation defined by OPDS v1.0. For example, by including something like the following in each atom:entry element:

<link type="application/epub+zip" rel="http://opds-spec.org/acquisition/open-access" href="http://books.google.com/books/download/Kidnapped.epub?id=ENMWAAAAYAAJ&output=epub" />

Theoretically it should be possible to construct the appropriate link for the epub, based on what data is available in the Atom. But it would enable quite a bit of use of the epubs to make their URLs available explicitly in a programmatic way. Unfortunately we would still be limited to dipping into the full dataset using a query, instead of being able to crawl the entire archive, with something like a paged Atom feed. From a conversation over on get-theinfo it appears that this approach might not be as easy as it sounds. Also, it turns out that magically, many of the books have been uploaded to the Internet Archive. 902,188 of them in fact.

So maybe not that much work needs to be done. But presumably more public domain content will become available from Google Books, and it would be nice to be able to say there was at least one other copy of it elsewhere, for digital preservation purposes. It would be great to see Google step up and do some good, by making their API usable for folks wanting to replicate the public domain content. Still, at least they haven’t of done evil by locking it away completely. Dan Brickley had an interesting suggestion to possibly collaborate on this work.

For me, the Odersky piece is a really nice essay on why simplicity is often in the eye of the beholder. Often the key to simplicity is working with people who see things in roughly the same way. People who have similar needs, that are met by using particular approaches and tools. Basically a shared and healthy culture to make emergent complexity palatable.

Brian made the point about simplicity for programmers having an inversely proportional relationship to simplicity for end users, or in his own words:

I think that the simpler we make it for the programmers, usually the more complicated it becomes for the end users, and visa versa.

I think the only thing to keep in mind is that the distinction between programmers and end users isn’t always clear.

As a software developer I’m constantly using, or inheriting someone else’s code: be it a third party library that I have a dependency on, or a piece of software that somebody wrote once upon a time, who has moved on elsewhere. In both these cases I’m effectively an end-user of a program that somebody else designed and implemented. The interfaces and abstractions that this software developer has chosen are the things I (as an end user) need to be able to understand and work with. Ultimately, I think that it’s easier to keep software usable for end users (of whatever flavor) by keeping the software design itself simple.

Simplicity makes the software easier to refactor over time when the inevitable happens, and someone wants some new or altered behavior. Simplicity also should make it clear when a suggested change to a piece of software doesn’t fit the design of the software in question, and is best done elsewhere. One of the best rules of thumb I’ve encountered over the years to help get to this place is the Unix Philosophy:

Write programs that do one thing and do it well. Write programs to work together.

As has been noted elsewhere, composability is one of the guiding principles of the Microservices approach–and it’s why I’m a big fan (in principle). Another aspect to the Unix philosophy that Microservices seems to embody is:

Data dominates.

The software can (and will) come and go, but we are left with the data. That’s the reality of digital preservation. It could be argued that the programs themselves are data, which gets us into sci-fi virtualization scenarios. Maybe someday, but I personally don’t think we’re there yet.

Another approach I’ve found that works well to help ensure code simplicity has been unit testing. Admittedly it’s a bit of a religion, but at the end of the day, writing tests for your code encourages you to use the APIs, interfaces and abstractions that you are creating. So you notice sooner when things don’t make sense. And of course, they let you refactor with a safety net, when the inevitable changes rear their head.

And, another slightly more humorous way to help ensure simplicity:

Always code as if the person who ends up maintaining your code is a violent psychopath who knows where you live.

Which leads me to a jedi mind trick my former colleague Keyser Söze Andy Boyko tried to teach me (I think): it’s useful to know when you don’t have to write any code at all. Sometimes existing code can be used in a new context. And sometimes the perceived problem can be recast, or examined from a new perspective that makes the problem go away. I’m not sure what all this has to do with digital preservation. The great thing about what CDL is doing with microservices is they are trying to focus on the *what*, and not the *how* of digital preservation. Whatever ends up happening with the implementation of Merritt itself, I think they are discovering what the useful patterns of digital preservation are, trying them out, and documenting them…and it’s incredibly important work that I don’t really see happening much elsewhere.

Before:

< ?xml version="1.0" encoding="UTF-8"?>
<mets TYPE="urn:library-of-congress:ndnp:mets:encyclopedia:encyclopediaEntry" PROFILE="urn:library-of-congress:mets:profiles:ndnp:encyclopediaEntry:v1.1" LABEL="The National Forum Scope Note" 
     xmlns:mods="http://www.loc.gov/mods/v3" xmlns="http://www.loc.gov/METS/" xmlns:dsig="http://www.w3.org/2000/09/xmldsig#" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
 
     <!--METS HEADER-->
     <metshdr CREATEDATE="2007-01-10T09:00:00" ><!--CREATEDATE should be populated with creation date of the record.  RECORDSTATUS should only be set by Validation Application.-->
        <agent ROLE="CREATOR" TYPE="ORGANIZATION">
            <name><!--Awardee, if the awardee created the METS record.  If LC created the METS record, should be "Library of Congress"-->Library of Congress</name>
        </agent>
    </metshdr>
 
    <!--Descriptive metadata for encyclopedia entry-->
    <dmdsec ID="encyclopediaMods">                
        <mdwrap MDTYPE="MODS" LABEL="Encyclopedia entry metadata">
            <xmldata>
                <mods:mods>
                    <mods:name type="corporate">
                            <mods:displayform><!--Awardee-->Library of Congress</mods:displayform>
                    </mods:name>
                    <mods:relateditem>
                        <mods:identifier type="lccn"><!--LCCN-->sn82015056</mods:identifier>
                        <!-- mods:identifier elements are repeatable -->
                    </mods:relateditem>
                </mods:mods>
            </xmldata>
        </mdwrap>
    </dmdsec>
 
 
 
    <!--FILE SECTION-->
    <filesec>
        <filegrp>
            <file ID="encyclopediaEntryText">
                <fcontent>
                    <xmldata>
                        <xhtml:html xmlns:xhtml="http://www.w3.org/1999/xhtml"  lang="en">
                           <xhtml:head>
                              <xhtml:title><!--Newspaper title-->The National Forum (Washington, DC), 1910-19??</xhtml:title>
                           </xhtml:head>
 
                           <xhtml:body>
                              <xhtml:p>
The first issue of the <xhtml:cite>National Forum</xhtml:cite> was likely released on April 30, 1910 
and the newspaper ran through at least November 12 of that year. The four-page African-American 
weekly covered such local events as Howard University graduations and Baptist church activities, but its 
pages also included national news, sports, home maintenance, women's news, science, editorial 
cartoons, and reprinted stories from national newspapers. Its primary focus was on how the news 
affected the city's black community. A unique feature was its coverage of Elks Club meetings and 
activities.  Business manager John H. Wills contributed the community-centered "Vanity Fair" column that
 usually appeared on the front page of each issue. The publisher and editor was Ralph W. White, who 
went on to publish another African-American newspaper, the <xhtml:a href="info:lccn/sn86092050">
<xhtml:cite>McDowell Times</xhtml:cite></xhtml:a> of Keystone, West Virginia. Originally located at 
609 F St., NW, the newspaper's offices moved in August to 1022 U Street, N.W. to be closer to the 
African-American community it served.  No extant first issue of the <xhtml:cite>National
 Forum</xhtml:cite> exists.
                              </xhtml:p>
                           </xhtml:body>
                        </xhtml:html>
                    </xmldata>
                </fcontent>
            </file>
        </filegrp>
    </filesec>
 
    <!--STRUCTURAL MAP-->    
    <structmap xmlns:np="urn:library-of-congress:ndnp:mets:newspaper">
        <div TYPE="np:encyclopediaEntry" DMDID="encyclopediaMods">
            <fptr FILEID="encyclopediaEntryText"/>
        </div>
    </structmap>    
</mets>

After:

< ?xml version="1.0" encoding="utf-8"?>
< !DOCTYPE html PUBLIC "-//W3C//DTD XHTML+RDFa 1.0//EN" "http://www.w3.org/MarkUp/DTD/xhtml-rdfa-1.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:dcterms="http://purl.org/dc/terms/">
  <head>
    <title>The National Forum (Washington, DC), 1910-19??</title>
    <meta content="The National Forum (Washington, DC), 1910-19??" property="dcterms:title"/>
    <meta content="2007-01-10T09:00:00" property="dcterms:created"/>
    <meta content="Library of Congress" property="dcterms:creator"/>
    <meta content="http://chroniclingamerica.loc.gov/lccn/sn82015056#title" property="dcterms:subject"/>
  </head>
  <body rel="dcterms:description">
    <p>The first issue of the <cite>National Forum</cite> was likely released on April 30, 1910 and the 
newspaper ran through at least November 12 of that year. The four-page African-American weekly 
covered such local events as Howard University graduations and Baptist church activities, but its pages 
also included national news, sports, home maintenance, women's news, science, editorial cartoons, and 
reprinted stories from national newspapers. Its primary focus was on how the news affected the city's black community. A unique feature was its coverage of Elks Club meetings and activities.  Business 
manager John H. Wills contributed the community-centered "Vanity Fair" column that usually appeared 
on the front page of each issue. The publisher and editor was Ralph W. White, who went on to publish 
another African-American newspaper, the <a href="http://chroniclingamerica.loc.gov/lccn/sn86092050">
<cite>McDowell Times</cite></a> of Keystone, West Virginia. Originally located at 609 F St., NW, the 
newspaper's offices moved in August to 1022 U Street, N.W. to be closer to the African-American 
community it served.  No extant first issue of the <cite>National Forum</cite> exists.</p>
  </body>
</html>

I basically took a complicated METS wrapper around some XHTML, which was really just expressing metadata about the HTML, and refactored it as XHTML. Not that METS is a bad XML smell generally, but in this particular case it was overkill. If you look closely you’ll see I’m using RDFa, similar to what Facebook are doing with their OpenGraph Protocol. There’s less to get wrong, what’s there should look more familiar to web developers who aren’t versed in arcane library standards, and I can now read the metadata from the XHTML with an RDFa aware parser, like Python’s rdflib:

>>> import rdflib
>>> g = rdflib.Graph()
>>> g.parse('essays/1.html', format='rdfa')
>>> for triple in g: print triple
... 
(rdflib.term.URIRef('file:///home/ed/Projects/essays/essays/1.html'), rdflib.term.URIRef('http://purl.org/dc/terms/creator'), rdflib.term.Literal(u'Library of Congress'))
(rdflib.term.URIRef('file:///home/ed/Projects/essays/essays/1.html'), rdflib.term.URIRef('http://purl.org/dc/terms/title'), rdflib.term.Literal(u'The National Forum (Washington, DC), 1910-19??'))
(rdflib.term.URIRef('file:///home/ed/Projects/essays/essays/1.html'), rdflib.term.URIRef('http://purl.org/dc/terms/description'), rdflib.term.Literal(u'\n    <p xmlns="http://www.w3.org/1999/xhtml">\nThe first issue of the <cite>National Forum</cite> was likely released on April 30, 1910 and the newspaper ran through at least November 12 of that year. The four-page African-American weekly covered such local events as Howard University graduations and Baptist church activities, but its pages also included national news, sports, home maintenance, women\'s news, science, editorial cartoons, and reprinted stories from national newspapers. Its primary focus was on how the news affected the city\'s black community. A unique feature was its coverage of Elks Club meetings and activities.  Business manager John H. Wills contributed the community-centered &quot;Vanity Fair&quot; column that usually appeared on the front page of each issue. The publisher and editor was Ralph W. White, who went on to publish another African-American newspaper, the <a href="http://chroniclingamerica.loc.gov/lccn/sn86092050"><cite>McDowell Times</cite></a> of Keystone, West Virginia. Originally located at 609 F St., NW, the newspaper\'s offices moved in August to 1022 U Street, N.W. to be closer to the African-American community it served.  No extant first issue of the <cite>National Forum</cite> exists.\n</p>\n  ', datatype=rdflib.term.URIRef('http://www.w3.org/1999/02/22-rdf-syntax-ns#XMLLiteral')))
(rdflib.term.URIRef('file:///home/ed/Projects/essays/essays/1.html'), rdflib.term.URIRef('http://purl.org/dc/terms/subject'), rdflib.term.Literal(u'http://chroniclingamerica.loc.gov/lccn/sn82015056#title'))
(rdflib.term.URIRef('file:///home/ed/Projects/essays/essays/1.html'), rdflib.term.URIRef('http://purl.org/dc/terms/created'), rdflib.term.Literal(u'2007-01-10T09:00:00'))

So I downloaded the enwiki-latest-page.sql.gz, loaded it in, and then joined on it in my query to come up with a new list. Jodi was right, it’s a lot more interesting:

This removed a lot of the interwiki links between the English wikipedia and other language wikipedias (which would be interesting to look at in their own right). It also removed administrative links to things like www.dnsstuff.com. Also interesting is that it removed www.facebook.com from the list, which probably were linked to from user profile pages? The neat thing is it introduced new sites into the top 100 like the following:

We can see a lot more pop culture media present: newspapers, magazines, sporting information. Also we can see research oriented websites like worldcat.org, ssd.jpl.nasa.gov, adsabs.harvard.edu make it into the top 100.

I work for the US federal government so I was interested to look at what .gov domains were in the top 100:

Which points to the importance of federal biomedical, geospatial, scientific, demographic and biographical information to wikipedians. It would be interesting to take a look at higher education institutions at some point. Doing these one off reports is giving me some ideas about what linkypedia could turn into. Thanks Jodi.

In the section on Taylorist Production Processes, Dorothea makes a really good point that libraries have tended to optimize workflows for particular types of materials (e.g. photographs, books, journal articles, maps). When materials require a great deal of specialized knowledge to deal with, and the tools that are used to manage and provide access to the content are similarly specialized, it’s not hard to understand why this balkanization has happened. On occasion I’ve heard folks (/me points finger at self) bemoan the launch of a new website as the creation of “yet another silo”. But the technical silos that we perceive are largely artifacts of the social silos we work in: archives, special collections, rare books, maps, etc. The collections we break up our libraries into…the projects that mirror those collections. We need to work better together before we can build common digital preservation tools. To paraphrase something David Brunton has said to me before: we need to think of our collections more as sets of things that can be rearranged at will, with ease and even impunity. In fact the architecture of the Web (and each website on it) is all about doing that.

Even though it can be tough (particularly in large organizations) I think we can in fact achieve some levels of common tooling (in areas like storage and auditing); but we must admit (to ourselves at least) that some levels of access will likely always be specialized in terms of technical infrastructure and user interface requirements:

Some, though not all, data can be shoehorned into a digital library not optimised for them, but only at the cost of the affordances surrounding them. Consider data over which a complex Web-based interaction environment has been built. The data can be removed from the environment for preservation, but only at the cost of loss of the specialised interactions that make the data valuable to begin with. If the dataset can be browsed via the Web interface, a static Web snapshot becomes possible, but it too will lack sophisticated interaction. If the digital library takes on the not inconsiderable job of recreating the entire environment, it is committing to rewriting interaction code over and over again indefinitely as computing environments change.

Dorothea’s statement about committing to rewriting interaction code over and over again is important. I’m a software developer, and a web developer to boot — so there’s nothing I like more than yanking the data out of one encrusted old app, and creating it a-fresh using the web-framework-du-jour. But in my heart of hearts I know that while this may work for large collections of homogeneous data, it doesn’t scale very well for a vast sea of heterogeneous data. However, all is not lost. As the JISC are fond of saying:

The coolest thing to do with your data will be thought of by someone else.

So why don’t us data archivers get out of the business of building the “interaction code”. Maybe our primary service should be to act as data wholesalers who collect it, and make it available in bulk to those who do want to build access layers on top of it. Lets make our data easy for other people to use (with clear licensing) and reference (with web identifiers) so that they can annotate it, and we can pull back those annotations and views. In a way this is kind of hearkening back to the idea of Data Providers and Service Providers that was talked about a lot in the context of OAI-PMH. But in this case we’d be making the objects available as well as the metadata that describes them, similar to the use cases around OAI-ORE. I got a chance to chat with Kate Zwaard of the GPO at CurateCamp a few weeks ago, and learned how the new Federal Register is a presentation application for raw XML data being made available by the GPO. Part of the challenge is making these flows of data public, and giving credit where credit is due — not only to the creators of the shiny site you see, but to the folks behind the scenes who make it possible.

Another part of Dorothea’s essay that stuck out a bit for me, was the advice to split ingest, storage and access systems.

Ingest, storage, and end-user interfaces should be as loosely coupled as possible. Ideally, the same storage pool should be available to as many ingest mechanisms as researchers and their technology staff can dream up, and the items within should be usable within as many reuse, remix, and re-evaluation environments as the Web can produce.

This is something we (myself and other folks at LC) did as part of the tooling to support the National Digital Newspaper Program. Our initial stab at the software architecture was to use Fedora to manage the full life cycle (from ingest, to storage, to access) of the newspaper content we receive from program awardees around the US. The only trouble was that we wanted the access system to support heavy use by researchers and also robots (Google, Yahoo, Microsoft, etc) building their own views on the content. Unfortunately the way we had put the pieces together we couldn’t support that. Increasingly we found ourselves working around Fedora as much as possible to squeeze a bit more performance out of the system.

So in the end we (and by we I mean David) decided to bite the bullet and split off the inventory systems keeping track of where received content lives (what storage systems, etc) from the access systems that delivered content on the Web. Ultimately this meant we could leverage industry proven web development tools to deliver the newspaper content…which was a huge win. Now that’s not saying that Fedora can’t be used to provide access to content. I think the problems we experienced may well have been the result of our use of Fedora, rather than Fedora itself. Having to do multiple, large XSLT transforms to source XML files to render a page is painful. While it’s a bit of a truism, a good software developer tries to pick the right tool for the job. Half the battle there is deciding on the right granularity for the job … the single job we were trying to solve with Fedora (preservation and access) was too big for us to do either right.

Having a system that’s decomposable, like the approach that CDL is taking with Microservices is essential for long-term thinking about software in the context of digital preservation. I guess you could say “there’s no-there-there” with Microservices, since there’s not really a system to download–but in a way that’s kind of the point.

I guess this is just a long way of saying, Thanks Dorothea! :-)

Anyhow, I wanted to get a handle on just how many external links there are in the full snapshot of English Wikipedia. A month or so ago Jakob Voss pointed me at the External Links SQL dump over at Wikipedia as a possible way to circumvent heavy use of the Wikipedia’s API, by providing a baseline to update against. So I thought to myself that I could just suck this down, and import it into MySQL and run some analysis on that to see how many links and what sorts of host name concentrations there were.

Sucking down the file didn’t take too long. But the mysql import on the dump spent about 24hrs (on my laptop) before I killed it. On a hunch I peeked into the 4.5G sql file and noticed that the table had several indexes defined. So I went through some contortions with csplit to remove the indexes from the DDL, and lo and behold it loaded in something like 20 minutes. Then I wrote a some python to query the database, get each external link url, extract the hostname from the url, and write it out through a unix pipeline to count up the unique hostnames:

./hosts.py | sort -S 1g | uniq -c | sort -rn > enwiki-externallinks-hostnames.txt

This is a little unix trick my old boss Fred Lindberg taught me years ago, and it stills works remarkably well: 30,127,734 urls were sorted into 2,162,790 unique domains in another 20 minutes or so. If you are curious the full output is available here. The number 1 host was toolserver.org with 3,169,993 links. This wasn’t too surprising since it is a hostname heavily used by wikipedians as they go about their business. Next is www.google.com at 2,117,967 links, which appeared to be quite a few canned searches. This wasn’t terribly exciting either. So I removed toolserver.org and www.google.com (so as not to visually skew things too much), and charted the rest of the top 100:

I figured that could be of some interest to somebody, sometime. I didn’t find similar current stats available anywhere on the web. But if you know of them please let me know. The high ranking of www.ncbi.nlm.nih.gov and dx.doi.org were pleasant surprises. I did a little superficial digging and found some fascinating bots like Citation Bot and ProteinBoxBot, which seem to trawl external article databases looking for appropriate Wikipedia pages to add links to. Kind of amazing.

It is very easy to replace an electronic dataset with an updated copy, and … the replacement can have wide-reaching effects. The processes of authorship … produce different versions which in an electronic environment can easily go into broad circulation; if each draft is not carefully labeled and dated it is difficult to tell which draft one is looking at or whether one has the “final” version of a work.

For example at $work we have a pilot project to process journal content submitted by publishers. We don’t have the luxury of telling them exactly what content to submit (packaging formats, xml schemas, formats, etc), but on receipt we want to normalize the content for downstream applications by bagging it up, and then extracting some of the content into a standard location.

So a publisher makes an issue of a journal available for pickup via FTP:

adi_v2_i2
|-- adi_015.pdf
|-- adi_015.xml
|-- adi_018.pdf
|-- adi_018.xml
|-- adi_019.pdf
|-- adi_019.xml
|-- adi_v2_i2_ofc.pdf
|-- adi_v2_i2_toc.pdf
`-- adi_v2_i2_toc.xml

The first thing we do is bag up the content, to capture what was retrieved (manifest + checksums) and stash away some metadata about where it came from.

adi_v2_i2
|-- bag-info.txt
|-- bagit.txt
|-- data
|   |-- adi_015.pdf
|   |-- adi_015.xml
|   |-- adi_018.pdf
|   |-- adi_018.xml
|   |-- adi_019.pdf
|   |-- adi_019.xml
|   |-- adi_v2_i2_ofc.pdf
|   |-- adi_v2_i2_toc.pdf
|   `-- adi_v2_i2_toc.xml
`-- manifest-md5.txt

Next we run some software that knows about the particularities of this publishers content, and persist it into the bag in a predictable, normalized way:

adi_v2_i2
|-- bag-info.txt
|-- bagit.txt
|-- data
|   |-- adi_015.pdf
|   |-- adi_015.xml
|   |-- adi_018.pdf
|   |-- adi_018.xml
|   |-- adi_019.pdf
|   |-- adi_019.xml
|   |-- adi_v2_i2_ofc.pdf
|   |-- adi_v2_i2_toc.pdf
|   |-- adi_v2_i2_toc.xml
|   `-- eco
|       `-- articles
|           |-- 1
|           |   |-- article.pdf
|           |   |-- meta.json
|           |   `-- ocr.xml
|           |-- 2
|           |   |-- article.pdf
|           |   |-- meta.json
|           |   `-- ocr.xml
|           `-- 3
|               |-- article.pdf
|               |-- meta.json
|               `-- ocr.xml
`-- manifest-md5.txt

The point of this post isn’t the particular way these changes were layered into the filesystem–it could be done in a multitude of other ways (mets, oai-ore, mpeg21-didl, foxml, etc). The point is rather that the data has undergone two transformations very soon after the it was obtained. And if we are successful, it would undergo many more as it is preserved over time. The advantage of layering content like this into the filesystem is that makes some attempt to preserve the data by disentangling it from the code that operates on it, which will hopefully be swapped out at some point. At the same time we want to be able to get back to the original data, and to also possibly rollback changes which inadvertently corrupted or destroyed portions of the data. Shit happens.

So the way I see it we have three options:

• Ignore the problem and hope it will go away.
• Create full copies of the content, and link them together
• Version the content

While tongue in cheek, option 1 is sometimes reality. Perhaps your repository content is such that it never changes. Or if it does, it happens in systems that are downstream from your repository. The overhead of managing versions isn’t something that you need to worry about…yet. I’ve been told that DSpace doesn’t currently handle versioning, and that they are looking to upcoming Fedora integration to provide versioning options.

In option 2 version control is achieved by copying the files that make up a repository object, making modifications to the relevant files, and then linking this new copy to the older copy. We currently do this in our internal handling of the content in the National Digital Newspaper Program, where we work at what we call the batch level, which is essentially a directory of content shipped to the Library of Congress, containing a bunch of jp2, xml, pdf and tiff files. When we have accepted a batch, and later have discovered a problem, we then version the batch by creating a copy of the directory. When service copies of the batches are 50G or so, these full copies can add up. Sometimes just to make a few small modifications to some of the XML files we end up having what appears to be largely redundant copies of multi-gigabyte directories. Disk is cheap as they say, but it makes me feel a bit dirty.

Option 3 is to leverage some sort of version control system. One of the criteria I have for a version control system for data is that versioned content should not be dependent on a remote server. The reason for this is I want to be able to push the versioned content around to other systems, tape backup, etc and not have it become stale because some server configuration happened to change at some point. So in my mind, subversion and cvs are out. Revision control systems like rcs, git, mercurial, bazaar on the other hand are possibilities. Some of the nice things about using git or mercurial:

• they are very active projects, so there are lots of eyes on the code fixing bugs, making enhancements
• the content repositories are distributed, so they allow content to migrate into other contexts
• developers who come to inherit content will be familiar with using them
• you get audit logs for free
• they include functionality to publish content on the web, and to copy or clone repository content

But there are trade offs with using git, mercurial, etc since these are quite complex pieces of software. Changes in them sometimes trigger backwards incompatible changes in repository formats–which require upgrading the repository. Fortunately there are tools for these upgrades, but one must stay on top of them. These systems also have a tendency to see periods of active use, followed by an exodus to a newer shinier version control system. But there are usually tools to convert your repositories from the old to the new.

To anyone familiar with digital preservation this situation should seem pretty familiar: format migration, or more generally data migration.

The California Digital Library have recognized the need for version control, but have taken a slightly different approach. CDL created a specification for a simplified revision control system called Reverse Directory Deltas (ReDD), and built an implementation into their Storage Service. The advantage that ReDD has is that the repository format is quite a bit more transparent, and less code dependent than other repository formats like git, mercurial, etc. It is also a specification that CDL manages, and can change at will, rather than being at the mercy of an external opensource project. As an exercise about a year ago I wanted to see how easy it was to use ReDD by implementing it. I ended up calling the tool dflat, since ReDD fits in with CDL’s filesystem convention D-flat specification.

In the pilot project I mentioned above I started out wanting to use ReDD. I didn’t really want to use all of D-flat, because it did a bit more than I needed, and thought I could refactor the ReDD bits of my dflat code into a ReDD library, and then use it to version the journal content. But in looking at the code a year later I found myself wondering about the trade offs again. Mostly I wondered about the bugs that might be lurking in the code, which nobody else was really using. And I thought about new developers that might work on the project, and who wouldn’t know what the heck ReDD was and why I wasn’t using something more common.

So I decided to be lazy, and use mercurial instead.

I figured this was a pilot project, so what better way to get a handle on the issues of using distributed revision control for managing data? The project was using the Python programming language extensively, and Mercurial is written in Python — so it seemed like a logical choice. I also began to think of the problem of upgrading software and repository formats as another facet to the format migration problem…which we wanted to have a story for anyway.

I wanted to start a conversation about some of these issues at Curate Camp, so I wrote a pretty simplistic tool that compares git, mercurial and dflat/redd in terms of the time it takes to initialize a repository, and commit changes to an arbitrary set of content in a directory called ‘data’. For example I ran it against a gigabyte of mp3 content, and it generates the following output:

ed@curry:~/Projects$ git clone git://github.com/edsu/versioning-metrics.git
ed@curry:~/Projects/versioning-metrics$ cp ~/Music data
ed@curry:~/Projects/versioning-metrics$ ./compare.py 

create repository times
git        0:02:16.842117
hg         0:02:48.397038
dflat      0:01:21.196579

repository size
git        1233106981
hg         1287443209
dflat      1347108177

checkout sizes
git        2580197304
hg         2634533532
dflat      2694216030

commit one byte change to all files
git        0:01:03.694969
hg         0:00:48.456455
dflat      0:00:22.926949

repository sizes after commit
git        1233112423
hg         1287450930
dflat      1347166431

Anyhow this is just the start of some analysis. I’d like to see how it performs on text content where there are probably some compression gains to be had in the repository size. I think this also demonstrates how it might not be feasible to version large amounts of content as a synchronous operation in a web application. Hopefully this will be a useful seed for a discussion at Curate Camp. I don’t want to pretend to have all the answers here, and want to have more conversations about approaches within the digital curation community.

Archival context information consists of information describing the circumstances under which records (defined broadly here to include personal papers and records of organizations) have been created and used. This context includes the identification and characteristics of the persons, organizations, and families who have been the creators, users, or subjects of records, as well as the relationships amongst them.
Toronto Tenets

I’ll admit I’m jaded. So much XML has gone under the bridge, that it’s hard for me to get terribly excited (anymore) about yet-another-xml-schema. Yes, structured data is good. Yes, encouraging people make their data available similarly is important. But for me to get hooked I need a story for how this structured data is going to live, and be used on the Web. This is the main reason I found Daniel Pitti’s talk about the Social Networks of Archival Context (SNAC) to be so exciting.

SNAC is an NEH (and possibly IMLS soon) funded project of the University of Virginia, California Digital Library, and the Berkeley School for Information. The project’s general goal is to:

… unlock descriptions of people from descriptions of their records and link them together in exciting new ways.

Where “descriptions of their records” are EAD XML documents, and “linking them together” means exposing the entities buried in finding aids (people, organizations, etc), assigning identifiers to them, and linking them together on the web. I guess I might be reading what I want a bit into the goal based on the presentation, and my interest in Linked Data. If you are interested, more accurate information about the project can be found in the NEH Proposal that this quote came from.

Even though SNAC was only very recently funded, Daniel was already able to demonstrate a prototype application that he and Brian Tingle worked on. If I’m remembering this right, Daniel basically got a hold of the EAD finding aids from the Library of Congress, extracted contextual information (people, families, corporate names) from relevant EAD elements, and serialized these facts as EAC-CPF documents. Brian then imported the documents using an extension to the eXtensible Text Framwork, which allowed XTF to be EAC-CPF aware.

The end result is a web application that lets you view distinct web pages for individuals mentioned in the archival material. For example here’s one for John von Neumann

All those names in the list on the right are themselves hyperlinks which take you to that person’s page. If you were able to scroll down (the prototype hasn’t been formally launched yet) you could see links to corporate names like Los Alamos Scientific Laboratory, The Institute for Advanced Study, US Atomic Energy Commission, etc. You would also see a Resources section that lists related finding aids and books, such as the John von Neumann Papers finding aid at the Library of Congress.

I don’t think I was the only one in the audience to immediately see the utility of this. In fact it is territory well trodden by OCLC and the other libraries involved in the VIAF project which essentially creates web pages for authority records for people like John Von Neumann who have written books. It’s also similar to what People Australia, BibApp and VIVO are doing to establish richly linked public pages for people. As Daniel pointed out: archives, libraries and museums do a lot of things differently; but ultimately they all have a deep and abiding interest in the intellectual output, and artifacts created by people. So maybe this is an area where we can see more collaboration across the cultural divides between cultural heritage institutions. The activity of putting EAD documents, and their transformed HTML cousins on the web is important. But for them to be more useful they need to be contextualized in the web itself using applications like this SNAC prototype.

I immediately found myself wondering if the URL say for this SNAC view of John von Neumann could be considered an identifier for the EAC-CPF record. And what if the HTML contained the structured EAC-CPF data using xml+xslt, a microformat, rdfa or a link rel pointing at an external XML document? Notice I said an instead of the identifier. If something like EAC-CPF is going to catch on lots of archives would need to start generating (and publishing) them. Inevitably there would be duplication: e.g. multiple institutions with their own notion of John von Neumann. I think this would be a good problem to have, and that having web resolvable identifiers for these records would allow them to be knitted together. It would also allow hubs of EAC-CPF to bubble up, rather than requiring some single institution to serve as *the* master database (as in the case of VIAF).

A few things that would be nice to see for EAC-CPF would be:

• Instructions on how to link EAD documents to EAC-CPF documents.
• Recommendations on how to make EAC-CPF data available on the Web.
• A wikipage or some low-cost page for letting people share where they are publishing EAC-CPF.

In addition I think it would be cool to see:

• A microformat for making EAC-CPF data available in HTML. If not an official Microformat then Plain Old Semantic HTML will do. It might also be possible to leverage some existing Microformats like hCard.
• A EAC-CPF RDF schema for embedding EAC-CPF data in HTML using RDFa and other Linked Data environments. Heck, Facebook is doing it. It might also be useful to leverage existing vocabularies like FOAF, Organization Ontology, etc using a Dublin Core Application Profile.

Anyhow I just wanted to take a moment to say how exciting it is to see the stuff hiding in finding aids making it out onto the Web, with URLs for resources like People, Corporations and Families. I hope to see more as the SNAC project integrates more with existing name authority files (work that Ray Larson at Berkeley is going to be doing), and importing finding aids from more institutions with different EAD encoding practices.

I’m no usability expert, but just a few minutes browsing the new site and comparing it to the old one make it clear what a leap forward this is. Hopefully the legal status of the new site will be ironed out shortly.

Most of all it’s great to see that the Federal Register is now a single web application. The service it provides to the American public is important enough to deserve its own dedicated web presence. As the developers point out in their video describing the effort, they wanted to make the Federal Register a “first class citizen of the web”…and I think they are certainly helping do that. This might seem obvious, but often there is a temptation to jam publications from the print world (like the Federal Register) into dumbed down monolithic repositories that treat all “objects” the same. Proponents of this approach tend to characterize one off websites like Federal Register 2.0 as “yet another silo”. But I think it’s important to remember that the web was really created to break down the silo walls, and that every well designed web site is actually the antithesis of a silo. In fact, monolithic repository systems that treat all publications as static documents to be uniformly managed are more like silos than these ‘one off’ dedicated web applications.

As a software developer working in the federal government there were a few things about the Federal Register 2.0 that I found really exciting:

• Fruitful collaboration between federal employees and citizen activist/geeks initiated by a software development contest.
• Extensive use of opensource technologies like Ruby, Ruby on Rails, MySQL, Sphinx, nginx, Varnish, Passenger, Apache2, Ubuntu Linux, Chef. Opensource technologies encourage collaboration by allowing citizen activists/technologists to participate without having to drop a princely sum.
• Release of the source code for the website itself, using decentralized revision control (git) so that people can easily contribute changes, and see how the site was put together.
• Extensive use of syndicated feeds to communicate how how content is being added to the site, ical feeds to keep on top of events going on in your area, and detailed XML for each entry.
• The robots.txt file for the site makes the content fully crawlable by web indexers, except for search related portions of the website. Excluding dynamic search results is often important for performance reasons, but much of the article content can be discovered via links, see below about permalinks. They also have made a sitemap available for crawlers to efficiently discover URLs for the content.
• Deployment of the web application to the cloud using Amazon’s EC2 and S3 services. Cloud computing allows computing resources to scale to meet demand. In effect this means that government IT shops don’t have to make big up front investments in infrastructure to make new services available. I guess the jury is still out, but I think this will eventually prove to greatly lower the barrier to innovation in the egov sector. It also lets the more progressive developers in government leap frog ancient technologies and bureaucracies to get things done in a timely manner.
• And last, but certainly not least … now every entry in the Federal Register has a URL!. Permalinks for the Federal Register are incredibly important for citability reasons. I predict that we’ll quickly see more and more people referencing specific parts of the Federal Register in social media sites like Facebook, Twitter and out on the open web in blogs, and in collaborative applications like Wikipedia.

I would like to see more bulk access to XML data made available, for re-purposing on other websites–although I guess it might be able to walk from the syndicated feeds to the detailed XML. Also, the search functionality is so rich it would be useful to have an OpenSearch description that documents it, and perhaps provides some hooks for getting back JSON and/or XML representations. Perhaps even following the lead of the London Gazette and trying to make some of the structured metadata available in the the HTML using RDFa. It also looks like content is only available for 2008 on, so it might be interesting to see how easy it would be to make more of the historic content available.

But the great thing about what these folks have done is now I can fork the project on github, see how easy it is to add the changes, and let the developers know about my updates to see if they are worth merging back into the production website. This is an incredible leap forward for egov efforts–so hats off to everyone who helped make this happen.