Comments on: crawling bibliographic data http://inkdroid.org/journal/2009/01/22/crawling-bibliographic-data/ $pithy_personal_mission_statement Wed, 10 Mar 2010 02:04:23 +0000 http://wordpress.org/?v=2.9.2 hourly 1 By: teetsm http://inkdroid.org/journal/2009/01/22/crawling-bibliographic-data/comment-page-1/#comment-80868 teetsm Fri, 20 Feb 2009 19:41:26 +0000 http://inkdroid.org/journal/?p=663#comment-80868 Ed, I wanted to jump in and clarify something... You are absolutely right in your description of robots.txt and potential implications. What is not obvious from looking at our robots.txt is that we have alternative methods in place with the major engines. When our relationships with the engines started many years ago, it was necessary to put special feeds to accommodate the size and structure of Worldcat. We worked directly with them to provide the data in a way that worked best for their services. There are aspects such as our robots.txt and no-follows that are remnants of their requests in previous years and prevented multiple feeds from conflicting within their environments. With advances and changes within the engines and our services, much of what we did then can now be accomplished more standard methods. You will see some changes such as what Th suggests above, as well as others over the next while as we transition away from the older models with the search engine partners. Mike Teets, OCLC Ed,

I wanted to jump in and clarify something… You are absolutely right in your description of robots.txt and potential implications. What is not obvious from looking at our robots.txt is that we have alternative methods in place with the major engines. When our relationships with the engines started many years ago, it was necessary to put special feeds to accommodate the size and structure of Worldcat. We worked directly with them to provide the data in a way that worked best for their services. There are aspects such as our robots.txt and no-follows that are remnants of their requests in previous years and prevented multiple feeds from conflicting within their environments. With advances and changes within the engines and our services, much of what we did then can now be accomplished more standard methods. You will see some changes such as what Th suggests above, as well as others over the next while as we transition away from the older models with the search engine partners.

Mike Teets, OCLC

]]> By: Thom http://inkdroid.org/journal/2009/01/22/crawling-bibliographic-data/comment-page-1/#comment-80852 Thom Fri, 13 Feb 2009 21:54:03 +0000 http://inkdroid.org/journal/?p=663#comment-80852 Actually, we notice the major crawlers and send them HTML rather than the raw XML. We do this if we think you are on a mobile device too, since most the the current browsers on phones don't handle the XML+XSLT very well. We've also turned off the 'do not follow' header to make harvesting of Identities work better. --Th Actually, we notice the major crawlers and send them HTML rather than the raw XML. We do this if we think you are on a mobile device too, since most the the current browsers on phones don’t handle the XML+XSLT very well.

We’ve also turned off the ‘do not follow’ header to make harvesting of Identities work better.

–Th

]]>