Comments on: cloaking and fulltext

By: ed

ed — Wed, 11 Nov 2009 14:17:15 +0000

@sgillies in Chronicling America we do the same for all the big search engine bots. But, I agree: it does seem like the out-of-band coordination breaks the web a bit. Sometimes I try to rationalize it as a variant of content-negotiation, similar to what happens in practice on the mobile web…but it doesn’t work for very long. I’m definitely open to other solutions. I wish that rel=’canonical’ could help here, but I don’t think it does. It would be nice if there were some rel=”fulltext” or something that bots could follow. Perhaps @ardvaark is right and we should just bite the bullet now. But what does that do for @martin’s problem? /me shrugs

By: sgillies.net/

sgillies.net/ — Wed, 11 Nov 2009 13:37:24 +0000

But what about other non-Google, non-browser agents? Would you want them to masquerade as the Googlebot, or arrange with you for the same special treatment (which you’d readily provide, I don’t mean to imply otherwise at all)? Either seems to break the web a bit, yes? Tricky situation.

By: Martin Haye

Martin Haye — Wed, 11 Nov 2009 00:14:49 +0000

@aardvark: Trouble is some of our (CDL’s) items are entire monographs. Even compressed these would add quite a bit of overhead to each page view. Admittedly we’re not totally optimized for tiny downloads, but currently we *could* optimize for size. Serving up the entire OCR text would foil that.

This is a subtle point and we debated it quite a bit. In the end we went with what’s practical, and put our hopes in Google recognizing that we’re not cloaking — we’re giving them what they need. Search engines need text, people need the whole page experience.

By: ed

ed — Tue, 10 Nov 2009 18:31:32 +0000

@gluejar I agree in principle — but google’s docs don’t really say that. The worry at LC was that If google are trying to identify cloaked content at scale on the web they may inadvertently flag Chronicling America content as cloaked — since determining juicy-ness could be infeasible.

By: gluejar

gluejar — Tue, 10 Nov 2009 17:38:01 +0000

What CA is doing is not cloaking- it’s more akin to content-type negotiation. The one thing I’d worry about is that the OCR is poor enough for your example page that a human reviewing the text could get confused and think that it’s not the ocr of the image.

Cloaking is when you give a spider juicy content and then give spam to a human.

By: ardvaark.net/identity/br…

ardvaark.net/identity/br… — Tue, 10 Nov 2009 17:31:25 +0000

How significantly will it impact page loads? Just on my poking around, with a couple of test pages, I’m showing a text-heavy page with 41K bytes in the OCR text. If GZIP encoding were enabled, that would knock it down to 17K. Viewing that image with all caching working correctly is 158K, so you’re talking about a 10% increase in size, although most of that is in the JPG and not the HTML.

This is one of those nasty situations for which there is no good answer. The reality here is that, like a Major League Umpire, “correct” is whatever Google says it is on any given day.