Blind bots and worker bees

by Craig Raw

This post I'm going to talk about something that affects everyone in this space intimately on a daily basis. It's those busy little creatures released from their maternal nest to scuttle around the web in a harvesting frenzy, gorging for the glory of the hive back home. I use the word hive assiduously, for the activity resembles in many ways a colony of worker bees on the first day of spring, laden with nectar to satiate an ever demanding need back home. Yes, I'm talking about search bots - just trade nectar for information and flowers for websites. And, there is cross pollination in the way that a bot finding a link from one website to the next may value the latter on the strength of the former, and so cause fertilisation and growth.

But bees in many ways outperform their digital counterparts - they are adaptable and efficient, and all flowers are plunderable and plundered. But here is where the metaphor ends, for flowers have, despite their value in beauty to us, expressly evolved for the purpose of attracting insects. (One may argue that many species have benefitted from human attention due to their appearance, but this is recent on a Darwinian scale.) Websites, on the other hand, must attract humans, be usable by them, and, more recently, still be accessible to digital consumption. This is no minor task - we have here a fundamental difference in approach.

Bots began by trying to mimic human readers as closely as possible. But, to quote Orgel, evolution is smarter than you are. Despite many years of effort by many very clever people, bots remain blind in many ways.

Take, for example, frames. Everyone knows a bot does not like frames, and deframing a site is often a first step in the search optimisation process. We humans can see and understand them (usually), but a bot remains blissfully unaware of all but the most simple framesets and their constituents. Cookies are another blindspot. Search bots as a rule do not support them, yet any site above the most simple brochure level uses cookies to understand a series of clicks as a clickstream. I'd really like to know why bots can't support cookies, because it seems an easy enough thing to do. But anyway, they don't, and as a result a good portion of many websites are unspiderable.

Then we have Flash animation, for which some support seems to be promised, but not worth relying on. Add to this the tremendous advantage humans have in cognitive understanding of pictures and you start appreciating just how wide the gulf is.

So, in order to help the ailing bots, we started to design and build our sites with them in mind. The HTTP standard has been circumvented to avoid URLs that look dynamic (although they really are). Flash is avoided, cookies made non-compulsory, frames are removed and copy is rewritten to attract the digital bees. Some sites take it too far, losing sight of the needs of their original human audience.

There will clearly be compromises necessary for some time in the digital world for all to share the nectar, and in fact it will probably take a Web version 2 to give the bots a fighting chance. Until then, we have the SEO industry. Helping the blind.

2005/06/17 | permalink | comments (0) | trackbacks (0)

  Bookmark this on ... BlinkBits BlinkList Blogmarks Buddymarks CiteUlike Connotea del.icio.us Digg it FeedMarker feedmelinks Furl Give a Link Gravee igooi Lilisto Linkagogo Linkroll ma.gnolia Maple.nu Netvouz Onlywire RawSugar reddit Scuttle Shadows Simpy Sphinn Spurl tagtooga TalkDigger Wink Yahoo MyWeb