# Becomebot is really raping our reviews page. # Googlebot is too but I hate to do anything to discourage Google's bot # from archiving as much as possible. # We'll probably just redesign the page to use javascript sorting # to minimise "unique" pages that bots can crawl User-agent: BecomeBot Crawl-Delay: 600 Disallow: /cgi-bin #Wow, sooooooooo many poll votes from bots User-agent: * Disallow: submit_vote.html