Clango is a well-mannered robot!

Welcome to, the new home of The Web Robots Pages.

This is the main source for information on the robots.txt Robots Exclusion Standard and other articles about writing well-behaved Web robots.

Thanks to WebCrawler and Excite for having hosted these pages at over the years. || Thanks also to Diesel Sweeties for the Clango cartoon. || Maintainer: Martijn Koster