-
Standard being developed on behalf of content publishers to communicate permissions information more extensively than is the case with robots.txt. Project documents, implementation and background information.
-
Search Tools Consulting explains how the search engine programs called "robots" or "spiders" work, and reviews related sites.
-
This large database lists user agents in categories and distinguishes between robots and browsers.
-
An alphabetical list of user agents and the deployer behind them, compiled by Christoph Rüegg.
-
A list from PGTS of Web robots with the identifying data they leave in Web site logs.
-
Brian Dunnintg provides a list of all the major search engine robot IP addresses, by full class C only.
-
Information on the robots.txt Robots Exclusion Standard and other articles about writing well-behaved Web robots.
-
Lists IP addresses of search engine spiders. Can be searched by IP address. Also links to resources on spiders.
-
John A. Fotheringham presents data in tabular form on the robots sent by search engines and other sites to read and index Web pages: their origins, names and IP addresses.
-
Tool from ASAP Consulting s.r.o. for detailed user agent string analysis using an online form. Includes databases of browsers and robots.
-
Large list of search engine spiders, similar web robots, and Web browsers: their web-log identification and links to their originators.
