home	fourm	entertainment	Quotes	a b c d e f g h i j k l m n o p q r s t u v w x y z 0 1 2 3 4 5 6 7 8 9 
Knowledge Base Robots


Many search engines use programs called robots to gather web pages for indexing. 
These programs are not limited to a pre-defined list of web pages, 
they can follow links on pages they find, which makes them a form of intelligent agent. 
The process of following links is called spidering, wandering, or gathering.
Search engine robots will check a special file in the root of each server called 
robots.txt (eg. www.domain.com/robots.txt). Robots.txt implements the Robots Exclusion 
Protocol, which allows the web site administrator to define what parts of the site are 
off-limits to specific robot user agent names. Web administrators can disallow access 
to cgi, private and temporary directories, for example, because they do not want pages 
in those areas indexed. The following links may intrest you:

Robots tutorial
Robots FAQ