What is a robots.txt file?

Question

What is a robots.txt file?

Answer 1

A text file located in the root directory of your web site written to instruct search engine robots and spiders where they are allowed to crawl. robots.txt files can be bot specific or written to include all robots who follow robots.txt files.

Answer 2

this is a file that you put on your web server to tell robots how to act when they come and visit your website.  for example, google has one:

http://www.google.com/robots.txt

as you can see, it is always at the root of the website.

inside it, you see a lot of lines, but they all start with one of 3 things -

user agent - this is a string that gets sent with every request on the web. usually, it tells the web server what kind of computer you have, what operating system, and what web browser you have.

allow - this means that the user agent defined above should be allowed to look at the directory that follows this statement.

disallow - fairly obvious.

remember, this won't stop you or me from browsing your site, but it will stop a robot (like googlebot, slurp (yahoo's bot) or msnbot) from indexing those pages on your site into their engine.