Can any one explain briefly, about ROBOT.txt ?

Question

what are the advantages with good examples

Answer 1

a robots.txt (note it robots.txt not robot.txt) file placed on your website is just a simple text file that contains commands that tell webcrawlers (google, yahoo, etc) what they should or should not index in the search engines.  If it doesn't exist, their webcrawlers will index your entire website for their search engines.  But you can put commands in the file that tell them either don't index anything or restrict indexing to certain folders, etc.

for example to instruct crawlers from indexing anything you'd enter this in your text file

User-agent: *

Disallow: /

For a complete list of commands and explanation of robots.txt go to:

http://www.outfront.net/tutorials_02/adv...

or just google "robots.txt" to find plenty of websites that show you how to use this tool.

Answer 2

Robots.txt is a small text file use to allow or disallow search engine bots to index/collect your web site infomation. For example

To allow:

-----

User-agent: *

Allow:

-----

Disallow:

-----

User-agent: *

Disallow:

You can also specify the file or directory too.