Robots generator
Robots text is one of the crucial part of your web site maintenance.
You can prevent crawlers to index some of your sensitive pages. Of
course you won't want your private page which contain sensitive information
to appear right in search engine result pages. Besides, it is also
very useful in saving your bandwidth.
When a search engine bot visit your site, they will first look
for a file in your root directory named "robots.txt".
Theyl see what they are allowed to do and what don't. If you don't
have any robots file, they would probably think that you allow everything
including your most important part.
Fill in how many directories you want to restrict. Commonly is 2
directories. One is your image file and the other one is your cgi-bin
where you store your scripts.
