Advertise    My Panel    Home    Directory    Free Content
Web tools
Web Tools
  • Link Popularity Check
  • XML Extractor
  • Google Sitemap
  • Form Generator
  • Robots Generator
  • Meta Tag Generator
  • Domain Generator
  • HTML Converter
  • CSS Generator
  • MD5 Email Protector
  • Countdown Generator
  • Photo Gallery generator

  • Testing Tools
  • Broken link Checker
  • Meta Crawler
  • Bandwidth Calculator
  • Simulator
  • Density Analyzer
  • Download time
  • Whois check

  • Other Resources
  • Free Web Content
  • Articles
  • HTML Help

  • Free Scripts
  • PHP scripts

  • Place your ads here

    Robots generator
    Robots text is one of the crucial part of your web site maintenance. You can prevent crawlers to index some of your sensitive pages. Of course you won't want your private page which contain sensitive information to appear right in search engine result pages. Besides, it is also very useful in saving your bandwidth.

    When a search engine bot visit your site, they will first look for a file in your root directory named "robots.txt". Theyl see what they are allowed to do and what don't. If you don't have any robots file, they would probably think that you allow everything including your most important part.

    Fill in how many directories you want to restrict. Commonly is 2 directories. One is your image file and the other one is your cgi-bin where you store your scripts.