Robots.txt Generator


Default - All Robots are:  
    
Crawl-Delay:
    
Sitemap: (leave blank if you don't have) 
     
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo
  Yahoo MM
  Yahoo Blogs
  Ask/Teoma
  GigaBlast
  DMOZ Checker
  Nutch
  Alexa/Wayback
  Baidu
  Naver
  MSN PicSearch
   
Restricted Directories: The path is relative to root and must contain a trailing slash "/"
 
 
 
 
 
 
   



Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.


Robots.txt Generator is a handy tool to generate a Robots.txt file in order to direct Google and other search engines that which page should be indexed by the crawlers and which page should be excluded and not be indexed. Creating Robots.txt is not a big deal, it's so easy and simple to generate the file.

What is the Robots.txt file?

Robots.txt is a text file that is created by webmasters to tell or instruct the search engine's crawlers how to crawl webpages on their website. The file instructs the robots which page should be crawled and which shouldn't be.

Actually, this file is part of Robots Exclusion Protocol (REP), it's the web standard that regulates the robot's crawling function and tells them how to crawl the website, and which page should be index. The crawl instructions are specified by the terms such as "disallowing" and "allowing".

Structure of Robots.txt:

The basic structure or format of the file is listed below;

User-Agent: [user-agent name]

Disallow:[URL string not to be crawled]

Directives in a Robots.txt File:

There are many directives or terms used in the robots.txt file, some important directives are listed below;

Crawl-Delay: This directive is very important and is used to prevent the crawlers from overloading the host. It's treated differently by different search engine bots. For Bing, it's a time interval in which the bot will visit the site. And for Google, you can control the visits of the bots by using Search Console.

User-Agents: This indicates the name of a specific crawler such as Googlebot for Google, baiduspider for Baidu, bingbot for Bing, and * for all web crawlers.

Allowing: Allowing means that a specific URL is allowed to be crawled by bots. You can add as many URLs as you want.

Disallowing: Disallowing means that a specific page or URL is not allowed to be crawled by bots. Add many URLs which you don't want to be crawled and indexed.

Difference Between Sitemap and Robot.txt?

A sitemap is a structure or blueprint of the website that the Search Engine's robots crawl and index all the website (all pages and posts) whereas robot.txt instruct crawlers which page to crawl and which to not crawl. A sitemap is mandatory in order to index your site whereas robots.txt is not necessary, you can run a site without robot.txt but it's highly recommended that you generate robot.txt.

How to Use Robot.txt Generator?

In order to use this tool, follow the steps below:

  • Visit the tool by clicking on the Robot.txt Generator or navigate through SEOToolsCloud's site and find the tool by its name.

  • Enter all detail that is being asked such as Crawl-Delay, Sitemap, and other information.

  • Once you fill in all the fields, now click on the "Create Robots.txt" button or hit enter.

  • Wait for few seconds and you'll get the result.


Related Tools