How to create seo friendly robots.txt for improve seo


Robots.txt is the important back end of websites for allow bots to crawl site. Here is the tutorial for How to create seo friendly robots.txt file. Blogger and wordpress blogs have different types of robots.txt for allow bots to crawl site. Crawling bots of google and other search engines working as per robots.txt rules. So we have to make this more seo friendly for improve search engine bot performance. Every line have command with Allow and Disallow tag that drive bots to pages for crawling and indexing. So that you can make your site search engine visibility as per your wish. Here am sharing tutorial about robots.txt for make your site more seo friendly.

Basically you can track your Robots tag in your root domain with robots.txt after the domain name. For example, Here is my robots.txt file  here am using disallow tag for restrict search engine bots to my wp admin dashboard. Many bloggers have some questions about their website index status of other pages. If you made mistakes when your robots tag missing important lines. For example, if you want to disable search engine access for your blog’s search results section, You can simply use Disallow tag and specify Search in the line of your file. If you are newbie in editing robots.txt file and searching for blogger seo friendly robots.txt file, This post will helps you to solve your issues about search engine bot access of your web pages.

Best Seo Friendly Custom Robots.Txt For Blogger :

When you create a blog in blogger, Your robots.txt file will be created as default and you can modify it yourself for make your site seo friendly. Here am sharing this seo friendly robots.txt for bloggers. Follow the below steps to edit robots.txt in blogger for more seo.

  • Sign in to your blogger dashboard.
  • Click on settings and choose Search Preferences.
  • Here click on Enable custom robots.txt content? and select Yes.
  • Now a box will appear for paste your custom robots.txt file.
  • Simply paste below seo friendly robots.txt code for blogger :
User-agent: *
Allow: /
Allow: /*?m=*
Allow: /atom.xml?redirect=false&start-index=1&max-results=100000
Allow: /sitemap.xml?page=1
Allow: /sitemap.xml?page=2
Disallow: /search
Disallow: /*?*
Disallow: /*=*

Screenshot :

  • Now choose the Custom Robots header tags and customize it as below :

Then save it and open google webmaster tools. Here select crawl section and choose Robots.Txt testet. Then update your robots.txt file here and confirm your new robots.txt file. It is important to update your robots.txt file in google webmaster tools for crawling your site in future.

Best Seo Friendly Custom Robots.Txt For WordPress :

Fo wordpress you can edit your robots.txt file directly via yoast seo plugin. Here is the steps to edit robots.txt in wordpress.

  • Login to your wordpress site wp admin dashboard.
  • Click on SEO and select Tools section.
  • Here select file editor and choose Robots.Txt editor.
  • Simply paste below robots.txt in the text box :
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /comments/feed/
Disallow: /trackback/
Disallow: /index.php
Disallow: /xmlrpc.php
Disallow: /wp-content/plugins/
Disallow: /uploads/
Disallow: *?nonamp=1

User-agent: NinjaBot
Allow: /

User-agent: Mediapartners-Google*
Allow: /

User-agent: Googlebot-Image
Allow: /wp-content/uploads/

User-agent: Adsbot-Google
Allow: /

User-agent: Googlebot-Mobile
Allow: /


Then save the robots file and update your robots.txt in google webmaster tools. Sometimes you may face some errors like file editor not available in yoast seo, In this time you can go to your website cpanel and paste your robots.txt file directly in the file manager > Public files. Then save the file and open for confirm the changes.

  • How to ping sitemap to all search engines for faster index

Above robots.txt file recommended and used by many seo experts and bloggers. Make your robots.txt file neat and clean for search engine bots crawling access. I hope this post will helps you to make your site more seo friendly with custom robots.txt tag.



Please enter your comment!
Please enter your name here