Best Practices for Setting Up Meta Robots Tags & Robots.txt

Best Practices for Setting Up Meta Robots Tags & Robots.txt
‘ );

h3_html = ‘



cta = ‘‘+cat_head_params.cta_text.textual content+’
atext = ‘


scdetails = scheader.getElementsByClassName( ‘scdetails’ );
sappendHtml( scdetails[0], h3_html );
sappendHtml( scdetails[0], atext );
sappendHtml( scdetails[0], cta );
// brand
sappendHtml( scheader, “” );
sc_logo = scheader.getElementsByClassName( ‘sc-logo’ );
logo_html = ‘‘;
sappendHtml( sc_logo[0], logo_html );

sappendHtml( scheader, ‘


‘ );

if(“undefined”!=typeof __gaTracker)
} // endif cat_head_params.sponsor_logo

Is it simply me, or does the phrases “meta robots tags” and “robots.txt” sound like one thing Schwarzenegger stated in “Terminator 2”?

That’s one motive why I began working in search engine optimization — it appeared futuristic however overwhelmingly techy for my abilities on the time.

Hopefully, this text makes establishing your meta robots tags and robots.txt information much less nauseating. Let’s get began.

Meta Robots Tags vs. Robots.txt

Before we dig into the fundamentals of what meta robots tags and robots.txt information are, it’s essential to know that there’s not one facet that’s higher than the opposite to make use of in search engine optimization.

Robots.txt information instruct crawlers about your entire website.

While meta robots tags get into the nitty-gritty of a selected web page.

I favor to make use of meta robots tags for many issues that different search engine optimization professionals could use the simplicity of the robots.txt file.

There isn’t any proper or unsuitable reply. It’s a private desire primarily based in your expertise.

What Is Robots.txt?

A robots.txt file tells crawlers what ought to be crawled.

It’s a part of the robots exclusion protocol (REP).

Googlebot is an instance of a crawler.

Google deploys Googlebot to crawl web sites and file data on that website to know the right way to rank the location in Google’s search outcomes.

You can discover any website’s robots.txt file by add /robots.txt after the online handle like this:

Here is what a primary, contemporary, robots.txt file seems to be like:

robots txt file example

The asterisk * after user-agent tells the crawlers that the robots.txt file is for all bots that come to the location.

The slash / after “Disallow” tells the robotic to not go to any pages on the location.

Here is an instance of Moz’s robots.txt file.

moz robots file

You can see they’re telling the crawlers what pages to crawl utilizing user-agents and directives. I’ll dive into these a little bit later.

Why Is Robots.txt Important?

I can’t inform what number of purchasers come to me after an internet site migration or launching a brand new web site and ask me: Why isn’t my website rating after months of labor?

I’d say 60% of the reason being that the robots.txt file wasn’t up to date accurately.

Meaning, your robots.txt file nonetheless seems to be like this:

robots txt file example

This will block all internet crawlers are visiting your website.

Another motive robots.txt is essential is that Google has this factor referred to as a crawl price range.

Google states:

“Googlebot is designed to be a good citizen of the web. Crawling is its main priority, while making sure it doesn’t degrade the experience of users visiting the site. We call this the “crawl rate limit,” which limits the utmost fetching fee for a given website.

Simply put, this represents the variety of simultaneous parallel connections Googlebot might use to crawl the location, in addition to the time it has to attend between the fetches.”

So, in case you have a giant website with low-quality pages that you simply don’t need Google to crawl, you possibly can inform Google to “Disallow” them in your robots.txt file.

This would unencumber your crawl price range to solely crawl the high-quality pages you need Google to rank you for.

There aren’t any exhausting and quick guidelines for robots.txt information…but.

Google introduced a proposal in July 2019 to start implementing sure requirements, however for now, I’m following one of the best practices I’ve completed for the previous few years.

Robots.txt Basics

How to Use Robots.txt

Using robots.txt is important for search engine optimization success.

But, not understanding the way it works can go away you scratching your head as to why you aren’t rating.

Search engines will crawl and index your website primarily based on what you inform them to within the robots.txt file utilizing directives and expressions.

Below are frequent robots.txt directives it is best to know:

User-agent: * — This is the primary line in your robots.txt file to clarify to crawlers the principles of what you need them to crawl in your website. The asterisk informs all spiders.

User-agent: Googlebot — This tells solely what you need Google’s spider to crawl.

Disallow: / — This tells all crawlers to not crawl your whole website.

Disallow: — This tells all crawlers to crawl your whole website.

Disallow: /staging/ — This tells all crawlers to disregard your staging website.

Disallow: /ebooks/* .pdf — This tells crawlers to disregard all of your PDF codecs which can trigger duplicate content material points.

User-agent: Googlebot

Disallow: /pictures/ — This tells solely the Googlebot crawler to disregard all pictures in your website.

* — This is seen as a wildcard that represents any sequence of characters.

$ — This is used to match the tip of the URL.

To create a robots.txt file, I take advantage of Yoast for WordPress. It already integrates with different search engine optimization options on my websites.

But, earlier than you begin to create your robots.txt file, listed here are just a few fundamentals to recollect:

  • Format your robots.txt accurately. SEMrush does an important instance of how a robots.txt ought to be correctly formatted. You see the construction follows this sample: User-agent → Disallow → Allow → Host → Sitemap. This lets search engine spiders entry classes and internet pages in the proper order.

semrush robots

  • Make positive that each URL you wish to “Allow:” or “Disallow:” is positioned on a separate line as Best Buy does beneath. And, don’t separate with spacing.

webceo robots

  • Always use lowercase to call your robots.txt like WebCEO does.

webceo robots

  • Do not use any particular characters besides * and $. Other characters will not be acknowledged.
  • Create separate robots.txt information for completely different subdomains. For instance, “” and “” have particular person information and each have completely different robots.txt information.
  • Use # to depart feedback in your robots.txt file. Crawlers don’t honor strains with the # character like I did right here with this robots.txt file.
  • If a web page is disallowed within the robots.txt information, the hyperlink fairness is not going to move.
  • Never use the robots.txt to guard or block delicate information.

What to Hide with Robots.txt

Robots.txt information are sometimes used to exclude particular directories, classes, or pages from the SERPs.

You can exclude through the use of the “disallow” directive.

Here are just a few frequent pages I conceal utilizing a robots.txt file:

  • Pages with duplicate content material (usually printer-friendly content material)
  • Pagination pages
  • Dynamic product and repair pages
  • Account pages
  • Admin pages
  • Shopping cart
  • Chats
  • Thank you pages

This is tremendous useful for ecommerce websites utilizing parameters like Macy’s does.

Best Practices for Setting Up Meta Robots Tags & Robots.txt

And, you possibly can see right here how I disallowed a thank web page.

leadfeeder robots

It’s essential to know that not all crawlers will comply with your robots.txt file.

Bad bots can fully ignore your robots.txt file, so ensure you’re not maintaining delicate information on blocked pages.

Common Robots.txt Mistakes

After managing robots.txt information for 10+ years now, listed here are just a few of the frequent errors I see:

Mistake #1: The File Name Contains Upper Case

The solely potential file identify is robots.txt, nor Robots.txt or ROBOTS.TXT.

Stick to lowercase, at all times with regards to search engine optimization.

Mistake #2: Not Placing the Robots.Txt File within the Main Directory

If you need your robots.txt file to be discovered, it’s important to place it in the primary listing of your website.



Mistake #three: Incorrectly Formatted User-Agent


Disallow: Googlebot


User-agent: Googlebot

Disallow: /

Mistake #four: Mentioning Multiple Catalogs in One ‘Disallow’ Line


Disallow: /css/ /cgi-bin/ /pictures/


Disallow: /css/

Disallow: /cgi-bin/

Disallow: /pictures/

Mistake #5: Empty Line in ‘User-Agent’





User-agent: *


Mistake #6: Mirror Websites & Url within the Host Directive

Be cautious when mentioning ‘host’ directives, in order that search engines like google perceive you accurately:


User-agent: Googlebot

Disallow: /cgi-bin


User-agent: Googlebot

Disallow: /cgi-bin


If your website has https, the proper choice is:

User-agent: Googlebot

Disallow: /cgi-bin


Mistake #7: Listing All the Files Within the Directory


User-agent: *

Disallow: /pajamas/flannel.html

Disallow: /pajamas/corduroy.html

Disallow: /pajamas/cashmere.html


User-agent: *

Disallow: /pajamas/

Disallow: /shirts/

Mistake #eight: No Disallow Instructions

The disallow directions are required in order that search engine bots perceive your intent.


User-agent: Googlebot



User-agent: Googlebot



Mistake #9: Blocking Your Entire Site


User-agent: Googlebot

Disallow: /


User-agent: Googlebot


Mistake #10: Using Different Directives within the * Section


User-agent: *

Disallow: /css/



User-agent: *

Disallow: /css/

Mistake #11: Wrong HTTP Header


Content-Type: textual content/html


Content-Type: textual content/plain

Mistake #12: No Sitemap

Always place your sitemaps on the backside of your robots.txt file.


robots no sitemap


robots correct sitemap

Mistake #13: Using Noindex

Google introduced in 2019 that it will now not acknowledge the noindex directive utilized in robots.txt information.

So, use the meta robots tags I speak about beneath as a substitute.


noindex thank you robots


robots thank you

Mistake #14: Disallowing a Page within the Robots.Txt File, however Still Linking to It

If you disallow a web page within the robots.txt file, Google will nonetheless crawl the web page in case you have inner hyperlinks pointing to it.

You must take away these hyperlinks for spiders to cease crawling that web page fully.

If you’re ever not sure, you possibly can examine which pages are being listed in your Google Search Console Coverage report.

You ought to see one thing like this:

And, you should use Google’s robots.txt testing software.

However, should you’re utilizing the mobile-friendly take a look at software by Google, it doesn’t comply with your guidelines within the robots.txt file.

robots google search console

What Are Meta Robots Tags?

Meta robots tags (additionally referred to as meta robots directives) are HTML code snippets that inform search engine crawlers the right way to crawl and index pages in your web site.

The meta robots tags are added to the <head> part of an internet web page.

Here is an instance:

<meta identify=”robots” content material=”noindex” />

Ther meta robots tags are made up of two elements.

The first a part of the tag is identify=’’’.

This is the place you establish the user-agent. For instance, “Googlebot.”

The second a part of the tag is content material=’’. This the place you inform the bots what you need them to do.

Types of Meta Robots Tags

Meta robots tags have two forms of tags:

  • Meta robots tag.
  • X-robots-tag.

Type 1: Meta Robots Tag

Meta robots tags are generally utilized by search engine optimization entrepreneurs.

It permits you to inform user-agents (suppose Googlebot) to crawl particular areas.

Here is an instance:

<meta identify=”googlebot” content material=”noindex,nofollow”>

This meta robots tag tells Google’s crawler, Googlebot, to not index the web page in the various search engines and to not comply with any backlinks.

So, this web page wouldn’t be a part of the SERPs.

I’d use this meta robots tag for a thanks web page.

Here is an instance of a thanks web page after you obtain an book.

leadfeeder thank you page

Now, should you take a look at the backend code, you’ll see it says noindex and nofollow.

noindex nofollow

If you’re utilizing completely different meta robots tag directives for completely different search user-agents, you’ll want to make use of separate tags for every bot.

It’s essential that you don’t place the meta robots tags outdoors of the <head> part. Glenn Gabe reveals you why on this case research.

Type 2: X-robots-tag

The x-robots-tag permits you to do the identical factor because the meta robots tags however throughout the headers of an HTTP response.

Essentially, it offers you extra performance than the meta robots tags.

However, you will want entry to the .php, .htaccess, or server information.

For instance, if you wish to block a picture or video, however not your entire web page, you’ll use x-robots-tag as a substitute.

Meta Robots Tag Parameters

There are some ways to make use of meta robots tag directives within the code. But, first, you’ll want to perceive what these directives are and what they do.

Here is a break down of meta robots tag directives:

  • all – No limitations for indexing and content material. This directive is being utilized by default. It has no affect on the various search engines’ work. I’ve used it as a brief lower for index, comply with.
  • index – Allow search engines like google to index this web page of their search outcomes. This is a default. You don’t want so as to add this to your pages.
  • noindex – Removes the web page from the various search engines index and search outcomes. This means searchers is not going to discover your website or click on via.
  • comply with – Allows search engines like google to comply with the interior and exterior backlinks on that web page.
  • nofollow – Do not enable following the interior and exterior backlinks. This means these hyperlinks is not going to move hyperlink fairness.
  • none – The similar as noindex, and nofollow meta tags.
  • noarchive – Do not present the ‘Saved Copy’ hyperlink within the SERPs.
  • nosnippet – Do not present the prolonged description model of this web page within the SERPs.
  • notranslate – Do not supply this web page’s translation within the SERPs.
  • noimageindex – Do not index the on-page pictures.
  • unavailable_after: [RFC-850 date/time] – Do not present this web page within the SERPs after specified date/time. Use RFC 850 format.
  • max-snippet – Establishes a most quantity for the character depend within the meta description.
  • max-video-preview – Establishes the variety of seconds a video will preview.
  • max-image-preview – Establishes a most dimension for the picture preview.

Sometimes, completely different search engines like google settle for completely different meta-tag parameters. Here is a breakdown:

Value Google Bing Yandex
index Yes Yes Yes
noindex Yes Yes Yes
none Yes Doubt Yes
noimageindex Yes No No
comply with Yes Doubt Yes
nofollow Yes Yes Yes
noarchive Yes Yes Yes
nosnippet Yes No No
notranslate Yes No No
unavailable_after Yes No No

How to Use Meta Robots Tags

If you’re utilizing a WordPress web site, there are lots of plugin choices for you to tailor your meta robots tags.

I favor to make use of Yoast. It’s an all-in-one search engine optimization plugin for WordPress that gives a number of options.

But, there’s additionally Meta Tags Manager plugin and GA Meta Tags plugin.

Joomla customers, I like to recommend EFSEO and Tag Meta.

No matter what your website is constructed on, listed here are three tricks to utilizing meta robots tags:

  • Keep it case delicate. Search engines acknowledge attributes, values, and parameters in each uppercase and lowercase. I like to recommend that you simply follow lowercase to enhance code readability. Plus, should you’re an search engine optimization marketer, it’s finest to get within the behavior of utilizing lowercase.
  • Avoid a number of <meta> tags. Using a number of meta tags will trigger conflicts in code. Use a number of values in your <meta> tag, like this: <meta identify=“robots” content material=“noindex, nofollow”>.
  • Do not use conflicting meta tags to keep away from indexing errors. For instance, in case you have a number of code strains with meta tags like this <meta identify=“robots” content material=“follow”> and this <meta identify=“robots” content material=“nofollow”>, solely “nofollow” will likely be considered. This is as a result of robots put restrictive values first.

Robots.txt & Meta Robots Tags Work Together

One of the most important errors I see when engaged on my shopper’s web sites is when the robots.txt file doesn’t match what you’ve acknowledged within the meta robots tags.

For instance, the robots.txt file hides the web page from indexing, however the meta robots tags do the other.

Remember the instance from Leadfeeder I confirmed above?

So, you’ll discover that this thanks web page is disallowed within the robots.txt file and utilizing the meta robots tags of noindex, nofollow.

In my expertise, Google has given precedence to what’s prohibited by the robots.txt file.

But, you possibly can remove non-compliance between meta robots tags and robots.txt by clearly telling search engines like google which pages ought to be listed, and which shouldn’t.

Final Thoughts

If you’re nonetheless reminiscing in regards to the days of shopping for a Blockbuster film in a strip mall, then the thought of utilizing robots.txt or meta tags should still appear overwhelming.

But, should you’ve already binge-watched “Stranger Things”, welcome to the longer term.

Hopefully, this information offered extra perception into the fundamentals of robots.txt and meta tags. If you had been hoping for robots flying in on jet packs and time journey after studying this publish, I’m sorry.

Image Credits

Featured Image: Paulo Bobita

Source hyperlink search engine optimization

Be the first to comment

Leave a Reply

Your email address will not be published.