<< BACK                                                                                                                              NEXT >>

A Fundamental Understanding Of Meta robots Tag

In the previous chapter – A Fundamental Understanding Of “robots.txt”,  we learned how to control the web crawlers behavior while crawling our website. In this chapter – A Fundamental Understanding Of Meta robots Tag – let us understand the role of meta “robots” tag to control the behavior of the web crawlers while crawling individual pages of our overall website. You will have a detailed understanding of Meta robots Tag by the time you complete reading this chapter. 

 

Understanding Of Meta robots Tag

 

understanding of meta robots tag

 

Topics Covered In This Chapter

What is the meta “robots” tag?

Meta “robots” tag is a small piece of HTML code that is placed in the <head> section of your page. 

It basically informs web-crawlers ( or search engines or bots or spiders or User-agents ):

  • whether to index your webpage or not
  • whether to follow the links on your page or not. It means it indicates the crawlers whether to pass on the link equity to the outbound links on your page or not.

The code looks like the following:

<meta name=”robots” content=”noindex,nofollow”>

What is the difference between the “robots.txt” and meta “robots” tag?

robots.txt” is a text file present in the root directory of your website that controls the behavior of web crawlers at the website level.

Meta “robots” tag is a small piece of HTML code (basically known as HTML tags), present in the individual pages of a website ( based on the need ) that controls the behavior of the web crawlers at the page level. It is more granular than “robots.txt”.

What are the various directives of the meta “robots” tag?

The various directives of the meta “robots” tag are mentioned below with an example:

a.noindex:

Informs the web-crawler not to index the page

<meta name=”robots” content=”noindex”>

b. nofollow:

Informs the web-crawler not to pass “link equity” or “link juice” to the webpage that is linked from this page. In simple words, not to follow the links on this page.

<meta name=”robots” content=”nofollow”>

c. noimageindex:

Informs the web-crawler not to index any image present in this page.

<meta name=”robots” content=”noimageindex”>

d. none:

Informs the web-crawler not to index as well as not to follow the links on this page. This is equivalent to using both the “noindex” and “nofollow” tags simultaneously.

<meta name=”robots” content=”none”>

is the same as writing

<meta name=”robots” content=”noindex,nofollow”>

e. index:

Informs the web-crawler to index the page. Note that, whether you use this tag or not, the web crawler by default indexes every page unless you explicitly instruct it not to crawl, either by using robots.txt or using “noindex” directive.

<meta name=”robots” content=”index”>

f. follow:

Informs the web-crawler to pass “link equity” or “link juice” to the webpage that is linked from this page. In simple words, it instructs web-crawlers to follow the links on this page.

<meta name=”robots” content=”follow”>

g. nosnippet:

Informs the web-crawler not to show the meta “description” of this page on search engine result pages.

h. unavailable_after:

Informs the web-crawler when it should no longer crawl that page.

<meta name=”robots” content=”unavailable_after: 1-Nov-2018 18:00:00 IST”>

i. noarchive:

Informs the web-crawler not to show a “cached” link in search results.

<meta name=”robots” content=”noarchive”>

j. notranslate:

Informs the web-crawler not to offer the translation of this page in search results.

<meta name=”robots” content=”notranslate”>

Note: To specially instruct one particular web crawler, use the name of the web crawler in the meta name section as shown below:

<meta name=”googlebot” content=”notranslate”>

What is the difference between meta “robots” tag and “X-Robots-Tag?

There is no much difference between meta “robots” tag and “X-Robots-Tag”. The meta “robots” tag is present as part of the head section of HTML code in the individual pages. Whereas, the “X-Robots-Tag” can be used as an element of the HTTP header response for a given URL of the web page. Both control the behavior of the web crawlers for an individual page.

An example of X-Robots-Tag instructing “Googlebot” not to follow the page and for others, web-crawlers not to index and not to follow is given below:

HTTP/1.1 200 OK

Date: Tue, 25 May 2010 21:42:43 GMT

(…)

X-Robots-Tag: googlebot: nofollow

X-Robots-Tag: otherbot: noindex, nofollow

(…)

If you have used “robots.txt” and “meta “robots” tag, there is no need to explicitly use “X-Robots-Tag”

Which one to use – “robots.txt” or meta “robots” tag?

Which one to use depends on the need of the websites or individual pages’. If you wish to disallow an entire page for the crawler to crawl, “robots.txt” is preferable. But, there are many malicious crawlers who ignore such instructions. In such cases, it is better to use a meta “robots” tag for the individual pages.

As a generic rule, if you wish to instruct the crawler at a granular level, say, for example, to follow a link on the page or not, meta “robots” tag is the best option.

 

Congratulations! You are done with the fifteenth chapter on “A Fundamental Understanding Of Meta – robots Tag”. Hope you enjoyed the reading.

Congratulations

All the best for your next chapter on “7 – Step Method Towards Generating Website Content ”. In the next chapter, you will learn the exact steps that need to be taken while generating content for your website.

Feel free to comment below if this blog post was useful or not. If yes, please do me a favor by sharing it with others who might benefit.

<< BACK                                                                                                                              NEXT >>

Interested In Full Time Digital Marketing Course?

Feel free to check out the modules covered in DIGITAL MARKETING TRAINING

Interested In SEO Course?

Feel free to check out the modules covered in SEO TRAINING 

Subhash.K.U is a Professional Programmer turned Digital Marketing Enthusiast. He is the most sought marketing consultants for small and medium scale businesses. He founded Subhash Digital Academy to teach professional digital marketing skills to students, entrepreneurs, and working professionals. He holds a Bachelor’s degree in Electrical Engineering and is an Oracle Certified Programmer. He also holds certificates of Google AdWords, Facebook Blueprint and Hubspot Marketing. He is the co-author of the best selling book – Cracking The C, C++ and Java Interview published by McGraw Hill. He is now penning another book on the subject of marketing and entrepreneurship.

No Comment

You can post first response comment.

Leave A Comment

Please enter your name. Please enter an valid email address. Please enter a message.