Metadata Elements - Googlebot Robots META Tag

The Robots META Tag for Googlebot is meant to provide users who cannot upload or control the /robots.txt file at their websites, with a last chance to keep their content out of Google's indexes and services.

The "robots" tag is obeyed by many different web robots. If you'd like to specify indexing restrictions just for googlebot, you may use "googlebot" in place of "robots".

<meta name="googlebot" content="robots-terms">

Googlebot obeys the noindex, nofollow, and noarchive Robots META Tag. If you place the tag in the head of your HTML/XHTML document, you can cause Google to not index, not follow, and/or not archive particular documents on your site.

The content="robots-terms" is a comma separated list used in the Robots META Tag for Google that may contain one or more of the following keywords without regard to case: noindex, nofollow and/or noarchive.

Document will not be indexed by Googlebot.

Internal and external links in the document will not be followed by Googlebot.

Google will not archive a copy of the document (Google's Cached Page).

Google will not display snippets and will not archive a copy of the document (Google's Cached Page). A snippet is a text excerpt from the returned result page that has all query terms bolded. Google Remove Snippets

If this Robots META Tag is missing, or if there is no content, or the robot terms are not specified, then the robot terms will be assumed to be "index, follow" (e.g. "all") which is the default indexing behavior for most major search engine spiders.

Examples of the Googlebot Robots META Tag

The tags to include and their effects are:

  1. The robots term of noindex will produce the following effect; Googlebot will retrieve the document, but it will not index the document.

    <meta name="googlebot" content="noindex">

  2. The robots term of nofollow will produce the following effect; Googlebot will not follow any links that are present on the page to other documents.

    <meta name="googlebot" content="nofollow">

  3. The robots term of noarchive will produce the following effect; "Google automatically takes a 'snapshot' of each page it crawls and archives it. This 'cached' version allows a webpage to be retrieved for your end users if the original page is ever unavailable (due to temporary failure of the page's web server). The cached page appears to users exactly as it looked when Google last crawled it, and we display a message at the top of the page to indicate that it's a cached version. Users can access the cached version by choosing the 'Cached' link on the search results page."

    <meta name="googlebot" content="noarchive">

You can also combine any or all of the above robots-terms into a single META Robots Tag for Google. For example:

<meta name="googlebot" content="noarchive, nofollow">

Additional information on specific Googlebot Robots META Tags can be found at Google's Web Crawler page and also at Remove Content from Google's Index page.

Misinterpretation of the Standards

Googlebot's default indexing behavior is to index, follow or all. The below Robots META Tag is not required nor is it suggested in the Google guidelines which clearly state that the use of the Robots META Tag is for restricting the indexing of content.

<meta name="googlebot" content="index, follow">

When utilizing the above Robots META Tag, you are adding page weight (HTML bloat) that is not required. You shift the text to html ratio when inserting the additional code within your documents. Utilizing metadata elements like the example shown above may not present a professional image to both your peers and potential clients.

More Information about the Robots META Tag


SEO Consultants Directory