The Robots META Tag for Googlebot is meant to provide users who cannot upload or control the /robots.txt file at their websites, with a last chance to keep their content out of Google's indexes and services.
The "robots" tag is obeyed by many different web robots. If you'd like to specify indexing restrictions just for googlebot, you may use "googlebot" in place of "robots".
<meta name="googlebot" content="robots-terms">
Googlebot obeys the
noarchive Robots META Tag. If you place the tag in the head of your HTML/XHTML document, you can cause Google to not index, not follow, and/or not archive particular documents on your site.
content="robots-terms" is a comma separated list used in the Robots META Tag for Google that may contain one or more of the following keywords without regard to case: noindex, nofollow and/or noarchive.
Document will not be indexed by Googlebot.
Internal and external links in the document will not be followed by Googlebot.
Google will not archive a copy of the document (Google's Cached Page).
Google will not display snippets and will not archive a copy of the document (Google's Cached Page). A snippet is a text excerpt from the returned result page that has all query terms bolded. Google Remove Snippets
If this Robots META Tag is missing, or if there is no content, or the robot terms are not specified, then the robot terms will be assumed to be
"index, follow" (e.g.
"all") which is the default indexing behavior for most major search engine spiders.
The tags to include and their effects are:
noindexwill produce the following effect; Googlebot will retrieve the document, but it will not index the document.
<meta name="googlebot" content="noindex">
nofollowwill produce the following effect; Googlebot will not follow any links that are present on the page to other documents.
<meta name="googlebot" content="nofollow">
noarchivewill produce the following effect; "Google automatically takes a 'snapshot' of each page it crawls and archives it. This 'cached' version allows a webpage to be retrieved for your end users if the original page is ever unavailable (due to temporary failure of the page's web server). The cached page appears to users exactly as it looked when Google last crawled it, and we display a message at the top of the page to indicate that it's a cached version. Users can access the cached version by choosing the 'Cached' link on the search results page."
<meta name="googlebot" content="noarchive">
You can also combine any or all of the above
robots-terms into a single META Robots Tag for Google. For example:
<meta name="googlebot" content="noarchive, nofollow">
Googlebot's default indexing behavior is to
index, follow or
all. The below Robots META Tag is not required nor is it suggested in the Google guidelines which clearly state that the use of the Robots META Tag is for restricting the indexing of content.
<meta name="googlebot" content="index, follow">
When utilizing the above Robots META Tag, you are adding page weight (HTML bloat) that is not required. You shift the text to html ratio when inserting the additional code within your documents. Utilizing metadata elements like the example shown above may not present a professional image to both your peers and potential clients.