There are many misconceptions about applying nofollow, noindex and robots.txt restriction. Whether or not it has any backlinking value, does it get crawled?, do they index it?…and so on! Well, fortunately we have specific answers to all these questions. However different search engines might differently look at them, I think nobody will mind if we discuss only about Google and Yahoo!
First of all let me clear few things,
Nofollow can be applied in two ways, either through Meta tags or link attribute.
You may make a specific link nofollow like rel=”nofollow” – generally reffered as nofollow attribute
Or use <meta name=”robots” content=”nofollow” /> to make all the links nofollow on that page – referred as nofollow tag
Similarly you can do with noindex. rel=noindex applicable for destination page and meta tag applicable for host page
Next is robots.txt file that is well known to webmasters for preventing search engines to crawl and index entire site or specific parts of it.
Nofollow
Here’s what Google says,
- The Googlebot does not follow any link with nofollow attribute nor the links on a page with nofollow meta tag
- If a page has got only nofollow backlinks, it is not likely to be indexed or shown.
- No pagerank or link juice will be passed from a nofollow backlink.
- There is no attribution assigned to a webpage, through nofollow backlinks on authority domains.
Here’s what Yahoo says,
- Despite of nofollow attribute or meta tag, the links are crawled and outbound links are discoverd.
- If a page has got only nofollow backlinks it is still indexed and shown whenever crawler finds it.
- No anchor text value or link juice is given to the landing page.
- Similarly, no trust is passed through nofollow backlinks from authority sites.
Conclusion: Its not completely true that search engines don’t see the target of nofollow links. Although nofollow links may not transmit any value in terms of pagerank, anchor text or trust factor; they still count as a backlink and their collective influence might be considerable in search engine rankings. However, you can use it effectively on internal links of your site to sculpt PageRank.
Noindex
Here’s what Google says,
- Page with noindex is followed by Google and links on it are crawled (unless it is used in conjunction with nofollow)
- Google does not index or show any page with noindex meta tag nor a page that has only backlink with rel=noindex.
- Page with noindex tag still flows a pagerank and link juice towards its outbound links, same with noindex attribute on links
Here’s what Yahoo says,
- Yahoo crawlers follow a Page with noindex and links on it are crawled (regardless of being used in conjunction with nofollow)
- Yahoo can index and show a ‘noindexed’ page as far as its heavily linked to with related anchor text. (It appears as only url and no snippet in SERPs because anchor text relevancy is considered here, not any terms on that page)
- Link juice and trust is passed to destinations found on noindex pages.
Conclusion: Using noindex tag for preventing a page to be displayed in search engine return pages may be a good solution in case of Google but not in yahoo. If that page is quite relevant to search querry on the measure of backlinking anchor text, Yahoo! is anyway going to show its url in search results (as a reference link from other indexed web pages) though with no title and description i.e. snippet.
Robots.txt
Google and Yahoo both say,
- Pages restricted in robots.txt are not at all crawled, nor the links on it, are followed.
- Here in this case the page is not indexed but Google and Yahoo can still show its URL in their SERPs for the same reason of anchor text relevancy. (Again no search engine snipptes as no data is retrieved from site itself, though DMOZ/ODP data can be used by Google and Yahoo directory data by Yahoo)
- A page can still consume pagerank and/or link juice to outbound links.
Conclusion: Google bot and Yahoo’s spider both seem to be obeying robots.txt, since they are not crawling or indexing the contents on the page. But they sometimes show URLs in their SERPs that are not indexed if found highly relevant for the search terms.
Unless using some creative solution you can not stop Yahoo from doing it, however you can remove such a result from Google (or generally any page that is already indexed and showing up in SERPs for whatever reason) by doing as follows,
- Add Noindex meta tag on the page
- Unblock the page in robots.txt file and let Google crawl it
- Submit a webpage removal request via Google WMT (Webmaster Tools)
To revise the different approaches of Google and Yahoo, here is the simple chart I made:
*If they think that the page would be a good resource for a web-searcher based on the anchor text of backlinks.
**If there are no other pages linking to it without nofollow
I hope you will use this informtion to manage different issues those often arise for sitemap pages, affiliate links, 404 error pages, insignificant pages, duplicate contents, pages with sesitive information and so on
P.S: No information in the world is valid forever. Make sure that you check recent posts regarding the topic:)
July 2nd, 2010 at 11:31 am
[...] to external links like comments, trackbacks etc Heres a nice article that describes about how nofollow , noindex etc tags work Link Directory Blog Directory Reply With Quote [...]
August 21st, 2010 at 10:20 pm
i want to know how to send a command to search engines so they dont show results on my name
please reply
August 22nd, 2010 at 3:54 am
Hi Russell,
Please let us know more about it. If you are a webmaster of the pages showing up in search engines after searching your name, then its easily possible to vanish those results using noindex and robots.txt but if those are third party pages, then its to be rectified through online reputation management.