Stop Your Robots.txt From Creating A Black Hole For Link Juice
If you are using robots.txt to block access to a page, it probably is creating a very large black hole sucking up your site’s link juice. Especially if every page in your site is linking to the black hole from the footer or header sections.
Where did all my link juice go?
Google and other SE’s respect the robots.txt commands, and each follows them to a certain extent. The problem is that robots.txt does not stop the flow of link juice to the disallowed page.
This wasn’t a problem in the past because you could simply use nofollow and save the link juice for other pages. However, since Google has stopped allowing nofollow to be used in this fashion, the link juice itself is completely lost. Matt Cutts has actually recommended not using nofollow for link juice sculpting.
The solution is to use the meta noindex on the actual page you do not want indexed. Google can still visit the page and pass the link juice through whatever links are available. Usually just the menu links.
Using meta noindex
Don’t use “noindex,nofollow” together because that will defeat the purpose and Google won’t pass any link juice on to other pages.
noindex = “Don’t include this page in your index”
nofollow = “Don’t follow (or pass any authroity) to any links on this page”
This should go in the head section of whatever page you don’t want included in the SERPS:
Some other useful resources:
- Matt Cutts discusses the behavior of search engines around meta noindex
- Why pages disallowed with robots.txt still appear in Google
Related posts:
- There is now no reason to ever use nofollow! This is a lesson in how fast something can change...
- RSS Footer Plugin Is A Great Link Building Tool Caution: Your content is being stolen! Black hatters are scraping...
- Finding Authoritative Links One authoritative link can help your search engine rankings increase...
- Wikipedia Is Using Nofollow Attribute Perhaps I am just late with this discovery, but it...
- Great Link Building Tool I have been using a new tool for commenting (link...
You should sign up for my RSS feed here
I don’t now much about robot.text , but if we use this tag
Search engine will stop crawler our site’s page?.
People actually use nofollow in robots? I’m not sure if it had any purpose or whatever but isn’t noindex good enough either way?
Thanks for the article. I’m using robots.txt in all of my projects but I didn’t know about the lost linkjuice I’ve got. Really helpful post.
.-= schmied73@design´s last blog ..Designer Möbel: Barhocker =-.
I didnt know not to use noindex and nofollow together.
.-= Carla@Muscle Max´s last blog ..Muscle Building Diets =-.
thanks this seo article.
good idea.
.-= finance business articles´s last blog ..Terminating an employee : Best ways to fire an employee =-.
Thanks for sharing. I don’t know much about robot.text.
Matt Cutts recently noted that one of the ways to control spam is to disallow pages using the robots.txt file. Too bad he did not further explain the impact this has on pagerank and your site as a whole.
You got me lost there mate, search engine will stop from crawling a site using a robot.text right? if so, then how can we benefit from this. Additional explanation will be appreciated. Thanks in advance
I am sure this post will be of major interest to all those wondering how Google’s recent changes affect what has come to be known as PageRank sculpting so I reviewed it at StumbleUpon and from there it will end up on Twitter, Facebook, FriendFeed and possibly Delicious.
.-= Gail @ Social Media Marketing´s last blog ..Are You Playing Hide and Seek With Your Market? =-.
Saw this one coming…had an SEO friend that was all into ‘nofollow’ for link sculpting but I couldn’t understand why Google would allow alteration of their algo.
I just follow everything now except external domains…I don’t even bother trying to sculpt anymore. Just write good content, have good link structure, anchor text and H1 titles. All my stuff pops up right away. It’s not that hard. Good times.
why not follow external domains? @search engine reports? If everyone did what you do, no one would get anything. Follow them all I say…
.-= vacation software´s last blog ..New Product =-.
Good, fresh content and keeping google out of your admin section is still the way to go!
I agree, there’s no reason to not follow external domains. It doesn’t harm your site when you link to other sites.
I agree with John. Good quality content is the best thing. And managing everything is the key. Thanks for the article. Cheers
Ok I tried this awhile ago and I noticed some changes in my rankings. Granted I also tried a lot of other stuff, but thanks anyway.
To be honest, I don’t worry to much about my robots.txt file, but maybe I should.
Thanks for keep us up to date on this. I see now how this could hurt your link juice. I’ll check out my site for this.