WordPress SEO: Search Engines


In a default WordPress installation you get different instances of your content repeated trough the site. For example take a post. It will appear in the homepage, in it’s single version, listed under a category, in the author pages, in search result, etc. Probably you are not comfortable having duplicated content indexed and letting Search Engines pick/guess which version of it is the one you consider the most relevant. Thats why it’s good to take some things into consideration.

What to index

There’s a lot of dicussion around the web of what you should and shouldn’t index. Unfortunately for us there isn’t one solution that will fit perfectly for all sites, to achieve the best results it will usually require planning ahead and even doing some experimenting and tweaking on the road. The type of content, the blog structure even the theme structure and its elements are things that affects these plans and should be taken into consideration prior making a decision. For example, not all themes will always show duplicate content. If your theme category pages show excerpts of the content that you previously took the time to write, then it certainly is good to index that page. Same thing could happen with tags, author and archives pages as well. If those provide different or extra content then why would you not index them.

How can you control this?

Create a robots.txt file

Im sure most of you know what a robots.txt file is for. For those who don’t know, it’s basically a file that goes in the root folder of your site with instructions of what specific files or directories you want to allow or disallow Search Engines to index. Here is a sample of a WordPress robots.txt file:

User-agent: *
Disallow: /tag
Disallow: /author

To prevent duplicated content, in the sample above decided to disallow the indexing of everything under /tag and /author. Remember not all themes/sites should have these same instructions, you have to choose carefully what you’ll want to remove. As a live example you can see my robots.txt file. If you feel uncomfortable editing files in your server there’s a nice plugin to create this file from within the Admin panel.

Add the correct robots meta tags to your theme pages

To have more specific control, or if you don’t trust enough your robots.txt file, you can always opt to use the robots meta tag. You can use this tag to tell Search Engines not to index the content of a page, and/or not scan it for links to follow. Joost de Valk has put together a great plugin to handle this easily from within WordPress control panel.

Hope you enjoyed. Thanks!

Reactions (15)

  1. Pingback WordPress SEO: Control Search Engines | DeveloperFox

  2. Pingback WordPress SEO: Control Search Engines | Money and SEO

  3. We might even add that there is also the Google Sitemap which is now regarded as a standard sitemap (it is now read by Yahoo! and MSN) which might be the opposite protocol to the robots.txt. The robots file is a must and the sitemap helps the robots to index the most important pages or those which are too deep in the site map to be instantly found and indexed. I use the excellent Google Sitemap Plugin by Arne Brachhold.

    You can also optimise your pages with the correct titles, descriptions and meta keywords using the “All in one seo plugin”.

    P.S: Matt, the new theme rooocks!

  4. @WebdesignBureau Thanks, those are very important aspects as well, great addition.
    Glad you like the theme! not sure how much ill hold on to it tho, was thinking of doing a redesign :)

  5. Pingback Wordpress: controllare l’attività dei motori di ricerca sul nostro blog : cssblog.it

  6. Pingback Link della settimana #19 - Amoga WebMania

  7. Check Pagerank

    Finally someone who can write a good blog ! . This is the kind of information that is useful to those want to increase their SERP’s. I loved your post and will be telling others about it. Subscribing to your RSS feed now. Thanks

  8. i am still looking for a good SEO plugin for WordPress. my blog is not ranking high enought for the keywords that i wanted to rank.

  9. Pingback WordPress SEO: Control Search Engines | DeveloperFox

  10. I am still a beginner in website SEO and i usually submit to website directories and article directories to boost the ranking of my site. .;-`

  11. Great post, we find that so many people rely on the search engines to do everything as far as ranking…i guess thats why they make so much money with ppc.