Dec 13 2007

URLs Restricted by robots.txt

solution | SEO | 0 Comments

I get questions on why Google Webmaster Tool shows numerous links as restricted. The reason for this may due to information in your robots.txt file and/or information in your sitemap. If you have both, it’s a good idea to check the restricted links against robots.txt and sitemap file. Overall the best way to troubleshoot this problem is to follow Google’s guideline:

Debugging blocked URLs

From my experience many of these restricted links are valid; good examples are buy now and product review links. It’s obvious that a search engine robot will neither be able to make a purchase or write a product review. For the case of buy now; in product_listing.php, you can wrap the code in an if statement like so:

case 'PRODUCT_LIST_BUY_NOW':
$lc_align = 'center';
if($session_started) $lc_text = ‘…’;
else $lc_text = ”;
break;