Read these announcements, these areas of often overlooked.
Improving on Robots Exclusion Protocol
One Standard Fits All: Robots Exclusion Protocol for Yahoo!, Google and Microsoft
Robots Exclusion Protocol: Joining Together to Provide Better Documentation
Recent Comments