Search Engine Academy of South Carolina

Let’s do a quick review of a  robots.txt file. Basically it is a protocol for spiders to immediately find instructions on what they can and cannot have access to on your site. If you don’t want spiders indexing your images, just disallow them. If you’d prefer not to have certain areas of your site indexed and available for the searching public – go ahead and restrict access by using a robots.txt file.

There is nothing difficult about creating a basic robots.txt file. It can be created using notepad or whatever is your favorite text editor. Each entry has just two lines:

User-Agent: [Spider or Bot name]
Disallow: [Directory or File Name]

This line can be repeated for each directory or file you want to exclude, or for each spider or bot you want to exclude.

Googlebot, MSNbot and Slurp (Yahoo!) all look for the robots.txt file on your site so why not give them what they are looking for and index only those pages you want to share with the engines.