HowTo configure robots.txt properly

[michael_s]

robots.txt : what is it?
robots.txt is an instruction file for search engine robots, like googlebot or slurp! This file sets up rules for the search engines as to what they can and cannot index on your site. This is very helpful if you have certain areas of your site that you do not want showing up in search engines, like login pages, ssl protected pages, or anything else you want kept out of the indexes.

How does it work?
The robots.txt file works by specifying a user-agent and then a command for that user-agent.

These are the different variables that can be included in a robots.txt file:

• User-agent:
• Disallow:
• Crawl-Delay:

To indicate comments that are to be ignored, precede the comment line with a # symbol.

User-agent: - This is just the name that the search engine robot uses to identify itself to your server when accessing pages.

Disallow: - This is where you tell the search engine robot what directories or files it is not allowed to spider or index.

Crawl-Delay: - This tells search engine robots how many seconds to wait before moving to the next page. This can help if a robot is causing problems with your server load as they sometimes can.

robots.txt examples
Now that you understand what the terms are, lets see some examples of robots.txt files.

1. This first example is the most basic and it tells all User-agents that they can spider your entire site:
   

   User-agent: *
   Disallow:

2. This second example blocks all search engine robots from indexing your cgi-bin directory - a common directory that you would not want indexed.
   
   

   User-agent: *
   Disallow: /cgi-bin/

3. This example limits Yahoo Slurp! from the /admin/ directory, but lets others index it:
   
   

   User-agent: yahoo! Slurp
   Disallow: /admin/

These are simple examples, but you can see how powerful it can be. You can block individual search engine spiders from specific content on your site. As a final, more comprehensive example, I am going to provide an application specific robots.txt file. This robots.txt is written specifically for use with vBulletin forums and keeps the search engine robots out of areas that they do not belong:

User-agent: *
#Crawl-Delay: 10
Disallow: /forums/ajax.php
Disallow: /forums/ajax_cron.php
Disallow: /forums/attachment.php
Disallow: /forums/checkspelling.php
Disallow: /forums/cron.php
Disallow: /forums/editpost.php
Disallow: /forums/external.php
Disallow: /forums/faq.php
Disallow: /forums/global.php
Disallow: /forums/image.php
Disallow: /forums/joinrequest.php
Disallow: /forums/login.php
Disallow: /forums/misc.php
Disallow: /forums/moderator.php
Disallow: /forums/newattatchment.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/payment_gateway.php
Disallow: /forums/payments.php
Disallow: /forums/pdfthread.php
Disallow: /forums/poll.php
Disallow: /forums/postings.php
Disallow: /forums/printthread.php
Disallow: /forums/private.php
Disallow: /forums/profile.php
Disallow: /forums/register.php
Disallow: /forums/report.php
Disallow: /forums/reputation.php
Disallow: /forums/search.php
Disallow: /forums/sendmessage.php
Disallow: /forums/subcription.php
Disallow: /forums/subcriptions.php
Disallow: /forums/threadrate.php
Disallow: /forums/usercp.php
Disallow: /forums/usernote.php

Disallow: /forums/admincp/
Disallow: /forums/images/
Disallow: /forums/modcp/
Disallow: /forums/articlebot/
Disallow: /forums/clientscript/
Disallow: /forums/customavatars/
Disallow: /forums/customprofilepics/
Disallow: /forums/files/
Disallow: /forums/install/
Disallow: /forums/cpstyles/
Disallow: /forums/images/
Disallow: /forums/includes/
Disallow: /forums/subscriptions/
Disallow: /forums/attachments/
Disallow: /forums/frm_attach/
Disallow: /forums/vbweather/

[michael_s]

Another case that I have been asked about is how to deny all robots from indexing everything on your site. Occasionally there is a need for such a directive. It is very simple:

User-agent: *
Disallow: /

That tells all robots that everything on your site should not be indexed.