The Role of the Robots.txt file to improve site ranking!

Not many web master take the time to use a robots.txt file for your site. For search engines using the robots.txt file to see what directories to search through, the robots.txt file can be very helpful in keeping the spiders indexing your actual pages and not other information, like flipping through your stats!

The robots.txt file is useful to keep the spiders access to parts of files and folders in the directory of hosting that are completely unrelated to the actual content of the website. You can choose to have the spiders kept out of areas that contain programming that search engines can not parse properly, and keep them out of the web site stats.

Many search engines can not properly display dynamically generated content, mainly created by programming languages ​​like PHP or ASP. If you have an online store programmed in your hosting account, and is in a separate directory, you would be wise to block the spiders from this directory so that it is only relevant information.

The robots.txt file must be placed in the directory where the main files for your hosting. So it would be advisable to create a blank text file and save it as robots.txt and then upload to your web hosting in the same directory as the file is index.htm.

Here are examples of use of the robots.txt file:

To block a directory in a robots.txt file, such as a folder for your online store called / store / should do the following:
Disallow: / store /

Another example is to capture your stats directory:
Disallow: / stats /

You may also want to disallow individual files that you want searched by search engines. For example, you do not want search.php to be parsed by search engines. To do this you type the following own line:

Disallow: / search.php

Following the rules and create a robots.txt file, which will keep the search engines of unwanted files and directories, and let them pass through the important files to see what your website is all! ......