What Is A Robots.txt File?
Written by Garry Conn on January 10th, 2008 | 9 Comments
Ok, so you want to know what a robots.txt file is and how to use it? I am sure by now many have heard about robots.txt files, but what in the world is it and most importantly, what does it do and how will it benefit you? All the answers to these questions and more are found in this article.
The Simple Explanation
A robots.txt is a very small file that resides on your server that gives specific instructions to webcrawlers such as the famous Googlebot on which directories and files are allowed to be crawled and indexed. It is important to ... Continue reading What Is A Robots.txt File?
