If you use any of the multitude of SEO tools available on the web to analyze your website you will likely notice a field for robots.txt files. Does your site have such a file? Do you actually need one?
While your website will work fine without one all major search engines look for a robots.txt file. Think of it as an invitation to your website so this alone is a good reason to include one. Most people use robots.txt files to disallow spiders from crawling certain webpages. Keep in mind spiders may crawl them anyway but the ‘disallow’ command may help deter spiders from crawling certain pages.
What pages might you wish to disallow? If you are building your site some pages might be incomplete and not ready to crawl. Pages that are ‘Still under Construction’ could be included in your disallow command. The cgi-bin or a thank-you and error pages might be excluded too.
What if you don’t care and wish spiders to crawl all pages? This is the case for many websites and a simple robots.txt file can be created and uploaded to the root folder. If correctly uploaded it should appear by entering this url: www.yourwebsite.com/robots.txt
You can create the file in notepad as below:
(leave two lines blank below Disallow:)
Since you have not included any instructions on what to Disallow everything on your site will be crawled.
You can also use Google’s robots.txt analysis tool to check if you are using proper syntax and confirm the directories are allowing or disallowing robots to crawl your site.