Many Web sites don’t offer robots.txt, Although this file is very important for
SEO if you want to have a good ranking on search engines,
Most SEO services providers offer robots.txt as one of their service,but after you read this article, you will know how to create the robots.txt yourself!