Google is currently experimenting with new robots.txt commands. If your robots.txt file accidentally contains one of the new commands, it might be that your robots.txt file tells Google to go away and be bad for your search engine optimization.
Archive
Many Web sites don’t offer robots.txt, Although this file is very important for
SEO if you want to have a good ranking on search engines,
Most SEO services providers offer robots.txt as one of their service,but after you read this article, you will know how to create the robots.txt yourself!