Today is the 20th anniversary of the robots.txt an easy and convenience way to block search engines from crawling their pages.
Search Engine Optimization
Visibility is essential for achieving any kind of success on the Internet. There is no better form of visibility than showing up at the top of Search Results on Google, YouTube, Yahoo, Bing, and other search engines.