Cyberbox

  • Home
  • About
  • Services
    • Web Design
    • Search Engine Optimization
    • Social Media Marketing
    • E-commerce / Stores
    • Internet Advertising Campaigns
    • E-mail Marketing
  • Portfolio
  • News
  • Contact

Happy 20th anniversary Robots.txt !

Robot Today is the 20th anniversary of the robots.txt an easy and convenience way  to  block search engines from crawling their pages.

The robots.txt was created by Martijn Koster in 1994. All major search engines back then, including WebCrawler, Lycos and AltaVista, quickly adopted it; and even 20 years later, all major search engines continue to support it and obey it.

It works likes this: a robot wants to vists a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds:

User-agent: *
Disallow: /
The “User-agent: *” means this section applies to all robots. The “Disallow: /” tells the robot that it should not visit any pages on the site.

There are two important considerations when using /robots.txt:

robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don’t want robots to use.

 

Filed Under: News, SEM Tagged With: Bing, Google, SEM, SEO, Yahoo

Recent Posts

  • Google PPC audience targeting
  • Technology Trends for 2016
  • iPhone 6, iPhone 5s top most popular cameras used on Flickr in 2015

Testimonials

Charlie Martinez

"Mr. Wolf (Cyberbox's Founder) is a true professional. He met all of the firm's deadlines and his work is excellent. We have received a lot of positive feedback from our clients. I would highly recommend him".

www.charliemartinezpa.com

Charlie Martinez PA Miami Lakes, Florida

Omar Arcia

"My experience with the staff at Cyberbox has been nothing short of phenomenal. Their creativity, responsiveness and knowledge of website design and search engine optimization has been second to none. The referrals we have received through the website designed by Cyberbox has tripled over the last year. I give Cyberbox my full and absolute recommendation to anyone looking for a website design, hosting and search optimization firm."

www.arcialawfirm.com

Arcia Law Firm Pembroke Pines, Florida

Edgar Gunness

Cyberbox is a great agency, they have successfully developed several SEO marketing plans for our company that have resulted in increased revenue.

www.peoplestrustinsurance.com

People's Trust Insurance Deerfield Beach, Florida

Mark Bockstein

Our site is great, these guys did it faster and with a very professional look. The attention was great.
www.bockstein.net

Mark Bockstein Plantation, Florida

Luisa Liccardo

It is hard to find companies that truly care about their clients' needs, but CyberBox is one of them.  William, its founder, is knowledgeable, professional, and a very hard worker. When working towards fulfilling our company's web site needs, he put forth both a patient and a didactic approach. Most importantly, he truly loves what he does, as just everyone in his team does. Thanks for your help! Luisa Liccardo-Ortiz, Tres Leches Factory and beyond. 

www.3lechesfactory.com

Tres Leches Factory and beyond Doral, Florida

Tag Cloud

Android Apple Bing Cyberbox E-commerce E-mail Marketing E-marketing Facebook Google PPC SEM SEO Shopping Carts Social Media Technology Twitter VR Web Design WordPress Yahoo

About

Founded in 1996 Cyberbox is a full-service Web interactive agency. We offer creative and functional website designs, web-enabled applications, shopping carts, e-commerce solutions, … more...

Subscribe to our Free Newsletter

Follow Us!

  • Email
  • Facebook
  • LinkedIn
  • Twitter

Copyright © 2021 Cyberbox.net