SEO & Marketing

What is robots.txt?

A text file in the root directory telling crawlers which areas of the website they may visit. Important: it is a request to well-behaved bots, not protection – anyone who really wants to lock content away needs directory protection or a login.

All glossary terms Question about your project? Drop us a line →