What is robots.txt, and what happens without one?
A plain-text file at the root of your site that tells automated visitors which parts they may read. It is the most powerful single file on most sites, because one careless line in it can remove an entire business from search results.
What it actually is
Think of it as the sign inside the front door rather than the lock on it. It is a set of requests, not a security measure: well-behaved crawlers read it and respect it, and badly-behaved ones ignore it entirely. That distinction matters, because it means robots.txt is the wrong tool for hiding anything genuinely private, and the right tool for steering attention.
Why it matters
This file decides whether the rest of your work is ever seen. A site can be beautifully built, fast and full of excellent content, and be completely absent from search because of a single line left over from a staging environment. It is also increasingly where you say what AI crawlers may do, which makes it the place your position on being read by answer engines is actually expressed.
What good looks like
A valid file that exists, loads as plain text, and permits the pages you want found while excluding only the parts that genuinely should not be indexed — internal search results, admin areas, checkout steps. It points to your sitemap so discovery has somewhere to start, and every rule in it is one somebody meant.
Being honest about it
A missing robots.txt is much less dangerous than a wrong one. If you have no file at all, crawlers assume they may read everything, which for most small sites is exactly what you want anyway. We flag its absence because ambiguity is worth removing and because the sitemap reference is genuinely useful, not because your site is in peril without it.
Where most sites go wrong
The single most expensive failure on this entire list is a blanket exclusion left behind when a site moves from staging to live. It is one short line, it looks deliberate, and it silently removes the whole site from search until somebody thinks to look. The more common but milder failure is a file that has accumulated rules over years, where nobody remembers what half of them were for and at least one is now blocking something that matters.