robots.txt

In plain language

A small text file on your website that tells search engines and other bots which pages they may visit.

The metaphor

Like an "Employees only" sign on the staff-room door: polite visitors respect it, but it is not a lock.

Going deeper

How it works

Every site can have one at yoursite.com/robots.txt.
It lists rules per bot: which folders to skip, and where the XML sitemap is.
Well-behaved bots follow it. Bad bots ignore it, so it is not security.
It can also allow or block AI crawlers, such as OpenAI's GPTBot.

What can go wrong (example scenario)

A web designer left a "block everything" rule in robots.txt from the test site. The new website went live and quietly fell out of Google over the next few weeks.

How WILDCARD handles it

Managed Hosting keeps the file correct when sites launch and change. Managed Search checks it as part of monthly SEO, so a stray rule never hides the site.

In practice