Crawl rules: let the right readers in
Your robots.txt is the sign on the door. It says what may be read, and by whom. Some readers gather pages to train AI. Others search, or read a page live when a customer asks a question. They are different readers, and blocking one does not block the other.
Blocking the training readers is your call. Block the ones that search and answer, and those tools cannot read your facts when someone asks about you. OpenAI says a site that blocks its search reader is left out of ChatGPT's search answers.
This site lets AI companies read it and train on it, because we would rather AI tools learn about BeargrassAI from our own words than from someone else's.
You can block them by accident. A builder setting that hides your site from search adds a noindex tag to your home page, and our crawl rules check looks for it. Some networks that sit in front of websites now block some AI crawlers by default, so look at your host's settings too.
Sitemap and canonical: one map, one address
A sitemap is the table of contents crawlers use to find every page, your services and contact pages included. A canonical tag names your page's one true address, so search and AI know which copy counts. Most site builders make both for you. Check that they are switched on.
Structured data: get this right first
A person reads "Open Monday to Friday, 7 to 5" and knows those are your hours. A machine sees words and has to guess. Structured data removes the guess: a short block of code, unseen by visitors, that says this is a business, this is its name, and these are its hours.
Most sites write it as JSON-LD, in the words of schema.org, a shared vocabulary that Google, Microsoft and others agreed on. Give the closest type, such as Plumber, Electrician or Store, then your name, your address or the towns you serve, your phone, your hours, your website and one line about what you do.
It must say what your page says. Code that disagrees with the page does more harm than none. You can check it free with Google's Rich Results Test or the Schema Markup Validator.
Google says structured data is not required for its AI features, and is still worth using, because it makes a page eligible for rich results. Bing says it may help. When we read a site, it is the first place we look.
llms.txt: Google ignores it, others read it
llms.txt is a short plain-text guide to your site for AI: your name, one sentence about what you do and where, your phone and hours, and links to the pages that matter most. Jeremy Howard of Answer.AI proposed it in 2024.
Google Search ignores it. Google's own guide calls it fine to keep one for the other systems that read it, and says it will neither help nor harm you in Google. Chrome's Lighthouse checks for one in an experimental audit. No AI company has said its AI assistant relies on it.
We add it anyway. It costs nothing, and we expect AI tools to lean on files like it more over time. It is not a cure-all.
ai.json: our own file
ai.json is a name some sites use for a small file of facts in a machine format. There is no agreed standard. Ours is a short file of your site's facts, and Validated.Directory reads it when it reads your site. Like llms.txt, it costs nothing, and it is not a ranking trick.
Is good SEO enough?
Mostly, yes. Google says its AI features rest on the same systems as its search, and Bing says the same of Copilot. Plain words on your own pages, with your facts stated outright, do more than any one file.
