All 68 checks

Your robots.txt tells crawlers what they may read. One stale line, often copied from an old template, can shut out the assistants people now use to find you.

Why it matters, from somebody other than us: OpenAI — Overview of OpenAI crawlers

Even when robots.txt says yes, a firewall or bot rule can turn assistants away before they reach a page. We ask as each crawler and record what comes back.

Why it matters, from somebody other than us: Cloudflare — AI Crawl Control

A CAPTCHA or bot check on an ordinary page stops an assistant at the door. Those belong on forms and logins, not on pages anyone can read.

Why it matters, from somebody other than us: Google Search Central — HTTP status codes and network errors

A machine-readable line in robots.txt saying whether AI may search, quote or train on your content. Having a stated position — any position — is the point.

Why it matters, from somebody other than us: Content Signals Policy

A few plain sentences in robots.txt saying what you permit. Journalists and lawyers read this file, and most sites say nothing in it.

Why it matters, from somebody other than us: RFC 9309 — Robots Exclusion Protocol

Machine-readable licensing terms, so an AI company can find out what you allow without emailing you first.

Why it matters, from somebody other than us: Really Simple Licensing (RSL)

Licensing that travels with the content — in your feeds and your structured data, not only in one file nobody fetches.

Why it matters, from somebody other than us: Really Simple Licensing (RSL)

Verified agents get through and unsigned scrapers claiming to be them do not, because the edge checks a signature instead of trusting a user-agent string.

Why it matters, from somebody other than us: IETF — Web Bot Auth architecture

PDFs, images and JSON cannot carry a meta robots tag, so the directive has to travel in the HTTP header instead.

Why it matters, from somebody other than us: Google Search Central — Robots meta tag and X-Robots-Tag

Old settings that told search engines not to show an excerpt also stop an assistant quoting your page in an answer.

Why it matters, from somebody other than us: Google Search Central — Robots meta tag and X-Robots-Tag

A missing page that answers 200 OK teaches a crawler that nothing is ever missing, and spends its time on pages that are not there.

Why it matters, from somebody other than us: Google Search Central — Soft 404 errors

Every extra redirect is another chance to be dropped. One hop, straight to the destination.

Why it matters, from somebody other than us: Google Search Central — Redirects and Google Search

A canonical link names the true address of a page. Without one, the same content reads as several competing pages.

Why it matters, from somebody other than us: Google Search Central — Canonicalization

Filters and calendars can generate millions of URLs. A crawler spends its budget in there instead of on the pages you care about.

Why it matters, from somebody other than us: Google Search Central — Faceted navigation

A lastmod date should mean the content changed, not that you deployed. Once it is noise, crawlers stop trusting it.

Why it matters, from somebody other than us: Sitemaps XML protocol

Separate sitemaps per content type turn “coverage is bad” into “coverage is bad for news,” which is a problem somebody can fix.

Why it matters, from somebody other than us: Google Search Central — Build and submit a sitemap

Page two and everything after it has to be reachable by something that does not click.

Why it matters, from somebody other than us: Google Search Central — Pagination and incremental page loading

Tell search engines the moment something changes, rather than waiting to be crawled again.

Why it matters, from somebody other than us: IndexNow documentation

If your content only appears after JavaScript runs, a crawler that does not run it sees an empty page. This is the heaviest check in the Index.

Why it matters, from somebody other than us: Google Search Central — JavaScript SEO basics

Tabs and accordions are fine, as long as the text is already in the HTML before anybody clicks.

Why it matters, from somebody other than us: Google Search Central — JavaScript SEO basics

Meaningful elements rather than an undifferentiated wall of containers. It is the difference between a document and a soup of boxes.

Why it matters, from somebody other than us: W3C WAI — Page structure tutorial

One main heading, then sub-headings in order. Assistants use headings to work out which part of a page answers a question.

Why it matters, from somebody other than us: W3C WAI — Headings

Tuition, hours, fees, comparisons — in a real table with header cells, not an image and not a grid of divs.

Why it matters, from somebody other than us: W3C WAI — Tables tutorial

Stable IDs on your headings let a model cite the exact passage rather than the whole page.

Why it matters, from somebody other than us: WHATWG HTML — The id attribute

Alt text is the only text an image has. Decorative images should say so with an empty alt rather than describing themselves.

Why it matters, from somebody other than us: W3C WAI — Alt decision tree

A fact locked inside a PDF, a scan or an infographic is a fact an assistant cannot quote.

Why it matters, from somebody other than us: W3C WAI — Images of text

A file at /llms.txt listing your most important pages. It is early and cheap, and it forces a useful conversation about what actually matters on your site.

Why it matters, from somebody other than us: The llms.txt proposal

The expanded variant, with the text inline. Worth it when your content is small enough to ship whole.

Why it matters, from somebody other than us: The llms.txt proposal

A clean Markdown version of each page: none of the navigation, none of the markup a model has to wade through to reach your words.

Why it matters, from somebody other than us: RFC 7763 — The text/markdown media type

Publishing the Markdown is not enough. The HTML page has to point at it, or nothing will ever find it.

Why it matters, from somebody other than us: RFC 8288 — Web Linking

Ask the same URL for Markdown and get Markdown back. One address, two representations, no second set of links to maintain.

Why it matters, from somebody other than us: RFC 9110 — HTTP Semantics, content negotiation

Generated representations should come from the CDN. Otherwise every request rebuilds them at origin, and you will turn them off when the bill arrives.

Why it matters, from somebody other than us: Cloudflare — Cache concepts

Schema generated from your content model survives the next content change. Hardcoded schema rots quietly and nobody notices for a year.

Why it matters, from somebody other than us: Google Search Central — Introduction to structured data

Structured data that links together — your organization, your site, this page — rather than a scattering of disconnected labels.

Why it matters, from somebody other than us: JSON-LD

Links from your markup to authoritative records like Wikidata or ROR. Without one, an assistant is guessing which organization of your name it has found.

Why it matters, from somebody other than us: Schema.org — sameAs

Every page below the homepage should declare where it sits, so a model knows a program page belongs to a department.

Why it matters, from somebody other than us: Google Search Central — Breadcrumb structured data

Use the types that match what you actually publish — courses, programs, clinicians, grants — rather than a generic WebPage on everything.

Why it matters, from somebody other than us: Google Search Central — Structured data gallery

Mark up real questions and answers. Do not invent an FAQ block that is not on the page: fabricated markup is a policy problem, not a shortcut.

Why it matters, from somebody other than us: Google Search Central — FAQ structured data

Steps marked as steps, so a model does not have to infer the sequence from prose and get it wrong.

Why it matters, from somebody other than us: Schema.org — HowTo

Flag your own best short answer — the lede, the summary — so an assistant reads that rather than guessing at a paragraph.

Why it matters, from somebody other than us: Schema.org — speakable

Say whether the page is free to read, and link the entities it is actually about. Both are things an assistant would otherwise have to assume.

Why it matters, from somebody other than us: Schema.org — isAccessibleForFree

Prices and offers in markup, and current. A stale price in structured data is worse than no price at all.

Why it matters, from somebody other than us: Google Search Central — Product structured data

Expose your site search as a pattern an agent can call, instead of a box only a human can type into.

Why it matters, from somebody other than us: Google Search Central — Sitelinks search box

Broken markup should fail your build rather than reach production and sit there until the next audit.

Why it matters, from somebody other than us: Schema Markup Validator

Authors who resolve to real people with real credentials, on pages of their own — not a name in a byline and nothing behind it.

Why it matters, from somebody other than us: Google Search Central — Creating helpful, reliable content

Sign substantive pages, and point the byline at the author's entity rather than repeating their name as text on every article.

Why it matters, from somebody other than us: Schema.org — author

Health, legal and financial pages should name their reviewer in the markup, not only in the design where a machine cannot see it.

Why it matters, from somebody other than us: Schema.org — reviewedBy

A modified date that changes on every deploy is not a freshness signal. It is noise, and it costs you the trust of the real one.

Why it matters, from somebody other than us: Google Search Central — Article structured data

Where you cite a primary source, make the citation machine-readable so the chain from claim to evidence survives.

Why it matters, from somebody other than us: Schema.org — citation

Provenance that survives into the delivered file, so an image can prove where it came from after somebody else has reposted it.

Why it matters, from somebody other than us: C2PA — Content Credentials

Document how your content is produced, reviewed, corrected and funded — then point at that page from your markup.

Why it matters, from somebody other than us: Schema.org — publishingPrinciples

Founding date, leadership, address, contact details, legal identifiers. Thin About pages are the most common weakness we see.

Why it matters, from somebody other than us: Schema.org — Organization

A natural-language endpoint over your own structured data, so an assistant can ask your site a question instead of scraping around it.

Why it matters, from somebody other than us: NLWeb

The same endpoint, exposed the way agents already know how to call things.

Why it matters, from somebody other than us: Model Context Protocol

Your course catalog, provider directory or grant database, callable as documented tools rather than scraped out of a search results page.

Why it matters, from somebody other than us: Model Context Protocol

If you have a public API, describe it in a spec an agent can read, and version it so today's answer is still true next quarter.

Why it matters, from somebody other than us: OpenAPI Specification

Put your agent capabilities where agents look for them, rather than in a URL somebody has to be told about.

Why it matters, from somebody other than us: IANA — Well-Known URIs registry

An agent filling in your form has to know what each field is for. Labels, names and autocomplete tokens are how it finds out.

Why it matters, from somebody other than us: W3C WAI — Labeling controls

Challenge the actions that need challenging. Searching and filtering are not those actions, and a captcha there ends the visit.

Why it matters, from somebody other than us: W3C — Inaccessibility of CAPTCHA

Register, donate, apply, find a clinician — these should complete through ordinary form submissions and real URLs, not only through clicks.

Why it matters, from somebody other than us: W3C WAI — Forms tutorial

Hashed class names change on every build. An agent that found your Apply button yesterday cannot find it today.

Why it matters, from somebody other than us: WHATWG HTML — data-* attributes

Traffic from ChatGPT, Perplexity, Claude and the rest lands in Direct by default, where it disappears. A channel group makes it countable.

Why it matters, from somebody other than us: Google Analytics — Default channel groups

Some assistants strip the referrer before the browser sees it. Server-side tagging catches what the client-side tag cannot.

Why it matters, from somebody other than us: Google — Server-side tagging

Crawler logs have to survive long enough to show a before and an after. Many hosts discard them in days.

Why it matters, from somebody other than us: Cloudflare — Logs

Watch crawler behavior continuously rather than auditing it once. A crawler that stops appearing usually means somebody reintroduced a block.

Why it matters, from somebody other than us: Google Search Console — Crawl stats report

Ask the same questions of the same models every month. It is the only way to see whether your share of the answers is moving.

Why it matters, from somebody other than us: Google Search Central — AI features and your website

Feed your page to a model, ask it your key questions, and score the answers. Then do it again after the fixes and compare.

Why it matters, from somebody other than us: Google Search Central — AI features and your website

Verify both. Bing matters more than its search share suggests, because its index feeds several AI products.

Why it matters, from somebody other than us: Bing Webmaster Tools