How the Census is run

A census site is scanned by the same scanner, against the same 12.1 tests, and scored by the same engine as a site whose owner asked us to look at it. There is no census mode. If the two could diverge, comparing a census figure to your own score would be a lie, and the comparison is most of what makes the census useful.

Every result records which version of the Index it was scored against and the date that version took effect. A standard that moves cannot quietly rewrite a finding published under an older one.

That sentence is recorded beside every single result, not asserted once here — because "we had permission" and "this is a public measurement" are different claims and only the second one is true.

The crawler is AIRBot. It identifies itself, it never impersonates another company’s user agent, and it obeys any robots.txt rule addressed to AIRBot — full stop, no exception for the census. A site that turns us away is recorded as having turned us away and stays in the population: excluding sites that block crawlers from a survey about whether sites block crawlers would point the selection bias straight at the thing being measured.

If you would rather not be in the next one, the bot page says exactly what to add. We would rather you didn’t, and we will still count you as having said no.

• One request every 5 seconds to any single organization. Never more, whatever else is running.
• 16 organizations at once, which are 16 different hosts and so costs none of them anything.
• Up to 32 URLs per site: the homepage, the pages in the organization’s own navigation, and a sample drawn from its sitemap.
• 429 and 503 are obeyed: we honor a Retry-After, slow down on every refusal, and come back to a site that said no after the rest of the run is done, at half the pace each time. One that still refuses is recorded as blocked rather than pressed.
• Every raw response is cached, so a re-run or a re-score costs the site nothing.

The page sample is per site. It is not what carries a finding — the n behind a cohort claim is the number of organizations scanned, and it is printed beside every share.

Each of these travels with every published aggregate, because each is a way a headline number can be true and misleading at once.

Coverage

The denominator is the number of sites actually scanned, never the cohort size. Rows with no published URL, refusals and failures are counted and named rather than dropped. A site that turned us away, with a 401, 403, 406, 429 or 451 or a bot-challenge page in place of its homepage, is listed as blocked and left out of every score and average: a score for a firewall’s error page describes the firewall.

Sample depth

The 32-URL cap is a ceiling, not a promise. Some sites yield one page. The distribution is published so the cap cannot be read as a claim about what was actually read.

Renormalization

Two sites can differ in which tests applied to them at all. Scores are renormalized over the applicable points to stay comparable, and that denominator’s distribution is published too — it is the number somebody would need to argue a comparison is unfair.

A census run declares no engagement profile, which is the correct state rather than a gap: the tests that depend on a business fact nobody told us come out N/A and leave the denominator. Nothing is scored zero for a question we never asked.

• Sitecore membership is detected, not declared. The scan recognizes the platform from what a site serves, so headless and CDN-fronted installations are undercounted.
• Where a company’s vertical comes from its SIC code, or an industry classification such as GICS, it is our classification, with a written override list. It is not something the company declared.
• Websites come from self-reported government and regulator directories and have not been verified live before we scan them. A listed address can be stale, and a row that fails to answer may be a wrong address rather than a broken site.
• EDGAR’s website field is often blank. Those companies’ sites are found by crawling for them and are marked as found that way.
• A cohort’s size is its roster’s size on the build date, and the roster is cited with it. A cohort of several thousand sites is a different kind of statement from one of a hundred.
• A site read more than once took more than one attempt: the first failed or was abandoned and a retry pass read it again. The site’s page says which attempt produced the score.
• The mean of a cohort that has grown is not comparable with any earlier figure for that cohort from when it held a hundred sites. It describes a different population.

Every cohort is drawn from a published source we are legally free to build on and republish a derivative of. The filtering on top is ours and is disclosed as ours on each cohort page.

And what we turned down

Better lists exist. We cannot republish a cohort derived from them, and a source we cannot cite is a source that makes the census uncheckable.

• US News Best Colleges — Copyrighted compilation; ranks prestige, which the Census does not measure.
• Forbes / NPT largest charities — Copyrighted compilations.
• American Hospital Association directory — Licensed; redistribution prohibited.
• BuiltWith / Wappalyzer technology datasets — Licensed; the terms do not permit republishing a derived list.

Published: everything measured, named, with the SHA-256 of each result recorded at the moment the scan finished. Nothing is anonymized. A finding nobody can check is an opinion, and the chain from a number on a page to a file to a run is the whole argument for believing any of this.

Never published: a contact route for anyone. We keep a way to reach the organizations we scan, so that a body surveyed without being asked can at least argue with the result — role addresses over people, always sourced, never guessed from a staff directory. That store sits outside the published dataset by construction, not by a filter somebody has to remember.

Why those two numbers differ

The cohorts add up to 27068 rows across 21826 distinct websites, because 554 sites belong to more than one of them. A children’s hospital is one of the largest nonprofits and one of the largest health systems. A university can run Drupal. Kaiser Permanente files as six separate nonprofits and answers on one address.

Those rows stay in both cohorts, and the site is read once. The cohort is the unit of analysis here — the question this census answers is how state portals compare with universities, or nonprofits with listed companies, and a member sitting in two populations does not damage either comparison. What it would damage is a claim about the whole, so we do not make one: this is a census of 27068 rows over 21826 websites, and never 27068 distinct organizations.