Census Data
Every quarter we run the same scan, unchanged, across the same groups of organizations: state governments, universities, the largest nonprofits, health systems, the S&P 500, and a hundred sites each on Drupal and WordPress. Nobody asks us to. Nobody pays for it.
We want to know what the web actually looks like to a machine trying to read it, at a size where one badly built site can’t move the answer.
How common a failure is across a whole sector. How many state portals block AI crawlers outright. How many university sites publish machine-readable licensing. Whether a Drupal site is measurably better prepared than average.
Those are claims about a population, and 27,068 organizations can carry them.
That one organization is better than another. Two sites can differ in which tests apply to them at all, and we renormalize the scores so they stay comparable. A four-point gap between two named sites is noise.
That’s why we publish the spread around every number.
Who we scan, and where each list comes from
| Cohort | Drawn from | Sites | Median score |
|---|---|---|---|
| US State Government Sites state-sites
Ten sites for each of the 50 states: the portal and the revenue, motor vehicles, health, labor, education, courts, public safety, transportation and environment sites. |
Cybersecurity and Infrastructure Security Agency (CISA) | 333 of 500 |
median 28
|
| US Federal Government Sites federal-sites
500 federal executive-branch websites from the .gov registry, ranked by Tranco, with no agency above 10%. |
Cybersecurity and Infrastructure Security Agency (CISA); Tranco | 497 of 500 |
median 28
|
| US Members of Congress congress
Every sitting senator, representative, delegate and resident commissioner, with their official website. Left out because the seat was vacant at the build date: FL-20, TX-23. |
The @unitedstates project (unitedstates/congress-legislators) | 539 of 539 |
median 28
|
| US Tribal Governments tribal-governments
The federally recognized tribes the Bureau of Indian Affairs lists, with the website each publishes. |
Bureau of Indian Affairs, US Department of the Interior | 402 of 573 |
median 46
|
| US Colleges and Universities higher-ed
Every active, degree-granting US college and university that takes part in federal student aid, as IPEDS reports it. |
National Center for Education Statistics (NCES) | 3553 of 3954 |
median 58
|
| US Hospitals hospitals
The 1,000 US hospitals with the most revenue, as reported to Medicare. |
Centers for Medicare & Medicaid Services (CMS), with Wikidata and the IRS | 617 of 1000 |
median 54
|
| US Nonprofits nonprofits
The 1,000 largest US nonprofits by total revenue, excluding hospitals and universities. |
Internal Revenue Service (IRS), with Wikidata | 802 of 1000 |
median 54
|
| US Public Companies public-companies
Every SEC-registered operating company with a security listed on a US exchange, each with the vertical its SIC code gives. |
US Securities and Exchange Commission (SEC), with Wikidata and Wikipedia | 1745 of 5239 |
median 53
|
| US Banks and Credit Unions banks-credit-unions
Every active FDIC-insured bank and federally insured credit union, largest by total assets first. |
Federal Deposit Insurance Corporation (FDIC) and National Credit Union Administration (NCUA) | 7198 of 8519 |
median 54
|
| US Counties counties
Every US county and county equivalent. |
US Census Bureau, with CISA and Wikidata | 3016 of 3144 |
median 33
|
| US Cities (Top 1,000) cities-1000
The 1,000 most populous US cities. |
US Census Bureau, with CISA and Wikidata | 974 of 1000 |
median 28
|
| WordPress Sites wordpress-500
The 500 highest-ranked WordPress sites in the Tranco list. |
Tranco (Le Pochat et al., NDSS 2019) | 423 of 500 |
median 58
|
| Drupal Sites drupal-500
The 500 highest-ranked Drupal sites in the Tranco list. |
Tranco (Le Pochat et al., NDSS 2019) | 443 of 500 |
median 55
|
| Sitecore Sites sitecore-100
The 100 highest-ranked Sitecore sites in the Tranco list. Sitecore does not announce itself, so this is a floor. |
Tranco (Le Pochat et al., NDSS 2019) | 85 of 100 |
median 50
|
| All cohorts
20,627 of 27,068 sites scanned |
20,627 of 27,068 |
median 50
|
Government, by level
Every government site in the Census as one figure, with the same sites split by the level of government beneath it. A site with no website on record, one we could not reach and one that turned us away are counted in the first column and never in a median.
| Level | Sites | Scored | Median and spread |
|---|---|---|---|
| All government | 6,223 | 5,728 |
median 28
|
| Federal | 1,039 | 1,036 |
median 28
|
| State | 500 | 333 |
median 28
|
| County | 3,133 | 3,005 |
median 33
|
| City | 978 | 952 |
median 28
|
| Tribal | 573 | 402 |
median 46
|
A finding nobody can check is an opinion
We modeled this on the WebAIM Million, which has done the same thing for accessibility since 2019 and changed what our industry argues about by doing it. Our findings are aggregate. The dataset underneath them is public, named, with the hash of every result.
The Census methodology
How we chose the sites, how many pages of each we read, how hard we crawled, and why we scan organizations that never asked us to.
The Index itself
What a score means. Version 12.1. Every result records the version and date it was scored against, so a standard that moves can’t quietly rewrite an old finding.
All the tests
The same list we measure your own site against. One standard, no special cases for the Census.
AIRBot
The crawler that does the reading. What it fetches, how fast it goes, and how to turn it off.
If your organization is in a cohort and you’d rather we didn’t read it, AIRBot honors robots.txt. We’ll record that you opted out rather than quietly dropping you from the count.