How Accurate Are Accessibility Scanners? We Measured the Real Gap

Last updated: November 14, 2026

Accessibility scanners are highly accurate on the narrow slice of WCAG they're built to test, and structurally unable to test most of it. In TestParty's remediation work across 35 million-plus issues and 100+ brands, automated detection surfaces roughly 60–70% of what a full evaluation eventually confirms; independent research from the UK government and from Deque lands in overlapping but not identical ranges, because each measured something different.

Key numbers: Across 35 million-plus accessibility issues remediated for 100+ brands, automated detection reliably surfaces roughly 60–70% of issue volume, with the rest requiring expert manual testing (TestParty remediation data, as of 2026). On a page built with 143 known barriers, individual automated tools caught between 17% and 41%, and pooling all ten tools tested still missed 29% (UK Government Digital Service, 2017). Deque's own study of 13,000-plus pages found automated testing identifies 57% of issues by volume, not by WCAG success criteria (Deque, 2021). In 2026, 95.9% of the top one million home pages carried a detectable WCAG failure — a figure WebAIM's own methodology says measures what a scanner can find, not full conformance (WebAIM Million).

How accurate are accessibility scanners, really?

A scanner is close to 100% accurate on the rules it's built to check, and blind to everything outside that rule set — which is why "how accurate" needs two answers, not one number.

On programmatic rules — does this image have an `alt` attribute, does this text meet a contrast ratio, does this input have a label — a modern engine like axe-core produces very few false negatives. If a condition is machine-decidable, the scanner finds it almost every time. The inaccuracy people actually experience isn't the scanner lying about what it checked; it's the gap between what the scanner checked and what WCAG requires, which is larger than most teams assume. In TestParty's remediation work, that gap runs roughly 30–40% of issue volume. The UK Government Digital Service found something structurally similar, numerically lower, testing ten tools against a page seeded with 143 deliberate barriers: no single tool caught more than 41%. Both are honest; each measured something different, which is the whole subject of this article.

What can a scanner test with certainty?

A scanner can decide, with near-zero ambiguity, anything that reduces to a yes/no check against the rendered DOM — no interpretation of meaning or intent required.

That list is short but high-volume: whether an `alt` attribute exists, whether text-to-background contrast clears a numeric ratio, whether a form input has an associated label, whether `<html>` carries a `lang` attribute, whether an ARIA reference points at a real element, and whether two elements share an `id` that should be unique. Each is deterministic — the same page produces the same result every time, run by any conformant engine. The open-source axe-core rule library publishes exactly this boundary: every rule is tagged as an automatic "violation" or a "needs review" item requiring a human to confirm it — the industry's own admission of where certainty ends.

What can't a scanner decide?

A scanner can confirm a feature exists; it cannot judge whether it works for the person relying on it — and that judgment gap is where most legal risk lives.

The clearest example: `alt="IMG_4471.jpg"` passes every rule in every scanner on the market, exactly as well as `alt="Navy wrap dress, front view"` does — the attribute exists, so the check is satisfied, whether or not it means anything to a screen reader user. The same blindness repeats across a short list of judgment classes: focus order (a scanner confirms focusable elements exist, not that Tab moves through them in a usable sequence, or that focus trapped in a cart drawer can get back out); status announcements a screen reader never hears when a cart total updates; a visible label that doesn't match the accessible name announced in context; and cognitive load — whether an error message tells someone how to fix the problem. None of these fail a rule. All of them can fail a shopper.

It helps to sort every WCAG criterion into four tiers by how certain automation can be:

What do the public studies show?

Three credible sources have measured scanner accuracy with different units of measurement — which is why the numbers don't match, and why none of them is wrong.

The UK Government Digital Service ran the most literal accuracy test in 2017: it built a page seeded with 143 known barriers across 19 categories, then ran ten free and freemium tools against it. The best single tool, Asqatasun, caught 41% (counting manual-inspection prompts); the weakest, Google's Accessibility Developer Tools, caught 17%. Pooling every finding across all ten still left 42 barriers — 29% — undetected by any of them (GDS, 2017). That's a barrier-count method: each planted failure counts once, regardless of how often it repeats on a real site.

Deque, the company behind axe-core, published a different kind of study in 2021: an analysis of more than 13,000 pages and roughly 300,000 real-world issues, reporting that automated testing identifies 57% of issues — counted by issue volume, not by the share of WCAG success criteria a tool can evaluate. A handful of high-frequency, machine-checkable failure types repeat thousands of times across a templated site, which inflates a volume-based score relative to a criterion-based one.

WebAIM's Million takes a third framing: it scans one million real home pages and reports what its scanner, WAVE, actually detects — 95.9% of pages carried at least one such failure in 2026. WebAIM's own methodology says this measures detectable failures only: "absence of detected errors does not indicate that a page is accessible or conformant" (WebAIM). It isn't an accuracy percentage — it's a prevalence count, silent by design on what the scan missed.

Three methods, three numbers: barrier-count says 17–41% per tool; issue-volume says 57%; prevalence-of-detectable-failure says nothing about accuracy by design. None of these figures is comparable without knowing which one you're reading.

What did we measure at TestParty?

Our number is operational, not a controlled study: across 35 million-plus issues remediated for 100+ brands, automated detection consistently surfaces roughly 60–70% of what full remediation eventually confirms, with manual review supplying the rest.

That split holds a pattern worth stating plainly: automation finds most of the volume, manual testing finds most of the severity. In our monthly expert audits, the findings automation never reports cluster in a short list — focus traps in cart drawers and promotional modals, controls with no accessible name announced to a screen reader, and checkout flows that break partway through rather than failing a single element. This is a qualitative, operational pattern from remediation work across a large customer base, disclosed as exactly that — not a controlled lab study, and not a claim that our 60–70% is the same measurement as anyone else's percentage.

Does scanner accuracy even matter for compliance?

Yes, but not the way most teams assume — WCAG conformance is assessed criterion by criterion, and a clean scan only confirms the criteria a scanner can evaluate, which is a minority of the 55 Level A and AA criteria in WCAG 2.2.

That's where scanner accuracy becomes a business risk, not a technical detail. A false sense of coverage is common among sites relying on scan results, or a widget, as their whole program: TestParty's analysis of Court Listener public records found more than 1,000 businesses with accessibility widgets installed were named in lawsuits in 2024. Plaintiffs test the way real users do, and complaints skew toward the judgment-required tier above. That doesn't make scanning useless — it makes it a floor, not a finish line. TestParty's own thresholds — Lighthouse 90+, WAVE ≤5, axe ≤3 — are a regression floor a build must clear, never proof a page conforms. Only a formal accessibility audit, evaluating all 55 applicable criteria, can make that claim.

What's the right way to use scanners?

Run automated scanning continuously as the volume layer, then add manual testing on a fixed cadence for the judgment layer scanners can't reach — the two solve different problems, and neither substitutes for the other.

In practice: daily or per-deploy automated scans catch regressions the moment they ship, paired with scheduled expert manual audits — monthly, in TestParty's own model — walking the keyboard and screen-reader paths a scan can't evaluate. Our manual vs. automated testing guide covers the coverage, cost, and cadence tradeoffs in full, and automated accessibility testing: 2026 implementation covers wiring the automated layer into CI/CD. Start with TestParty's free accessibility checker to see where your own site lands, then use our WCAG compliance testing guide to map the full three-method program.

Frequently Asked Questions

Is a 100/100 Lighthouse or zero-error axe score proof of WCAG compliance? No. Both tools evaluate only the criteria markup can express — a minority of the 55 Level A and AA criteria in WCAG 2.2. A perfect automated score means the automatable checks passed; it says nothing about focus order, caption accuracy, alt-text quality, or whether a keyboard-only shopper can finish checkout.

Why do different accessibility studies report such different accuracy numbers? Because they count different things. A barrier-count study like GDS's 2017 audit counts each failure once, finding tools at 17–41%. An issue-volume study like Deque's counts repeat occurrences at 57%. A prevalence study like WebAIM's Million reports what a scanner detects across real pages — not an accuracy rate at all.

Does TestParty's 60–70% figure match the public research? It's in the same range as some methods and higher than others, and it measures something slightly different from all of them: our own remediation work, confirming what automation caught against what a full manual pass found, across 35 million-plus issues and 100+ brands — an operational pattern, not a controlled study built for external comparison.

What kind of accessibility issue is most likely to pass every scanner and still fail a user? Alt text is the clearest case: `alt="IMG_4471.jpg"` and even a plausible-sounding but inaccurate description both satisfy every automated rule, since the check only confirms the attribute exists. Focus traps, silent status announcements, and mismatched accessible names behave the same way — technically present, functionally broken.

If scanners can't catch everything, why run them at all? Because they catch the highest-volume failures for effectively zero marginal cost, on every page, every day — which is exactly the layer that would otherwise bury a manual tester in repetitive findings. Running scanners first is what makes manual testing affordable to run at all.

Humans + AI = this article. TestParty uses a cyborg approach to content — combining human accessibility expertise with AI capabilities to produce accurate, comprehensive guides. This content is for educational purposes and reflects our analysis of publicly available information as of the publication date. TestParty competes in the digital accessibility market, and we encourage readers to evaluate all solutions independently based on their specific needs.