Automated vs Manual Accessibility Testing: What Each One Actually Catches

The short answer to automated vs manual accessibility testing is that you need both, because they answer different questions. An automated checker such as axe, WAVE or Lighthouse reads your page's code and flags anything that breaks a programmable rule: an image with no alt attribute, text below the required contrast ratio, a form field with no label. It does this in seconds, across hundreds of pages, the same way every time.

What a tool cannot do is decide whether something makes sense to a person. It can confirm an image has alt text, but not whether "graph" describes a chart of quarterly sales. It can see that a menu has buttons, but not whether a keyboard user gets stuck inside it. Those judgements are where manual testing comes in: a trained tester working through the page with a keyboard, a screen reader, zoom and their own reading of the WCAG success criteria.

So a clean automated report is a good sign, not a verdict. If you need to know whether your site actually meets WCAG 2.0 Level AA under Ontario's AODA, or WCAG 2.1 or 2.2 for a contract, manual review is the part that gets you there.

Can accessibility testing be automated?

Partly. The W3C's Web Accessibility Initiative puts it plainly in its guidance on selecting evaluation tools: tools "can not determine accessibility, they can only assist in doing so," and "human judgement is required." It also notes that tools can produce false or misleading results.

The reason is built into WCAG itself. Many success criteria are about meaning and experience: whether text alternatives serve the same purpose as the image, whether focus moves in an order that preserves meaning, whether an error message tells someone how to fix the problem. A rule engine can test for the presence of a thing. It usually cannot test whether that thing is right.

What automated accessibility tools reliably catch

Automated checks are strongest on issues that have a clear, testable signature in the code. These are worth catching by machine, because they are common and tedious to find by hand:

• Missing text alternatives: images, image buttons and image map areas with no alt attribute or accessible name (1.1.1 Non-text Content).
• Colour contrast of plain text: text against a solid background colour that falls below 4.5:1, or 3:1 for large text (1.4.3 Contrast (Minimum)).
• Unlabelled form fields: inputs, selects and textareas with no programmatic label (1.3.1 Info and Relationships, 4.1.2 Name, Role, Value).
• Empty links and buttons: controls with no text or accessible name, often icon buttons (2.4.4 Link Purpose, 4.1.2).
• Missing page language: no lang attribute on the <html> element (3.1.1 Language of Page).
• Missing page title (2.4.2 Page Titled).
• Invalid ARIA: roles that don't exist, required attributes left off, aria-hidden on focusable elements.
• Duplicate IDs that break label and ARIA references.

Good tools also separate confirmed failures from items that need a human to look. axe-core, the open-source rules engine that also powers Lighthouse's accessibility audit, reports "violations" separately from "incomplete" results it couldn't decide on. Treat that second list as a to-do, not as noise.

What automated accessibility tools miss

These are the issues that take judgement, interaction or context. They are also the ones most likely to stop someone from finishing a task.

Whether alt text is meaningful

A tool sees that the attribute exists. It cannot tell whether the words describe the image's purpose, whether a decorative image should have been silent, or whether a chart needs a longer description nearby. Our guide to writing accessible alt text covers the judgement calls.

<img src="q3-revenue.png" alt="chart">
<img src="q3-revenue.png" alt="Bar chart: Q3 revenue up 12% over Q2, led by online sales">

Focus order and visible focus

Whether keyboard focus moves through the page in a sensible order (2.4.3 Focus Order) can only be checked by tabbing through it. Tools can spot a positive tabindex, a common cause, but not a modal that opens without moving focus into it, or a "skip to content" link that lands nowhere. Whether the focus indicator is actually visible (2.4.7 Focus Visible) is also mostly a human check, because custom focus styles vary so much.

Logical reading order

CSS can place content anywhere on screen regardless of where it sits in the source. A screen reader follows the source. If the visual order and the code order tell different stories, that's a 1.3.2 Meaningful Sequence issue, and finding it means comparing the two, usually with a screen reader running.

Keyboard traps in custom widgets

Date pickers, carousels, chat widgets and embedded video players are frequent sources of keyboard traps (2.1.2 No Keyboard Trap), where focus goes in and can't get out. A scanner doesn't press keys, so it can't find them. The same goes for custom dropdowns that only open on mouse click (2.1.1 Keyboard). Our keyboard accessibility guide walks through how to test these.

Whether error messages are useful

Most form errors only appear after someone submits the form, which an automated page scan typically never does. Even when an error is visible, only a person can judge whether "Invalid input" is good enough (it isn't) or whether the message names the field and explains the fix (3.3.1 Error Identification, 3.3.3 Error Suggestion).

Headings, labels and link text that make sense

A tool can find an empty heading. It cannot tell whether your headings describe their sections, whether a label says what the field wants, or whether five links all reading "Learn more" can be told apart (2.4.6 Headings and Labels, 2.4.4 Link Purpose).

Media, timing and screen reader behaviour

Caption accuracy (1.2.2), whether status messages such as "Item added to cart" are announced (4.1.3, added in WCAG 2.1), and how a custom component actually sounds in a screen reader all need someone to watch, listen and interact.

How much do automated tools find? The published figures

You will see percentages quoted everywhere, often without a source. Two studies are worth knowing, along with their caveats.

UK Government Digital Service (2017). The GDS accessibility team built a deliberately broken page containing 143 failures in 19 categories and ran 10 automated tools against it. In their write-up, the tools between them found 71% of the barriers, but the best single tool found 41% (counting its prompts for manual inspection), and 42 barriers, 29%, were missed by every tool. The caveats: the page was artificial, the study is from 2017, and tools have improved since.

Deque (2021). Deque, the company behind axe, analyzed anonymized data from over 2,000 audits covering more than 13,000 pages and nearly 300,000 issues, and reported that its automated testing covered 57% of issues on average. The caveats: it's a vendor measuring its own tools, and it counts by volume of issues, not by the share of WCAG success criteria a tool can test. Common issues like low contrast and missing labels occur many times per page, so a volume count rises fast.

Neither number means "a scan covers 57% of compliance." The practical reading is simpler: even on the most favourable measure, a large share of issues is left for people to find, and those are the judgement calls listed above.

Automated vs manual accessibility testing, issue by issue

If you're weighing axe vs manual testing, this table maps common issue types to what an automated checker can do and what still needs a person. WCAG numbers refer to WCAG 2.0 unless noted.

How to combine automated and manual testing

The efficient approach is to let the machine clear the obvious issues first, so the people doing manual review spend their time on judgement instead of counting missing labels.

1. Scan everything automatically. Run a tool across every template and key page. If you have a development team, add axe-core or a similar checker to your build so new regressions get caught before release.
2. Fix the machine-detectable issues. Contrast, labels, missing alt attributes and invalid ARIA are usually quick to fix, often once per template.
3. Choose a representative sample for manual review. Cover each page template plus the journeys that matter most: contact and quote forms, booking, checkout, login, search.
4. Do a keyboard-only pass. Unplug the mouse. Tab through each page and complete each journey. Note traps, invisible focus and anything you can't reach.
5. Do a screen reader pass. Use NVDA on Windows or VoiceOver on a Mac or iPhone. Our screen reader testing guide has the key commands and a page-by-page script.
6. Check zoom and reflow. Zoom to 200% and make sure nothing is cut off or overlapping.
7. Review content. Read the alt text, link text, headings and error messages as if you couldn't see the layout.
8. Re-scan after fixes. Automated tests are your regression net. Run them on every release; repeat manual review when templates or key journeys change.

For Ontario organizations, keep the legal baseline in mind. The AODA's Integrated Accessibility Standards require WCAG 2.0 Level AA for organizations they cover, and a scan report on its own isn't evidence that a site meets it. Our page on AODA requirements and deadlines covers who is covered and when.

Where automated tools mislead

Two patterns cause most of the confusion we see:

• A perfect score read as a pass. A Lighthouse accessibility score of 100 means none of the checks it ran failed. It runs a subset of axe-core rules, so it says nothing about keyboard traps, reading order or whether your content makes sense.
• Fixing for the tool, not the person. Adding alt="image" to silence a warning, or an aria-label that doesn't match the visible text, makes the report cleaner and the page no better. Sometimes it makes it worse.

The same logic applies to products that promise to fix accessibility automatically at runtime. We cover those separately in our article on accessibility overlays and compliance.

Frequently asked questions