In a previous post, we looked at how much automated accessibility testing can detect when compared against real, manually performed audits.
We found that strong automation can identify a meaningful share of the issues that auditors ultimately report (our own tools detected, on average, 63% of those issues).
That matters because every issue found late, in an audit checking live sites, comes with a cost.
At Evinced, we have a term for accessibility bugs that could have been detected early – prior to code being shipped – but weren’t. We call those bugs False Negatives.
In this blog post we’ll calculate just how much those accessibility bugs are costing your company.
Calculating workload costs
First, it’s important to understand that the cost of fixing a set of accessibility bugs depends entirely on when they are discovered in the development pipeline.
The advantage of automated tooling is that it can be run at a developer’s (or a designer’s) desk without slowing down the development process or requiring skills that that developer or designer might not have.
We estimate that an accessibility bug caught at the desk of the person doing the work, in their current context, would take about 1.25 hours to fix (a number which will go down as more AI coding solutions mature). At loaded developer rates (say, $200/hour), that’s a resource cost of $250.
Putting it all together
False negatives may well be the most important, and least discussed, part of an accessibility program, because the economics are stunning. And the math is straightforward.
The resource cost of a false negative is the cost to fix it later, minus the cost to fix it earlier.
Here’s that math:
Looked at this way, an accessibility program with limited company resources would absolutely look to minimize false negatives without slowing development to a halt.
Calculating program-wide costs
So far, we’ve looked at the costs of one false negative. But in reality, a program will encounter hundreds or thousands.
Consider three detection strategies we covered in our previous blog post:
- Manual only. One that relies solely on a post-production audit. At $14,725 per issue fixed, this program, over 100 false negatives will cost your company $1,472,500 in resources.
- Axe-core. This strategy uses an automated tool running axe-core earlier in the pipeline, followed by a production audit. As axe-core could detect 23% of issues earlier in the pipeline, this strategy would save some money over the manual-only strategy, though not much.
- Evinced. This strategy runs Evinced tools earlier in the pipeline (at a developer’s desk), where we would expect the developer to detect and fix 63% of issues.
It’s understood, in the axe-core and Evinced cases, that a manual production audit would be run later to catch any undetected bugs.
But for every 100 issues detected in that audit and fixed, the strategy to deploy Evinced tools at the developer’s desk saves almost a million dollars vs. a manual-only strategy, and it saves $600K versus a strategy that centers on axe-core and deploys at the exact same point in the cycle.
If that’s not clear already, remember that a modern enterprise could quite possibly have thousands of issues in even a single release of a website. So the ultimate numbers here are even larger.
False negatives are a problem with real consequences, but they can be drastically reduced with the right tooling.
As we write this, federal tax returns are coming due in the United States. And we have come to think of false negatives as a tax as well. But unlike death and federal taxes, false negatives are one thing you can avoid.