{"id":1910,"date":"2026-09-09T06:17:47","date_gmt":"2026-09-09T13:17:47","guid":{"rendered":"https:\/\/evinced.com\/blog\/?p=1910"},"modified":"2026-09-09T10:05:41","modified_gmt":"2026-09-09T17:05:41","slug":"the-promise-and-peril-of-prompting","status":"publish","type":"post","link":"https:\/\/www.evinced.com\/blog\/the-promise-and-peril-of-prompting\/","title":{"rendered":"The promise and peril of prompting"},"content":{"rendered":"\n

Post 2 of 5 in our series on AI-assisted code and accessibility.<\/em><\/p>\n\n\n\n

In our first discussion of LLM-assisted coding and accessibility<\/a>, we tested the native ability of LLMs, out of the box, to write accessible code.<\/p>\n\n\n\n

But many teams don\u2019t use models that way. Instead, they provide a sometimes-lengthy set of instructions to the LLM along with the basic work request. These \u201cprompts\u201d are something of a cottage industry, too \u2013 engineers compare them, discuss them, and share them.<\/p>\n\n\n\n

Indeed, there is a prompt that nearly every accessibility-minded  team tries sooner or later:<\/p>\n\n\n\n

Follow WCAG 2.2 AA. Use semantic HTML. Make it work with a keyboard. <\/code><\/p>\n\n\n\n

As a public service, we decided to re-run our experiment from our first discussion but with this in mind. <\/p>\n\n\n\n

So we asked the same three models using the same three frameworks to each build a single-page app \u2013 so nine apps in total were built. Only this time, we gave the models the exact prompt above to guide the model while it did its work. Then we analyzed the built apps for accessibility defects, exactly as we did before.  <\/p>\n\n\n\n

Results<\/strong><\/h2>\n\n\n\n

In the table below, you can see that the prompt did not fix the problem. It made a trade. Critical defects fell by 57%. Serious defects, the grade just below critical, rose<\/em> by 49%. The total count of meaningful defects didn\u2019t change.<\/p>\n\n\n\n

<\/p>\n\n\n\n

\n\n\n\n

Exhibit 1. Summary of Results With and Without Accessibility Prompt<\/strong><\/p>\n\n\n\n

<\/td>BASELINE<\/strong>
Average Defects Created Per App (no accessibility prompt)<\/td>		EXPERIMENT<\/strong>
Accessibility Prompt Included <\/td>		RESULTS<\/strong>
% Change, Experiment vs. Baseline<\/td><\/tr>
Critical defects<\/td>21<\/td>9<\/td>-57%<\/td><\/tr>
Serious defects<\/td>24<\/td>36<\/td>+49%<\/td><\/tr>
Critical + serious<\/td>45<\/td>45<\/td>0%<\/strong><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Source: Evinced Web Flow Analyzer. N = 9 apps, built using combinations of 3 frontier models and 3 frameworks. Moderate and minor defects were near zero in both runs; across all grades combined, the net change was a 1.2% decline.<\/em><\/p>\n\n\n\n

Fixing the famous<\/strong><\/h2>\n\n\n\n

Critical bugs \u2013 the ones that stop assistive technology users in their tracks \u2013  did see measurable improvement and were cut by slightly more than half. So by no means perfect, but definitely better.<\/p>\n\n\n\n

What we can say is that when you prompt for accessibility, models are best at cleaning up the problems the internet talks about most. Missing image descriptions. Unlabeled buttons. Jumbled headings. Text you cannot read against its background. Those show up constantly in training data for the  models, and they happen to be the worst offenders, which is why the critical count improves while the total barely moves.<\/p>\n\n\n\n

The pattern to remember: prompting fixes the famous bugs. Only the famous bugs.<\/p>\n\n\n\n

The other side of the trade<\/strong><\/h2>\n\n\n\n

Now the price. Serious defects climbed 48.8% on average once we asked for accessibility, and in four of the nine apps the prompt actually increased the serious count. Take Gemini with plain JavaScript: the prompt cleared every one of its 36 critical bugs, but quadrupled its serious count from 10 to 41. Gemini with Svelte fared worst of all: 22 serious bugs became 111, five times as many, and its critical count rose too, 35 to 51.<\/p>\n\n\n\n

This is what overcompensation looks like. Chasing the famous fixes, models sprinkle in accessibility markup they do not understand and create brand-new bugs while they are at it. <\/p>\n\n\n\n

Thinking about reliability<\/strong><\/h2>\n\n\n\n

What it all comes down to is, can your team rely<\/em> on an LLM\u2019s accessibility results, even when prompted specially?<\/p>\n\n\n\n

Recall from Exhibit 1 that on balance, the number of critical + serious defects when using the accessibility prompt was unchanged vs. the baseline. <\/p>\n\n\n\n

But we also wanted to show the variability inside that average. To help discuss that, we\u2019ve reproduced the results below for each of our nine trials.  <\/p>\n\n\n\n

<\/p>\n\n\n\n

\n\n\n\n

Exhibit 2. Critical and Serious Accessibility Bugs Created With and Without an Accessibility Prompt<\/strong><\/p>\n\n\n\n

\"Bar<\/figure>\n\n\n\n

Critical and serious bugs per app, as evaluated by Evinced Web Flow Analyzer, with and without the accessibility prompt. <\/em><\/p>\n\n\n\n

In two of the built apps, the number of critical + serious accessibility bugs was worse than without the prompt at all. In another three, the improvement \u2013 i.e., the reduction in bugs \u2013 was less than 50%. And in the remaining three, the improvement was substantial. Beneath these averages, we also noticed (not shown) that serious issues increased in four out of the nine trials.  <\/p>\n\n\n\n

Other researchers have shown better results for prompts, though with similar variability. As an example, Aaron Gustafson reports that base models pass 8 to 25% of automated accessibility checks, and written instructions lift that only to 37 to 60%. And Michael Fairchild’s A11y LLM Eval<\/a> found the same guidance pushed some models past 90% and left others near zero.  <\/p>\n\n\n\n

It\u2019s precisely this variability that makes these tools unreliable: a technique that can make things 2X worse depending on which model you happen to use, and under what conditions it builds code, is not a safeguard you can build a compliance program on.  <\/p>\n\n\n\n

Even written rules get ignored<\/strong><\/h2>\n\n\n\n

The strongest evidence is not a statistic. There\u2019s a well-known bug report by Portland, Oregon-based developer Esti Shay where Claude Code states frankly that it treats accessibility as optional no matter what instructions you give it:<\/p>\n\n\n\n

\n
\n

“I framed a11y fixes as optional effort rather than as requirements. That’s wrong\u2026For this specific issue, it would be worth framing it as a bias in the model’s decision-making: Claude treats accessibility fixes as optional trade-offs rather than requirements, even when the project’s own rules say otherwise.”<\/p>Claude Code issue 56079<\/a> (May 2026, @EstiShay)<\/cite><\/blockquote><\/figure>\n<\/blockquote>\n<\/blockquote>\n\n\n\n

This matters because it closes off the comforting idea that smarter models will fix this on their own. The instructions were right there. The model read them. It shipped inaccessible code anyway.<\/p>\n\n\n\n

Check the work, not the worker<\/strong><\/h2>\n\n\n\n

If rules are just suggestions to models, where does that leave us? Nine rebuilds later, the lesson is plain: you cannot lecture a model into competence, but you can check what it produces. <\/p>\n\n\n\n

In other words, the answer is to guide the model with explicit, ideally deterministic, tests.\u00a0 That is exactly what agent skills do, and in our next post we\u2019ll examine those.\u00a0Stay tuned.<\/p>\n","protected":false},"excerpt":{"rendered":"

Post 2 of 5 in our series on AI-assisted code and accessibility. In our first discussion of LLM-assisted coding and accessibility, we tested the native ability of LLMs, out of the box, to write accessible code. But many teams don\u2019t use models that way. Instead, they provide a sometimes-lengthy set of instructions to the LLM […]<\/p>\n","protected":false},"author":15,"featured_media":1885,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[7],"tags":[],"class_list":["post-1910","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology"],"acf":{"authors_to_show":[1650]},"_links":{"self":[{"href":"https:\/\/www.evinced.com\/blog\/wp-json\/wp\/v2\/posts\/1910","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.evinced.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.evinced.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.evinced.com\/blog\/wp-json\/wp\/v2\/users\/15"}],"replies":[{"embeddable":true,"href":"https:\/\/www.evinced.com\/blog\/wp-json\/wp\/v2\/comments?post=1910"}],"version-history":[{"count":18,"href":"https:\/\/www.evinced.com\/blog\/wp-json\/wp\/v2\/posts\/1910\/revisions"}],"predecessor-version":[{"id":1938,"href":"https:\/\/www.evinced.com\/blog\/wp-json\/wp\/v2\/posts\/1910\/revisions\/1938"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.evinced.com\/blog\/wp-json\/wp\/v2\/media\/1885"}],"wp:attachment":[{"href":"https:\/\/www.evinced.com\/blog\/wp-json\/wp\/v2\/media?parent=1910"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.evinced.com\/blog\/wp-json\/wp\/v2\/categories?post=1910"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.evinced.com\/blog\/wp-json\/wp\/v2\/tags?post=1910"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}