We all know the arguments about why things should be accessible, and we’ve seen and heard the United Nations statistics about how 16% of the world is “disabled” at any one time.
At Evinced, we’ve always thought about this problem a little differently. If you’re building a house, why would you size the doors so that your taller guests have to stoop on the way into your house for dinner? Are they “disabled?” And if you’re building a website, why ship it so that some people can’t use it, if there’s a straightforward alternative?
Or to put it starkly, why ship wrong if you know how to ship right?
But there is a new reason for accessibility, one that we’ve proven in our labs to have some really interesting implications.
It turns out that it matters not just to whom something is accessible, but to what.
It turns out that machines, of all things, need accessibility too. To an Evinced reader, this should not be totally surprising, since after all what is a screen reader but a machine?
A long time coming
When we first got exposed to the need to make software accessible, we noticed a glaring technical problem right away.
The problem was semantics.
In the technology world, semantics is the art of describing the purpose and meaning of things. From there, it’s a short hop to understanding how something should behave as well.
In the case of a web page, semantics would be used to describe that the purple rectangle at the top of the page, the one with the word “Buy” on it, is in fact a button, and that clicking it relates to starting a purchase experience.
Sighted users can usually infer what they need to know – that this purple thing is a button, for example – but everybody else is up a creek without a paddle. And when the web started out, semantic HTML was one of those goals that everybody said they aspired to and few achieved.
It hasn’t been just the web that suffered from a lack of semantics, either. It was also:
- RPA (Robotic Process Automation): These were “screen scraping” and other tasks recorded by humans and then replicated by bots.
- Test Automation. Here, QA teams wanted to record flows so they could conduct a functional test, then re-run the exact same functional test after changes were implemented, to make sure nothing had been broken.
- Voice Assistants. Siri, Google Assistant, and Alexa aimed to answer many questions by interacting with a web page in the background.
But especially in earlier days, these technologies suffered because they could not truly “read” or even interact with the web pages they were asked to target. In the case of RPA and Voice Assistants, it was simply hard to be accurate. In the case of Test Automation efforts, recorded flows were subject to breaking constantly as the underlying web page changed slightly.
Same problem, new workaround
Now comes agentic browsing. This is the idea that a browser should be able not just to show you stuff, but to do stuff. For example, an agentic browser might be tasked with getting you two tickets on the next flight to Paris.
But there’s the trouble. If the agent doesn’t understand how the Air France website works, if it doesn’t know what each of the pieces on the website does, how will it figure out where to specify what day you want to leave and which airport you want to depart from?
The workaround that agentic browsing has adopted is simple in theory. It just takes a screenshot, sends it off to a Large Language Model (“LLM”), and asks the LLM to figure all that out. And it turns out it would have to ask many, many questions, many times, with lots of detail provided each time, to get everything filled out correctly.
If that sounds inefficient and expensive, it is. And frankly it’s the main reason agentic browsing hasn’t, ahem, taken off.
Introducing a Semantic Agent
What’s needed is an agent that understands web pages. And it’s beginning to look like we’ve made one in our labs. We’re calling it (no surprise here) a Semantic Agent. To work, it relies on our understanding of accessibility, and it improves with the quality of the accessibility on the target website. And more than that, it’s about to change the game.
In the next few blog posts coming, we’ll show you just how.
Read “How our Semantic Agent works” next