simonw 5 hours ago

Something I'm desperately keen to see is AI-assisted accessibility testing.

I'm not convinced at all by most of the heuristic-driven ARIA scanning tools. I don't want to know if my app appears to have the right ARIA attributes set - I want to know if my features work for screenreader users.

What I really want is for a Claude Code style agent to be able to drive my application in an automated fashion via a screenreader and record audio for me of successful or failed attempts to achieve goals.

Think Playwright browser tests but for popular screenreaders instead.

Every now and then I check to see if this is a solved problem yet.

I think we are close. https://www.guidepup.dev/ looks extremely promising - though I think it only supports VoiceOver on macOS or NVDA on Windows, which is a shame since asynchronous coding agent tools like Codex CLI and Claude Code for web only run Linux.

What I haven't seen yet is someone closing the loop on ensuring agentic tools like Claude Code can successfully drive these mechanisms.

  • devinprater 4 hours ago

    There are thousands of blind people on the net. Can't you hire one of them to test for you? Please?

    • PaulHoule an hour ago

      This. I've been doing a lot of accessibility work and it seems like the one thing nobody ever does is talk to a screenreader user.

    • m12k 4 hours ago

      If you don't want this to break eventually, you need it tested every time your CI/CD test suite runs. Manual testing just doesn't cut it

      • tdeck an hour ago

        We have the exact same problem with visual interfaces, and the combination of manual testing for major changes + deterministic UI testing works pretty well.

        Actually it could be even easier to write tests for the screen reader workflow, since the interactions are all text I/O and pushing keys.

      • cenamus 3 hours ago

        AI in your CI pipeline won't help either then, if it randomly gives different answers

        • simonw 3 hours ago

          An AI-generated automated testing script in your pipeline will do great though.

          • debugnik 3 hours ago

            And then we're back at your own:

            > I'm not convinced at all by most of the heuristic-driven ARIA scanning tools.

            • simonw 2 hours ago

              That's entirely different.

              ARIA scanning tools are things that throw an error if they see an element that's missing an attribute, without even attempting to invoke a real screenreader.

              I'm arguing for automated testing scripts that use tools like Guidepup to launch a real screenreader and assert things like the new content that was added by fetch() being read out to the user after the form submission has completed.

              I want LLMs and coding agents to help me write those scripts, so I can run them in CI along with the rest of my automated tests.

        • zamadatix 3 hours ago

          So does hiring a person or tests which rely on entropy because exhaustive testing is infeasible. If you can wrangle the randomness (each has different ways of going about that) then you end up with very useful tests in all 3 scenarios, but only automated tests scale to running every commit. You probably still want the non-automated tests per release or something as well if you can, depending what you're doing, but you don't necessarily want only invariant tests in either case.

    • angiolillo 41 minutes ago

      > There are thousands of blind people on the net. Can't you hire one of them to test for you?

      Testing is a professional skill -- not all blind people are good at accessibility testing, just as not all sighted people are good at GUI testing.

      My team has carved out an accessibility budget so that every couple years we can hire an accessibility consultancy (which employs a couple blind testers) for a few tens of hours of work to review one of our application workflows. Based on the issues they identify we attempt to write tests to prevent those classes of issues across the whole application suite, but our budget means that less than one percent of our UI has ever been functionally tested for accessibility.

      It comes down to cost/benefit. Good testers are expensive, good accessibility testers doubly-so. And while I personally think there's a moral imperative and maybe a marketing angle, improving accessibility truthfully doesn't seem to meaningfully improve sales. But if the testing costs came down by a couple orders of magnitude it would be a complete game-changer.

    • simonw 4 hours ago

      Realistically I'm unlikely to do that for my dozens of non-income-generating personal projects.

      I still want them to be accessible!

      (The amount of accessibility testing I want to do would bankrupt me very quickly.)

    • vintermann 42 minutes ago

      Most of us can't, no. Sorry. There's just not enough money in the things we make.

    • Andrex 3 hours ago

      Why not both?

      • javchz an hour ago

        I think this is the way, automated testing for all patches, small changes, and manual testing for big releases.

  • wouldbecouldbe 4 hours ago

    Not a joke. If truly you want a properly functioning website for blind/bad sight/ Step 1 would probably be to put on a blindfold and go through your website with a screenreader (cmd + f5 on a mac).

    • tdeck an hour ago

      I always wonder why this isn't a bigger part of the discussion. None of us would develop a visual UI flow without trying it manually at least once, but for some reason this idea isn't extended to discussions about accessibility. The advice always fits into these three groups:

      1. Follow a checklist

      2. Buy my software

      3. Hire blind people to test your app

      I'm not saying that these are bad (although some overlay software is actually worse than nothing), but aren't people even a little bit curious to try the user experience you're shipping?

      There are popular, free screen readers out there and using one can teach you a lot.

    • simonw 4 hours ago

      Here are notes from last time I tried that: https://tools.simonwillison.net/aria-live-regions

      It's not something I'm comfortable enough with to do on a regular basis though.

      • wouldbecouldbe 4 hours ago

        yeah its a real challenge, but probably the only way to really understand it. It's a completely different way of using the web. GPT can really help understand it while doing it.

    • hombre_fatal 2 hours ago

      MacOS has a VoiceOver tutorial.

      It’s pretty eye-opening (heh) to do it and then try to use your websites.

      Before you even get to aria labels, you’ll find a lot of things to fix like:

      - Add or remove components from tabindex

      - Esc should close the modal or the slide-out sidebar

      - Closing the sidebar/modal should return focus to the button the user toggled to open it (this one is huge for UX)

      I recommend it. These things are useful for keyboard nav in general.

    • PaulHoule an hour ago

      I'd argue it's pretty hard on Windows.

      NVDA hasn't worked for me since Windows 11.

      Narrator + IE and Narrator + Chrome basically work but make ARIA look like vandalism. It will just be reading the text and blurt out "LANDMARK WILL ROBINSON!" in the middle of the text for no obvious reason and doesn't do it the same very time. It's basically possible to use either one of those but Narrator + Firefox is especially spastic and blurts out random landmarks and ARIA junk to the extent that it's really difficult to evaluate.

      I mean, that's part of the problem, there is a spec for how HTML is supposed to be rendered, but ARIA is not specific about how ARIA markup is rendered which might means tools could bend to meet users' needs but it also means there is no such thing as "I've convinced myself that my ARIA markup is proper and will work for everyone with mainstream tools"

    • ErroneousBosh 2 hours ago

      I would very much like to be better at building websites that handle assistive technologies like screen readers well. I don't have a lot of blind users to worry about because they don't let you be a firefighter if you're blind.

      But a lot of firefighters are people who simply did not do well in school, even the very senior ones, and that's because they are often very clever people who are of an age where things like dyslexia were just not diagnosed early or often.

      So now I deal with a lot of people who use assistive technologies to help with dyslexia and related comorbidities. I have dyscalculia that wasn't diagnosed until I was 19 and at uni, and even then it was diagnosed initially by my mate's Educational Psychology undergrad girlfriend in the pub one evening. That was in the early 90s and we're better at it now - but not by much.

  • jillesvangurp 2 hours ago

    A more viable path might actually be agentic testing via agents that simply use a browser or screen reader that can work off high level test scenarios.

    I've done some UI testing via the agent mode in chat gpt and I got some pretty decent feedback out of that. I've been trying to do more of that.

    Accessibility testing might require a bit more additional tooling than comes with chat gpt by default. But otherwise, this could work.

  • PebblesHD 5 hours ago

    Rather than improving testing for fallible accessibility assists, why not leverage AI to eliminate the need for them? An agent on your device can interpret the same page a sighted or otherwise unimpaired person would giving you as a disabled user the same experience they would have. Why would that not be preferable? It also puts you in control of how you want that agent to interpret pages.

    • simonw 4 hours ago

      I'm optimistic that modern AI will lead to future improvements in accessibility tech, but for the moment I want to meet existing screenreader users where they are and ensure the products I build are as widely accessible as possible.

    • K0nserv 3 hours ago

      It adds loads of latency for one. If you watch someone who is a competent screen reader user you'll notice they have the speech rate set very high, to you it'll be hard to understand anything. Adding an LLM in the middle of this will add, at least, hundreds of milliseconds of latency to interactions.

    • 8organicbits 3 hours ago

      The golden rule of LLMs is that they can make mistakes and you need to check their work. You're describing a situation where the intended user cannot check the LLM output for mistakes. That violates a safety constraint and is not a good use case for LLMs.

    • devinprater an hour ago

      I, myself, as a singular blind person, would absolutely love this. But we ain't there yet. On-device AI isn't finetuned for this, and neither Apple nor Google have shown indications of working on this in release software, so I'm sure we're a good 3 years away from the first version of this.

    • eru 4 hours ago

      What you are describing is something the end user can do.

      What simonw was describing is something the author can do, and end user can benefit whether they use AI or not.

  • api 2 hours ago

    What about just AI assisted accessibility? Like stop requiring apps to do anything at all. The AI visually parses the app UI for the user, explains it, and interacts.

    Accessible is an also-have at best for the vast majority of software. This would open a lot more software to blind users than is currently available.

    • simonw 2 hours ago

      That's expensive, slow (listen to a screenreader user some time to see how quickly they operate) and likely only works online.

      I'm also not going to shirk my responsibilities as a developer based on a hope that the assistive tech will improve.

      • api 2 hours ago

        It’s expensive for now, slow for now, online for now, but it’s pretty clear that this is the future. If I were blind I’d want it to go here since it would just unlock so much more. Very little software or sites have good accessibility. Open source and indie stuff often has none.

        A custom local model trained only for this task seems like a possibility, and could be way smaller than some general purpose model being instructed for this task. I’m thinking screen reader and UI assist only. Could probably be like a 7B quantized model. Maybe smaller.

  • throawayonthe 4 hours ago

    can we just hire disabled people as testers please

    • simonw 4 hours ago

      How about both?

jillesvangurp 2 hours ago

I've looked a few times at this space. Most of the documentation seems to devolve into a depth first deep dive into here's a bunch of tags and attributes without laying out a coherent goal of what you should be doing.

My observations:

- most web developers have never seen a screen reader version of their application; or any application.

- most teams don't have visually handicapped people that use a screen reader that could provide feedback

- so, no bugs ever get reported regarding accessibility

That is, unless developers go out of their way to use proper tools and do proper testing for this. And testing practices for Aria probably is at the same level as it is for other application features: sketchy to non existent at best.

Let's face it, mostly Aria is pure box ticking for developers. There has to be some (regulations, and PMs insisting because of that). But it doesn't have to be good since nobody really checks these things. Including the PM.

Without a feedback loop, it's not surprising that most web apps don't get this even close to right. IMHO, over time, agentic tools might actually be more helpful to blind people as they can summarize, describe, and abstract what's on the screen. Agentic testing via a screen reader might also become a thing. I've done some testing via the agent mode in chat gpt and it was shockingly good at figuring out our UI. Not a bad process for automating what used to be manual QA. I've been meaning to put more time in this.

I actually have as a very low priority target to start driving some of this in our own application. Mostly that's just a hunch it might come up because of some government customers. But this is Germany and they seem to have lots of blind spots on software quality. I don't actually expect any feedback whatsoever from actual customers or anyone on this. I just want to pre-empt that.

Bengalilol 4 hours ago

2 cents here. Let's make this article optimistic and forward looking.

That's somehow intriguing to write an article that says "no" without providing "yes" examples. I don't view this as very generous.

Looking for further updates.

gampleman 5 hours ago

Is this really true? Messaging like this will cause a lot of developers to just give up. Most places I've worked at did accessibility at best as a best effort sort of thing. After reading this, there will be no attempts made to improve the state of affairs.

Perhaps that will be an improvement? I don't know.

  • robin_reala 3 hours ago

    Let’s give a concrete and catastrophic example of something I’ve seen in the wild in a professional product. A developer there had obviously seen the application role[1] in the ARIA specs, thought “I’m building a web app”, and added it to their html element.

    What role="application" means to assistive tech is: “I’m building a really complex application, so I’m going to handle absolutely everything for you, I don’t want you to have any default behaviour.” This meant that the web app in question was 100% unusable for any people using assistive technology, as that was broadly as far as they’d got with accessibility support.

    [1] https://developer.mozilla.org/en-US/docs/Web/Accessibility/A...

    • tdeck an hour ago

      Stories like this make me wonder if we could build a Chrome extension with a collection of crowd-sourced site-specific accessibility tweaks. Things like removing that bad ARIA tag or bodging in proper labels or tabindexes. It wouldn't be perfect, but neither is AdBlock and it offers a lot of benefit.

  • austin-cheney 4 hours ago

    ARIA is often a compensating technology more than a primary solution. I try to not use ARIA in my own code aside from the role attribute. I instead rely on the clear navigation and order HTML content and events as my primary solutions.

  • askew 4 hours ago

    The statement is about encouraging folks to use the platform: `<button>Hello!</button>` over `<div role=button tabindex=0>Hello</div>`

  • nabla9 4 hours ago

    It is true. This is not messaging. They are not selling you ARIA.

    If you are developer, just write semantically clear HTML instead. Just doing something is worse than doing nothing in accessibility.

  • pickpuck 3 hours ago

    This adage has been "the first rule of ARIA" since the beginning.

    There are a few ARIA "widgets" that have no HTML equivalent, such as Tabs or a spreadsheet-like Grid. Those are heavily documented so you can basically copy and paste whenever you need them.

    Avoiding sprinkling ARIA on already-semantic HTML, because this can lead to confusing or inconsistent behaviors for the end user.

wccrawford an hour ago

No examples are better than no examples.

They really needed to show what the proper way to implement those scenarios was, as well as the proper way to use those aria properties.

As it stands, they look good, and someone that isn't paying attention is going to think they're correct and use them.

i-con 5 hours ago

No CSS is better than bad CSS

In my browser that "Page Contents" box is hovering above the end of the line, so I can't read the full text. Kind of ironic, that this is on w3.org

  • the_other 4 hours ago

    > No CSS is better than bad CSS

    That's only true if the markup and JS are also good. If, for sake of argument, the HTML had been badly authored such that the links in that menu were DIVs with click event handlers, rather than real links, then removing CSS would likely make the experience worse rather than better.

    I guess that a key point underpinning your comment is that progressive enhancement is still better than assuming all potential users are on the bleeding edge, despite the evergreen update pattern for the most popular 3 or 4 browsers.

  • account42 3 hours ago

    What browser is that?

    • i-con 3 hours ago

      Tested with WebKit and Gecko. Apparently the position gets fixed up at runtime if JavaScript is enabled. But why have dynamic elements with CSS if you need JavaScript to fix it?

      • account42 an hour ago

        Ok yeah that's really weird. It uses javascript to unconditionally add the has-sidebar class to <body>. Maybe a limitation of their templating system but that's not really an excuse.

emsixteen 4 hours ago

I think it would be very beneficial if the browser vendors, together with screenreader vendors and those who use them, were able to come together to actually unify the approach to how elements/these attributes/etc. are communicated to the end user.

  • gostsamo 3 hours ago

    this part is covered. the issue are the web devs who need or decide that they need to repurpose a set of elements into another set of elements and skip informing the non visual user about that.

londons_explore an hour ago

The future of accessibility is AI agents.

Ie. Don't make the web page accessible to some ancient screen reader software - instead make sure AI agents can interact with it so the real user can instruct the AI to perform the task on their behalf.

notpachet 3 hours ago

This is also true of alt text for images. I'm active on Bluesky, and so often I see an image with an "ALT" tag (meaning that the user has specified alt text for the image) only for it to be some meaningless offhand bullshit like "lmao i guess i showed them". Pisses me off.

  • diegolas an hour ago

    people've been confusing the title and alt tags since like IE4

tonyhart7 3 hours ago

this is why I don't bother with ARIA