When an autonomous Claude agent clicks around a live web app, the first thing that goes sideways is visibility. A CLI‑only loop shouts “next element” while the browser silently moves focus under the hood. By the time the agent reports a WCAG violation, you’ve already lost context, and the test run may have already tripped over a modal dialog you never saw.

The fix is to give the agent a visual cockpit: a desktop UI that streams Claude’s thoughts, highlights the element being inspected, and lets a human pause or intervene if the AI wanders off‑track. The result is a reproducible, auditable accessibility audit that runs in seconds instead of minutes.

⚡ TL;DR — Key takeaways
  • Claude’s tool‑use API can drive a real‑time screen‑reader simulator.
  • A PyQt6 GUI visualizes the agent’s state and lets you steer execution.
  • Computer Use API beats PyAutoGUI for reliable, pixel‑free interaction.
  • Synchronizing three state machines (AI, GUI, browser) prevents race conditions.
  • Built‑in rate‑limit handling keeps long audits from blowing up.

Before you start: Python 3.11+, Anthropic Python SDK v1.5+, Claude 3.5 Sonnet API key, Claude Desktop (or claude-computer-use sandbox), PyQt6 6.6, PyAutoGUI 0.9.54 (fallback only), keyboard 0.13.5, pynput 1.7.6, asyncio, a recent Chrome/Chromium (≥117) with remote‑debugging enabled, and optional pyttsx3 for TTS.

Build a Claude AI accessibility agent: automated screen‑reader & keyboard navigation (2026 guide)

A Claude AI accessibility agent automates screen reader simulation and keyboard navigation testing by combining Claude’s tool‑use API with a desktop GUI controller (like PyQt). It uses Python to bridge the AI’s commands to the OS, performing WCAG audits, reporting violations, and streamlining compliance workflows, all visualized in real‑time.

System Architecture & Workflow

The solution consists of three loosely‑coupled services that talk over asyncio queues:

  • Agent Orchestrator – runs Claude 3.5 Sonnet, receives tool calls, maintains the reasoning context.
  • GUI Controller – PyQt6 window that displays logs, a live DOM tree, and the currently focused element.
  • Browser Bridge – either Claude Computer Use API (preferred) or PyAutoGUI for low‑level actions; feeds DOM/focus data back to the orchestrator.
graph TD
    A[Claude Agent] -->|tool calls| B[Browser Bridge]
    B -->|DOM updates| C[GUI Controller]
    C -->|user commands| A
    B -->|focus events| A
    style A fill:#f9f,stroke:#333,stroke-width:2px
    style B fill:#bbf,stroke:#333,stroke-width:2px
    style C fill:#bfb,stroke:#333,stroke-width:2px

*Data flow*: Claude emits a simulate_screen_reader tool call → Browser Bridge extracts the element’s aria-label and visual text → GUI highlights the element and speaks it via TTS → Claude recovers the spoken text, decides the next keystroke, and repeats. All three components share a single asyncio.Queue for state updates, guaranteeing that the UI never freezes while Claude streams tokens.

Core Stack

LayerLibrary / APIReason
LLMAnthropic Python SDK v1.5+, Claude 3.5 SonnetUp‑to‑date model with built‑in tool‑use support
Desktop UIPyQt6 (Qt 6.6)Native look, fine‑grained event loop control
Browser controlClaude Computer Use API (semantic) + fallback PyAutoGUISemantic actions avoid brittle screen coordinates
Speechpyttsx3 (offline TTS)No network latency, works on Windows/macOS/Linux
Async orchestrationasyncio, asyncio.QueueNon‑blocking streaming of Claude’s response

My take: If you already ship a Qt‑based app, stick with PyQt6. If you need a tiny installer footprint, consider Tauri (Rust + WebView) but be prepared to embed a Python runtime via pyembed. For this guide we stay with PyQt6 because its event loop integrates cleanly with asyncio.

—

Step 1: Building the Agent Core with Claude’s Tool‑Use

1.1 Prompt engineering for WCAG & accessibility

A well‑crafted system prompt gives Claude the mental model it needs. Keep it under 500 tokens; include a short WCAG 2.2 cheat sheet, ARIA conventions, and a clear contract for tool calls.

# prompt.txt – keep this file alongside main.py
SYSTEM_PROMPT = """You are an autonomous accessibility auditor.
Your goal is to walk a web page using only keyboard navigation (Tab, Shift+Tab, Arrow keys) and simulate a screen reader.
When you encounter an element, call `simulate_screen_reader` with its DOM selector.
If you need to move focus, call `execute_keystroke` with the key string (e.g., "Tab").
After each action, analyze the focus order and report any WCAG 2.2 violation you detect.
Only use the tools you have been given. Do not hallucinate element IDs."""

1.2 Defining custom tools

Claude’s Computer Use API works like a remote‑procedure call. We expose three Python wrappers that turn tool invocations into concrete actions.

# tools.py – version: 1.0
import asyncio
import json
import logging
from typing import Any, Dict

log = logging.getLogger("tools")

async def simulate_screen_reader(selector: str, bridge) -> Dict[str, Any]:
    """
    Ask the bridge to fetch text and ARIA data for `selector`.
    Returns a dict that Claude can read.
    """
    try:
        data = await bridge.fetch_element_info(selector)
        # Speak via TTS (non‑blocking)
        bridge.speak(data["accessible_name"])
        return {"status": "ok", "data": data}
    except Exception as e:
        log.exception("simulate_screen_reader failed")
        return {"status": "error", "msg": str(e)}


async def execute_keystroke(key: str, bridge) -> Dict[str, Any]:
    """
    Perform a key press using Claude Computer Use API.
    """
    try:
        await bridge.send_keystroke(key)
        return {"status": "ok"}
    except Exception as e:
        log.exception("execute_keystroke failed")
        return {"status": "error", "msg": str(e)}


async def analyze_focus_order(state: Dict[str, Any]) -> Dict[str, Any]:
    """
    Compare the accumulated focus sequence with expected ARIA order.
    """
    try:
        violations = state["focus_sequence"].detect_violations()
        return {"status": "ok", "violations": violations}
    except Exception as e:
        log.exception("analyze_focus_order failed")
        return {"status": "error", "msg": str(e)}

1.3 Agent loop with session state

The orchestrator runs a perpetual loop: send a system + user prompt, await streaming response, intercept tool calls, execute them, feed results back, and repeat. State is held in a Session dataclass.

# agent.py – version: 1.2
import asyncio
import json
import logging
from dataclasses import dataclass, field
from typing import List, Dict, Any

from anthropic import AsyncAnthropic
from tools import simulate_screen_reader, execute_keystroke, analyze_focus_order

log = logging.getLogger("agent")
anthropic = AsyncAnthropic(api_key="YOUR_ANTHROPIC_API_KEY")

@dataclass
class Session:
    url: str
    focus_sequence: List[str] = field(default_factory=list)
    violations: List[Dict[str, Any]] = field(default_factory=list)
    last_selector: str = ""

async def run_session(session: Session, bridge, ui_queue: asyncio.Queue):
    system_prompt = open("prompt.txt").read()
    user_msg = f"Audit the page at {session.url} for keyboard navigation and screen‑reader compliance."

    async with anthropic.messages.streaming(
        model="claude-3-5-sonnet-202406",
        max_tokens=1024,
        temperature=0.0,
        system=system_prompt,
        messages=[{"role": "user", "content": user_msg}],
        tools=[
            {"name": "simulate_screen_reader", "description": "Get accessible text", "input_schema": {"type": "object", "properties": {"selector": {"type": "string"}}}},
            {"name": "execute_keystroke", "description": "Send a keyboard event", "input_schema": {"type": "object", "properties": {"key": {"type": "string"}}}},
            {"name": "analyze_focus_order", "description": "Check the collected focus list", "input_schema": {"type": "object", "properties": {}}},
        ],
    ) as stream:
        async for event in stream:
            if event.type == "content_block_start":
                # forward token stream to UI
                await ui_queue.put({"type": "log", "content": event.content})
            elif event.type == "tool_use":
                # Dispatch to correct Python function
                tool_name = event.name
                args = json.loads(event.input)
                if tool_name == "simulate_screen_reader":
                    result = await simulate_screen_reader(args["selector"], bridge)
                elif tool_name == "execute_keystroke":
                    result = await execute_keystroke(args["key"], bridge)
                elif tool_name == "analyze_focus_order":
                    result = await analyze_focus_order(session)
                else:
                    result = {"status": "error", "msg": f"unknown tool {tool_name}"}
                # Send result back to Claude
                await stream.respond_tool_output(event.id, result)
                # Update GUI
                await ui_queue.put({"type": "state", "session": session})
            elif event.type == "error":
                log.error(f"Claude error: {event.error}")
                await ui_queue.put({"type": "error", "msg": event.error})
                break

The bridge object (implemented later) knows how to talk to Chrome via the Computer Use API, fetch DOM info, and speak via TTS. The ui_queue feeds real‑time logs and state updates to the PyQt6 frontend.

—

Step 2: Creating the Interactive Controller GUI

2.1 Layout – Log, Command, Element Tree

We keep the UI minimal but functional:

  • Log Panel – scrolling QPlainTextEdit that receives token stream.
  • Command Input – a QLineEdit that allows the user to send ad‑hoc prompts or halt the run.
  • DOM Tree Viewer – QTreeWidget that reflects the live DOM hierarchy; the currently focused node gets a colored background.
  • Control Bar – start, pause, and load‑URL buttons.
# gui.py – version: 2.0
import sys, asyncio, json, logging
from PyQt6.QtWidgets import (
    QApplication, QMainWindow, QWidget, QVBoxLayout, QHBoxLayout,
    QPlainTextEdit, QLineEdit, QPushButton, QTreeWidget, QTreeWidgetItem,
)
from PyQt6.QtCore import Qt, QTimer, pyqtSignal, QObject

log = logging.getLogger("gui")

class UIUpdateSignal(QObject):
    update = pyqtSignal(dict)   # passes dict from async queue

class MainWindow(QMainWindow):
    def __init__(self, ui_queue: asyncio.Queue):
        super().__init__()
        self.setWindowTitle("Claude Accessibility Agent")
        self.resize(1200, 800)

        self.signal = UIUpdateSignal()
        self.signal.update.connect(self.handle_update)

        # Layout
        central = QWidget()
        self.setCentralWidget(central)
        vbox = QVBoxLayout(central)

        # Log panel
        self.log_edit = QPlainTextEdit()
        self.log_edit.setReadOnly(True)
        vbox.addWidget(self.log_edit, stretch=2)

        # Command line
        cmd_hbox = QHBoxLayout()
        self.cmd_input = QLineEdit()
        self.cmd_input.setPlaceholderText("Enter admin command (e.g., pause, stop)")
        self.cmd_input.returnPressed.connect(self.send_command)
        cmd_hbox.addWidget(self.cmd_input)
        self.load_btn = QPushButton("Load URL")
        self.load_btn.clicked.connect(self.load_url)
        cmd_hbox.addWidget(self.load_btn)
        vbox.addLayout(cmd_hbox)

        # Tree viewer
        self.tree = QTreeWidget()
        self.tree.setHeaderLabels(["Element", "Selector"])
        vbox.addWidget(self.tree, stretch=3)

        # Control bar
        ctrl_hbox = QHBoxLayout()
        self.start_btn = QPushButton("Start")
        self.start_btn.clicked.connect(self.start_audit)
        self.pause_btn = QPushButton("Pause")
        self.pause_btn.clicked.connect(self.pause_audit)
        ctrl_hbox.addWidget(self.start_btn)
        ctrl_hbox.addWidget(self.pause_btn)
        vbox.addLayout(ctrl_hbox)

        # Async queue polling
        self.ui_queue = ui_queue
        self.timer = QTimer()
        self.timer.timeout.connect(self.poll_queue)
        self.timer.start(50)   # 20 Hz enough for UI responsiveness

    def poll_queue(self):
        try:
            item = self.ui_queue.get_nowait()
            self.signal.update.emit(item)
        except asyncio.QueueEmpty:
            pass

    def handle_update(self, payload: dict):
        typ = payload.get("type")
        if typ == "log":
            self.log_edit.appendPlainText(payload["content"])
        elif typ == "state":
            session = payload["session"]
            self.refresh_tree(session)
        elif typ == "error":
            self.log_edit.appendPlainText(f"⚠️ {payload['msg']}")

    def refresh_tree(self, session):
        # Simple placeholder: rebuild whole tree each update.
        # In production, diff and update only changed nodes.
        self.tree.clear()
        root = QTreeWidgetItem(["<html>", "html"])
        self.tree.addTopLevelItem(root)
        for selector in session.focus_sequence:
            child = QTreeWidgetItem([selector, selector])
            root.addChild(child)
        self.tree.expandAll()
        # Highlight latest focus
        if session.last_selector:
            items = self.tree.findItems(session.last_selector, Qt.MatchFlag.MatchRecursive)
            for it in items:
                it.setBackground(0, Qt.GlobalColor.yellow)

    # ----- UI callbacks -----
    def send_command(self):
        cmd = self.cmd_input.text().strip()
        self.log_edit.appendPlainText(f">>> {cmd}")
        # In a full implementation we would forward to the agent's control channel
        self.cmd_input.clear()

    def load_url(self):
        # Placeholder dialog; in practice use QInputDialog.getText
        url, ok = QInputDialog.getText(self, "Load URL", "Enter page URL:")
        if ok and url:
            self.log_edit.appendPlainText(f"Loading {url} …")
            # Propagate to agent via a shared state object
            # (not shown for brevity)

    def start_audit(self):
        self.log_edit.appendPlainText("🚀 Starting audit…")
        # Trigger coroutine in background – see main runner below

    def pause_audit(self):
        self.log_edit.appendPlainText("⏸️ Paused")
        # Actual pause logic would send a control message to the agent loop

2.2 Streaming Claude responses without blocking

The key is to avoid any direct await inside a Qt slot. Instead we let the timer poll the asyncio.Queue that the agent writes into. This keeps the UI responsive even when Claude streams a thousand tokens.

# main.py – version: 3.0
import asyncio, sys, logging
from pathlib import Path
from agent import run_session, Session
from gui import MainWindow
from bridge import BrowserBridge  # defined later

logging.basicConfig(level=logging.INFO)

async def async_main():
    ui_queue = asyncio.Queue()
    bridge = BrowserBridge(ui_queue)

    # Build Qt app in the default thread
    app = QApplication(sys.argv)
    window = MainWindow(ui_queue)
    window.show()

    # Kick off the agent coroutine but don't block the Qt loop
    session = Session(url="https://example.com")
    asyncio.create_task(run_session(session, bridge, ui_queue))

    # Run Qt event loop while letting asyncio run in the same thread
    loop = asyncio.get_running_loop()
    await loop.run_in_executor(None, app.exec)

if __name__ == "__main__":
    asyncio.run(async_main())

Tip: app.exec() blocks the thread, so we schedule

Written by

’m Nilesh, a Software Development Engineer with 2+ years of experience, specializing in Go, JavaScript, Python, Docker, Kubernetes, Git, Jenkins, microservices, and system design (LLD/HLD), backed by a strong foundation in data structures and algorithms. Alongside my engineering journey, I bring 4+ years of hands-on experience in SEO, where I’ve worked extensively on content strategy, keyword research, technical SEO, and organic growth, helping products and businesses scale efficiently by aligning solid technology with search-driven performance.