When an autonomous Claude agent clicks around a live web app, the first thing that goes sideways is visibility. A CLI‑only loop shouts “next element” while the browser silently moves focus under the hood. By the time the agent reports a WCAG violation, you’ve already lost context, and the test run may have already tripped over a modal dialog you never saw.
The fix is to give the agent a visual cockpit: a desktop UI that streams Claude’s thoughts, highlights the element being inspected, and lets a human pause or intervene if the AI wanders off‑track. The result is a reproducible, auditable accessibility audit that runs in seconds instead of minutes.
- Claude’s tool‑use API can drive a real‑time screen‑reader simulator.
- A PyQt6 GUI visualizes the agent’s state and lets you steer execution.
- Computer Use API beats PyAutoGUI for reliable, pixel‑free interaction.
- Synchronizing three state machines (AI, GUI, browser) prevents race conditions.
- Built‑in rate‑limit handling keeps long audits from blowing up.
Before you start: Python 3.11+, Anthropic Python SDK v1.5+, Claude 3.5 Sonnet API key, Claude Desktop (or claude-computer-use sandbox), PyQt6 6.6, PyAutoGUI 0.9.54 (fallback only), keyboard 0.13.5, pynput 1.7.6, asyncio, a recent Chrome/Chromium (≥117) with remote‑debugging enabled, and optional pyttsx3 for TTS.
Build a Claude AI accessibility agent: automated screen‑reader & keyboard navigation (2026 guide)
A Claude AI accessibility agent automates screen reader simulation and keyboard navigation testing by combining Claude’s tool‑use API with a desktop GUI controller (like PyQt). It uses Python to bridge the AI’s commands to the OS, performing WCAG audits, reporting violations, and streamlining compliance workflows, all visualized in real‑time.
System Architecture & Workflow
The solution consists of three loosely‑coupled services that talk over asyncio queues:
- Agent Orchestrator – runs Claude 3.5 Sonnet, receives tool calls, maintains the reasoning context.
- GUI Controller – PyQt6 window that displays logs, a live DOM tree, and the currently focused element.
- Browser Bridge – either Claude Computer Use API (preferred) or PyAutoGUI for low‑level actions; feeds DOM/focus data back to the orchestrator.
graph TD
A[Claude Agent] -->|tool calls| B[Browser Bridge]
B -->|DOM updates| C[GUI Controller]
C -->|user commands| A
B -->|focus events| A
style A fill:#f9f,stroke:#333,stroke-width:2px
style B fill:#bbf,stroke:#333,stroke-width:2px
style C fill:#bfb,stroke:#333,stroke-width:2px
*Data flow*: Claude emits a simulate_screen_reader tool call → Browser Bridge extracts the element’s aria-label and visual text → GUI highlights the element and speaks it via TTS → Claude recovers the spoken text, decides the next keystroke, and repeats. All three components share a single asyncio.Queue for state updates, guaranteeing that the UI never freezes while Claude streams tokens.
Core Stack
| Layer | Library / API | Reason |
|---|---|---|
| LLM | Anthropic Python SDK v1.5+, Claude 3.5 Sonnet | Up‑to‑date model with built‑in tool‑use support |
| Desktop UI | PyQt6 (Qt 6.6) | Native look, fine‑grained event loop control |
| Browser control | Claude Computer Use API (semantic) + fallback PyAutoGUI | Semantic actions avoid brittle screen coordinates |
| Speech | pyttsx3 (offline TTS) | No network latency, works on Windows/macOS/Linux |
| Async orchestration | asyncio, asyncio.Queue | Non‑blocking streaming of Claude’s response |
My take: If you already ship a Qt‑based app, stick with PyQt6. If you need a tiny installer footprint, consider Tauri (Rust + WebView) but be prepared to embed a Python runtime via
pyembed. For this guide we stay with PyQt6 because its event loop integrates cleanly withasyncio.
—
Step 1: Building the Agent Core with Claude’s Tool‑Use
1.1 Prompt engineering for WCAG & accessibility
A well‑crafted system prompt gives Claude the mental model it needs. Keep it under 500 tokens; include a short WCAG 2.2 cheat sheet, ARIA conventions, and a clear contract for tool calls.
# prompt.txt – keep this file alongside main.py
SYSTEM_PROMPT = """You are an autonomous accessibility auditor.
Your goal is to walk a web page using only keyboard navigation (Tab, Shift+Tab, Arrow keys) and simulate a screen reader.
When you encounter an element, call `simulate_screen_reader` with its DOM selector.
If you need to move focus, call `execute_keystroke` with the key string (e.g., "Tab").
After each action, analyze the focus order and report any WCAG 2.2 violation you detect.
Only use the tools you have been given. Do not hallucinate element IDs."""
1.2 Defining custom tools
Claude’s Computer Use API works like a remote‑procedure call. We expose three Python wrappers that turn tool invocations into concrete actions.
# tools.py – version: 1.0
import asyncio
import json
import logging
from typing import Any, Dict
log = logging.getLogger("tools")
async def simulate_screen_reader(selector: str, bridge) -> Dict[str, Any]:
"""
Ask the bridge to fetch text and ARIA data for `selector`.
Returns a dict that Claude can read.
"""
try:
data = await bridge.fetch_element_info(selector)
# Speak via TTS (non‑blocking)
bridge.speak(data["accessible_name"])
return {"status": "ok", "data": data}
except Exception as e:
log.exception("simulate_screen_reader failed")
return {"status": "error", "msg": str(e)}
async def execute_keystroke(key: str, bridge) -> Dict[str, Any]:
"""
Perform a key press using Claude Computer Use API.
"""
try:
await bridge.send_keystroke(key)
return {"status": "ok"}
except Exception as e:
log.exception("execute_keystroke failed")
return {"status": "error", "msg": str(e)}
async def analyze_focus_order(state: Dict[str, Any]) -> Dict[str, Any]:
"""
Compare the accumulated focus sequence with expected ARIA order.
"""
try:
violations = state["focus_sequence"].detect_violations()
return {"status": "ok", "violations": violations}
except Exception as e:
log.exception("analyze_focus_order failed")
return {"status": "error", "msg": str(e)}
1.3 Agent loop with session state
The orchestrator runs a perpetual loop: send a system + user prompt, await streaming response, intercept tool calls, execute them, feed results back, and repeat. State is held in a Session dataclass.
# agent.py – version: 1.2
import asyncio
import json
import logging
from dataclasses import dataclass, field
from typing import List, Dict, Any
from anthropic import AsyncAnthropic
from tools import simulate_screen_reader, execute_keystroke, analyze_focus_order
log = logging.getLogger("agent")
anthropic = AsyncAnthropic(api_key="YOUR_ANTHROPIC_API_KEY")
@dataclass
class Session:
url: str
focus_sequence: List[str] = field(default_factory=list)
violations: List[Dict[str, Any]] = field(default_factory=list)
last_selector: str = ""
async def run_session(session: Session, bridge, ui_queue: asyncio.Queue):
system_prompt = open("prompt.txt").read()
user_msg = f"Audit the page at {session.url} for keyboard navigation and screen‑reader compliance."
async with anthropic.messages.streaming(
model="claude-3-5-sonnet-202406",
max_tokens=1024,
temperature=0.0,
system=system_prompt,
messages=[{"role": "user", "content": user_msg}],
tools=[
{"name": "simulate_screen_reader", "description": "Get accessible text", "input_schema": {"type": "object", "properties": {"selector": {"type": "string"}}}},
{"name": "execute_keystroke", "description": "Send a keyboard event", "input_schema": {"type": "object", "properties": {"key": {"type": "string"}}}},
{"name": "analyze_focus_order", "description": "Check the collected focus list", "input_schema": {"type": "object", "properties": {}}},
],
) as stream:
async for event in stream:
if event.type == "content_block_start":
# forward token stream to UI
await ui_queue.put({"type": "log", "content": event.content})
elif event.type == "tool_use":
# Dispatch to correct Python function
tool_name = event.name
args = json.loads(event.input)
if tool_name == "simulate_screen_reader":
result = await simulate_screen_reader(args["selector"], bridge)
elif tool_name == "execute_keystroke":
result = await execute_keystroke(args["key"], bridge)
elif tool_name == "analyze_focus_order":
result = await analyze_focus_order(session)
else:
result = {"status": "error", "msg": f"unknown tool {tool_name}"}
# Send result back to Claude
await stream.respond_tool_output(event.id, result)
# Update GUI
await ui_queue.put({"type": "state", "session": session})
elif event.type == "error":
log.error(f"Claude error: {event.error}")
await ui_queue.put({"type": "error", "msg": event.error})
break
The bridge object (implemented later) knows how to talk to Chrome via the Computer Use API, fetch DOM info, and speak via TTS. The ui_queue feeds real‑time logs and state updates to the PyQt6 frontend.
—
Step 2: Creating the Interactive Controller GUI
2.1 Layout – Log, Command, Element Tree
We keep the UI minimal but functional:
- Log Panel – scrolling
QPlainTextEditthat receives token stream. - Command Input – a
QLineEditthat allows the user to send ad‑hoc prompts or halt the run. - DOM Tree Viewer –
QTreeWidgetthat reflects the live DOM hierarchy; the currently focused node gets a colored background. - Control Bar – start, pause, and load‑URL buttons.
# gui.py – version: 2.0
import sys, asyncio, json, logging
from PyQt6.QtWidgets import (
QApplication, QMainWindow, QWidget, QVBoxLayout, QHBoxLayout,
QPlainTextEdit, QLineEdit, QPushButton, QTreeWidget, QTreeWidgetItem,
)
from PyQt6.QtCore import Qt, QTimer, pyqtSignal, QObject
log = logging.getLogger("gui")
class UIUpdateSignal(QObject):
update = pyqtSignal(dict) # passes dict from async queue
class MainWindow(QMainWindow):
def __init__(self, ui_queue: asyncio.Queue):
super().__init__()
self.setWindowTitle("Claude Accessibility Agent")
self.resize(1200, 800)
self.signal = UIUpdateSignal()
self.signal.update.connect(self.handle_update)
# Layout
central = QWidget()
self.setCentralWidget(central)
vbox = QVBoxLayout(central)
# Log panel
self.log_edit = QPlainTextEdit()
self.log_edit.setReadOnly(True)
vbox.addWidget(self.log_edit, stretch=2)
# Command line
cmd_hbox = QHBoxLayout()
self.cmd_input = QLineEdit()
self.cmd_input.setPlaceholderText("Enter admin command (e.g., pause, stop)")
self.cmd_input.returnPressed.connect(self.send_command)
cmd_hbox.addWidget(self.cmd_input)
self.load_btn = QPushButton("Load URL")
self.load_btn.clicked.connect(self.load_url)
cmd_hbox.addWidget(self.load_btn)
vbox.addLayout(cmd_hbox)
# Tree viewer
self.tree = QTreeWidget()
self.tree.setHeaderLabels(["Element", "Selector"])
vbox.addWidget(self.tree, stretch=3)
# Control bar
ctrl_hbox = QHBoxLayout()
self.start_btn = QPushButton("Start")
self.start_btn.clicked.connect(self.start_audit)
self.pause_btn = QPushButton("Pause")
self.pause_btn.clicked.connect(self.pause_audit)
ctrl_hbox.addWidget(self.start_btn)
ctrl_hbox.addWidget(self.pause_btn)
vbox.addLayout(ctrl_hbox)
# Async queue polling
self.ui_queue = ui_queue
self.timer = QTimer()
self.timer.timeout.connect(self.poll_queue)
self.timer.start(50) # 20 Hz enough for UI responsiveness
def poll_queue(self):
try:
item = self.ui_queue.get_nowait()
self.signal.update.emit(item)
except asyncio.QueueEmpty:
pass
def handle_update(self, payload: dict):
typ = payload.get("type")
if typ == "log":
self.log_edit.appendPlainText(payload["content"])
elif typ == "state":
session = payload["session"]
self.refresh_tree(session)
elif typ == "error":
self.log_edit.appendPlainText(f"⚠️ {payload['msg']}")
def refresh_tree(self, session):
# Simple placeholder: rebuild whole tree each update.
# In production, diff and update only changed nodes.
self.tree.clear()
root = QTreeWidgetItem(["<html>", "html"])
self.tree.addTopLevelItem(root)
for selector in session.focus_sequence:
child = QTreeWidgetItem([selector, selector])
root.addChild(child)
self.tree.expandAll()
# Highlight latest focus
if session.last_selector:
items = self.tree.findItems(session.last_selector, Qt.MatchFlag.MatchRecursive)
for it in items:
it.setBackground(0, Qt.GlobalColor.yellow)
# ----- UI callbacks -----
def send_command(self):
cmd = self.cmd_input.text().strip()
self.log_edit.appendPlainText(f">>> {cmd}")
# In a full implementation we would forward to the agent's control channel
self.cmd_input.clear()
def load_url(self):
# Placeholder dialog; in practice use QInputDialog.getText
url, ok = QInputDialog.getText(self, "Load URL", "Enter page URL:")
if ok and url:
self.log_edit.appendPlainText(f"Loading {url} …")
# Propagate to agent via a shared state object
# (not shown for brevity)
def start_audit(self):
self.log_edit.appendPlainText("🚀 Starting audit…")
# Trigger coroutine in background – see main runner below
def pause_audit(self):
self.log_edit.appendPlainText("⏸️ Paused")
# Actual pause logic would send a control message to the agent loop
2.2 Streaming Claude responses without blocking
The key is to avoid any direct await inside a Qt slot. Instead we let the timer poll the asyncio.Queue that the agent writes into. This keeps the UI responsive even when Claude streams a thousand tokens.
# main.py – version: 3.0
import asyncio, sys, logging
from pathlib import Path
from agent import run_session, Session
from gui import MainWindow
from bridge import BrowserBridge # defined later
logging.basicConfig(level=logging.INFO)
async def async_main():
ui_queue = asyncio.Queue()
bridge = BrowserBridge(ui_queue)
# Build Qt app in the default thread
app = QApplication(sys.argv)
window = MainWindow(ui_queue)
window.show()
# Kick off the agent coroutine but don't block the Qt loop
session = Session(url="https://example.com")
asyncio.create_task(run_session(session, bridge, ui_queue))
# Run Qt event loop while letting asyncio run in the same thread
loop = asyncio.get_running_loop()
await loop.run_in_executor(None, app.exec)
if __name__ == "__main__":
asyncio.run(async_main())
Tip:
app.exec()blocks the thread, so we schedule