Claude AI GUI Agent: Build a PyQt Test Harness (2026)

When I first tried to let an autonomous Claude agent click through a PyQt‑based demo app, the UI locked up after the first tool call. The event loop was blocked by a synchronous HTTP request, and the agent’s streaming response never appeared in the window. After a night of hunting down the culprit—​mixing the main Qt thread with blocking `requests`—​I rewrote the whole bridge as an async, checkpoint‑driven loop that lives in a dedicated `QThread`. The result is an AI‑powered test harness that can explore, assert, and recover from crashes without freezing the GUI.

Building an AI‑Driven E2E Test Agent for PyQt (snippet)

A Claude Agent for E2E testing of PyQt apps is an AI‑driven framework. You equip a Claude model (via the Anthropic API) with a custom set of Python tools — like click, type, and assert — that interact with your PyQt GUI’s widgets. The agent autonomously interprets goals, executes tools, and validates UI state, enabling exploratory testing beyond static scripts.

Why Claude Agents Are a Game‑Changer for GUI Testing

Beyond Scripting: AI‑Driven Exploratory Testing

Traditional UI test suites are static: you record a sequence of clicks, then replay it. Any UI change breaks the script. Claude can interpret high‑level goals (“verify that the login flow accepts a wrong password”) and decide on‑the‑fly which widgets to touch, adapting to layout changes without rewriting code.

The Architecture: Claude, Tools, and Your PyQt Event Loop

1. **Claude Core** – A conversation with Claude 3.7 Sonnet, driven by the Anthropic Messages API.
2. **Tool Kit** – Python functions exposed to Claude (`click`, `type`, `assert`, `screenshot`). Each tool returns a structured JSON payload that Claude can reason about.
3. **Bridge Layer** – A `QThread` runs the async Claude loop, emitting Qt signals for UI updates. The main thread hosts the target PyQt app and a tiny controller widget that shows live reasoning.
4. **State Store** – A SQLite (or JSON) checkpoint file records conversation turns, widget snapshots, and timestamps. On restart, the agent loads the last checkpoint and resumes.

Prerequisites: Python 3.11+, Anthropic SDK (2026), PyQt6

The Anthropic SDK now ships with native async support and a `structured_output` helper, which we’ll lean on to keep the payloads tidy.

—

Project Setup: Core Dependencies & Initial Scaffolding

Virtual Environment & Installing the 2026 SDKs

python3.11 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install "anthropic>=0.7.0" "PyQt6>=6.6.0" pytest Pillow

Create a minimal `requirements.txt` for reproducibility:

# requirements.txt - 2026 versions
anthropic>=0.7.0
PyQt6>=6.6.0
pytest>=8.0
Pillow>=10.2

Building the Basic PyQt Application (Our Test Target)

Save this as `demo_app.py`. It’s deliberately tiny but covers enough widgets to showcase the agent.

# demo_app.py - version 1.0
import sys
from PyQt6.QtWidgets import (
    QApplication, QWidget, QLabel, QLineEdit,
    QPushButton, QVBoxLayout, QMessageBox,
)
from PyQt6.QtCore import Qt

class LoginWindow(QWidget):
    def __init__(self):
        super().__init__()
        self.setWindowTitle("Demo Login")
        self.setObjectName("login_window")
        layout = QVBoxLayout(self)

        self.info = QLabel("Enter credentials")
        self.info.setObjectName("info_label")
        layout.addWidget(self.info)

        self.user = QLineEdit()
        self.user.setPlaceholderText("Username")
        self.user.setObjectName("username_input")
        layout.addWidget(self.user)

        self.pwd = QLineEdit()
        self.pwd.setEchoMode(QLineEdit.EchoMode.Password)
        self.pwd.setPlaceholderText("Password")
        self.pwd.setObjectName("password_input")
        layout.addWidget(self.pwd)

        self.btn = QPushButton("Login")
        self.btn.setObjectName("login_button")
        self.btn.clicked.connect(self.authenticate)
        layout.addWidget(self.btn)

    def authenticate(self):
        if self.user.text() == "admin" and self.pwd.text() == "secret":
            QMessageBox.information(self, "Success", "Logged in")
        else:
            QMessageBox.warning(self, "Failed", "Bad credentials")

def main():
    app = QApplication(sys.argv)
    win = LoginWindow()
    win.show()
    sys.exit(app.exec())

if __name__ == "__main__":
    main()

Run it once to confirm it works:

python demo_app.py

Structuring the Agent’s Tool Kit & Prompt System

Create a package `agent/` with three modules: `tools.py`, `bridge.py`, and `state.py`.

mkdir -p agent
touch agent/__init__.py

`agent/tools.py` defines the primitive actions the model can call.

# agent/tools.py - version 1.0
import json
from typing import Any, Dict, Tuple
from PyQt6.QtTest import QTest
from PyQt6.QtCore import Qt, QPoint
from PyQt6.QtWidgets import QApplication, QWidget
from PIL import ImageGrab
import os

# Helper to locate a widget by objectName, falling back to a visual search
def find_widget(path: str) -> QWidget:
    widget = QApplication.instance().findChild(QWidget, path)
    if widget:
        return widget
    raise RuntimeError(f"Widget with objectName '{path}' not found")

def click(target: str) -> Dict[str, Any]:
    """Simulate a mouse click on a widget identified by its objectName."""
    try:
        w = find_widget(target)
        QTest.mouseClick(w, Qt.MouseButton.LeftButton)
        return {"status": "ok", "action": "click", "target": target}
    except Exception as e:
        return {"status": "error", "error": str(e), "action": "click"}

def type_text(target: str, text: str) -> Dict[str, Any]:
    """Send keystrokes to a QLineEdit."""
    try:
        w = find_widget(target)
        if not hasattr(w, "setText"):
            raise TypeError("Target is not an input widget")
        QTest.keyClicks(w, text)
        return {"status": "ok", "action": "type", "target": target, "text": text}
    except Exception as e:
        return {"status": "error", "error": str(e), "action": "type"}

def assert_widget(target: str, property: str, expected: Any) -> Dict[str, Any]:
    """Validate a widget property (e.g., text, isVisible)."""
    try:
        w = find_widget(target)
        actual = getattr(w, property)()
        if actual == expected:
            return {"status": "ok", "action": "assert", "target": target}
        return {
            "status": "fail",
            "action": "assert",
            "target": target,
            "expected": expected,
            "actual": actual,
        }
    except Exception as e:
        return {"status": "error", "error": str(e), "action": "assert"}

def screenshot(target: str = None) -> Dict[str, Any]:
    """Capture the whole screen or a single widget and return a base64 PNG."""
    try:
        if target:
            w = find_widget(target)
            rect = w.geometry()
            img = ImageGrab.grab(bbox=(
                rect.x(),
                rect.y(),
                rect.x() + rect.width(),
                rect.y() + rect.height(),
            ))
        else:
            img = ImageGrab.grab()
        path = f"tmp_{target or 'screen'}.png"
        img.save(path, "PNG")
        return {"status": "ok", "path": os.path.abspath(path)}
    except Exception as e:
        return {"status": "error", "error": str(e)}

`agent/state.py` holds conversation checkpoints and snapshot logic.

# agent/state.py - version 1.0
import json
import os
from pathlib import Path
from typing import List, Dict

CHECKPOINT_FILE = Path("agent_state.json")

def load_state() -> Dict[str, Any]:
    if CHECKPOINT_FILE.exists():
        with open(CHECKPOINT_FILE, "r", encoding="utf-8") as f:
            return json.load(f)
    return {"messages": [], "turn": 0, "snapshots": []}

def save_state(state: Dict[str, Any]) -> None:
    with open(CHECKPOINT_FILE, "w", encoding="utf-8") as f:
        json.dump(state, f, indent=2)

def add_snapshot(state: Dict[str, Any], description: str, img_path: str) -> None:
    state["snapshots"].append(
        {"turn": state["turn"], "desc": description, "path": img_path}
    )
    save_state(state)

`agent/bridge.py` glues Claude, the tool kit, and Qt signals.

# agent/bridge.py - version 1.0
import asyncio
import json
import os
from typing import Any, Dict, List

import anthropic
from PyQt6.QtCore import QObject, pyqtSignal, QThread, QTimer

from .tools import click, type_text, assert_widget, screenshot
from .state import load_state, save_state, add_snapshot

API_KEY = os.getenv("ANTHROPIC_API_KEY")
if not API_KEY:
    raise RuntimeError("Set ANTHROPIC_API_KEY environment variable")

client = anthropic.AsyncClient(api_key=API_KEY)

SYSTEM_PROMPT = """You are an autonomous QA agent. Your goal is to verify UI behavior described by the user. Use only the provided tools. Each tool call must be idempotent; if a step fails, you may retry after a checkpoint."""
# The checkpoint mechanism is handled by our local state store.

class ClaudeWorker(QObject):
    # Signals emitted to the main thread
    reasoning = pyqtSignal(str)      # Streaming text
    tool_result = pyqtSignal(dict)   # Raw tool JSON
    finished = pyqtSignal()

    def __init__(self, goal: str):
        super().__init__()
        self.goal = goal
        self.state = load_state()
        self.loop = asyncio.new_event_loop()
        self._cancel = False

    def start(self):
        asyncio.set_event_loop(self.loop)
        self.loop.run_until_complete(self.run_conversation())

    async def run_conversation(self):
        # Append system prompt if fresh
        if not self.state["messages"]:
            self.state["messages"].append({"role": "system", "content": SYSTEM_PROMPT})
        self.state["messages"].append({"role": "user", "content": self.goal})
        save_state(self.state)

        async for chunk in client.messages.stream(
            model="claude-3-7-sonnet-202406",
            max_tokens=1024,
            temperature=0.0,
            messages=self.state["messages"],
            tool_definitions=[
                {"name": "click", "description": "click(widget)", "input_schema": {"type": "object", "properties": {"target": {"type": "string"}}}},
                {"name": "type_text", "description": "type_text(widget, text)", "input_schema": {"type": "object", "properties": {"target": {"type": "string"}, "text": {"type": "string"}}}},
                {"name": "assert_widget", "description": "assert_widget(widget, property, expected)", "input_schema": {"type": "object", "properties": {"target": {"type": "string"}, "property": {"type": "string"}, "expected": {"type": "any"}}}},
                {"name": "screenshot", "description": "screenshot(widget?)", "input_schema": {"type": "object", "properties": {"target": {"type": "string"}}}},
            ],
            stream=True,
        ):
            if self._cancel:
                break
            if chunk.type == "content_block_start" and chunk.name == "assistant":
                # Start of a new assistant turn
                self.state["turn"] += 1
            if chunk.type == "content_block_delta":
                # Streaming text
                self.reasoning.emit(chunk.text)
            elif chunk.type == "tool_use":
                # Claude wants to invoke a tool
                result = await self.dispatch_tool(chunk.name, chunk.input)
                self.tool_result.emit(result)
                # Append tool result to conversation
                self.state["messages"].append(
                    {"role": "assistant", "content": [{"type": "tool_result", "name": chunk.name, "output": result}]}
                )
                save_state(self.state)
        self.finished.emit()

    async def dispatch_tool(self, name: str, args: Dict[str, Any]) -> Dict[str, Any]:
        # Map tool name to local functions (sync, so run in thread pool)
        if name == "click":
            return click(**args)
        if name == "type_text":
            return type_text(**args)
        if name == "assert_widget":
            return assert_widget(**args)
        if name == "screenshot":
            res = screenshot(**args)
            # Store snapshot for later inspection
            add_snapshot(self.state, f"Tool:{name}", res.get("path", ""))
            return res
        return {"status": "error", "error": f"Unknown tool {name}"}

    def stop(self):
        self._cancel = True

The bridge spawns a `QThread`, runs the async loop, and communicates back via signals. We’ll wire it up next.

—

Agent Core: Designing the Tool‑Execution Loop for PyQt

Essential Tool Definitions: Click, Type, Assert, Screenshot

All tools are pure functions that receive simple JSON arguments and return a JSON packet. By keeping them synchronous and side‑effect‑free (except the UI mutation), we can safely call them from the worker thread without worrying about the GIL holding the main loop.

State Management Between GUI Widgets and Agent Memory

The `state.py` module writes a single JSON file after each turn. The structure includes:

{
  "messages": [ ... ],
  "turn": 12,
  "snapshots": [
    {"turn": 3, "desc": "Tool:screenshot", "path": "/abs/tmp_login.png"}
  ]
}

On crash, the main program loads this file, restores `self.state`, and continues from the exact turn number. This satisfies the “durable conversation history” gap.

Implementing Async API Calls with Structured Outputs

Anthropic’s 2026 SDK supports `structured_output` where we define the expected schema for each tool. The worker uses that to validate Claude’s request before dispatch. Errors like “tool name not found” become deterministic JSON payloads rather than opaque exceptions.

—

GUI Integration: Bridging the Agent and PyQt’s Main Thread

Creating the Controller Widget for Agent Status & Control

We’ll embed a simple side‑panel that shows Claude’s reasoning live, a **Cancel** button, and a log view.

# controller.py - version 1.0
import sys
from PyQt6.QtWidgets import (
    QWidget, QVBoxLayout, QTextEdit, QPushButton, QLabel,
)
from PyQt6.QtCore import Qt, pyqtSlot
from agent.bridge import ClaudeWorker, QThread

class AgentConsole(QWidget):
    def __init__(self, goal: str):
        super().__init__()
        self.setWindowTitle("Claude Test Agent")
        self.setObjectName("agent_console")
        layout = QVBoxLayout(self)

        self.status = QLabel("Idle")
        layout.addWidget(self.status)

        self.log = QTextEdit()
        self.log.setReadOnly(True)
        layout.addWidget(self.log, stretch=1)

        self.btn = QPushButton("Start Test")
        self.btn.clicked.connect(self.start_agent)
        layout.addWidget(self.btn)

        self.cancel_btn = QPushButton("Cancel")
        self.cancel_btn.clicked.connect(self.stop_agent)
        self.cancel_btn.setEnabled(False)
        layout.addWidget(self.cancel_btn)

        self.goal = goal
        self.thread = None
        self.worker = None

    @pyqtSlot