Debugging Claude Computer Use Actions: Logging, Screenshots, and Replay

When Claude starts moving files, clicking buttons, or typing into a field on your machine, the CLI you used to launch the agent suddenly feels naked. A single mis‑click can cascade into a corrupted state, and without any visibility you’re left guessing which tool call went sideways. I hit that wall last week: the agent clicked “Save As” in Photoshop, but the dialog never appeared because the window lost focus. All I had was a cryptic error message and a stack trace that said nothing about the UI. That’s why a **real‑time, replay‑able debugger** is non‑negotiable for any production‑grade Claude Computer Use integration.

How to Debug Claude Computer Use Actions – A Three‑Pillar Approach

Debugging Claude Computer Use actions requires a three‑pillar system: 1) Structured logging of all tool calls, arguments, and responses via a wrapped Anthropic SDK client. 2) Automated, non‑blocking screenshot capture before/after actions using Python’s MSS library. 3) A replay engine that parses logs and associated screenshots to reconstruct the agent’s execution timeline for analysis.

System Architecture & Workflow

The debugger sits between Claude’s **Computer Use** runtime and the OS. Every tool call flows through a thin wrapper that writes a log entry, spawns an asynchronous screenshot task, and forwards the request to the real SDK client. A lightweight SQLite database (or a flat JSONL file) stores each entry together with a UUID that points at the screenshot files on disk. The optional GUI reads the DB, shows a timeline, and lets you jump to any point, replaying the original action or just visualizing the UI state.

graph TD
    A[Claude Agent Loop] --> B[Wrapped SDK Call]
    B --> C[Action Log Write]
    B --> D[Async Screenshot Capture]
    C --> E[SQLite / JSONL Store]
    D --> F[WebP Screenshot Files]
    E --> G[Debug GUI (PyQt/Streamlit)]
    F --> G
    G --> H[Replay Engine (Parse + Simulate)]

**My take:** Stick with SQLite for the first iteration; it gives you built‑in indexing for fast timeline queries without pulling in another dependency.

—

Step 1 – Capturing & Structuring the Action Log

1.1 Wrap the Anthropic SDK client

We subclass the `anthropic.AsyncClient` (the SDK is fully async in 2026) and intercept every `computer_use` request.

# file: debug_client.py
# Requires: anthropic>=2026.5, pydantic>=2.5
import uuid
import json
import datetime
import asyncio
from pathlib import Path
from typing import Any, Dict

from anthropic import AsyncClient, Completion
from pydantic import BaseModel, Field

# ----------------------------------------------------------------------
# Pydantic schema – a single immutable log entry
# ----------------------------------------------------------------------
class ActionLogEntry(BaseModel):
    id: str = Field(default_factory=lambda: str(uuid.uuid4()))
    timestamp: datetime.datetime = Field(default_factory=datetime.datetime.utcnow)
    tool_name: str
    arguments: Dict[str, Any]
    response: Dict[str, Any] | None = None
    tokens_used: int | None = None
    duration_ms: int | None = None
    screenshot_pre: str | None = None
    screenshot_post: str | None = None
    error: str | None = None

    class Config:
        frozen = True

# ----------------------------------------------------------------------
# Async logger – writes JSONL lines atomically
# ----------------------------------------------------------------------
LOG_PATH = Path("./logs/action_log.jsonl")
LOG_PATH.parent.mkdir(parents=True, exist_ok=True)

async def append_log(entry: ActionLogEntry) -> None:
    async with asyncio.Lock():
        async with aiofiles.open(LOG_PATH, mode="a") as f:
            await f.write(entry.model_dump_json() + "\n")

# ----------------------------------------------------------------------
# Debug client – delegates to the real Anthropic client
# ----------------------------------------------------------------------
class DebugAnthropicClient(AsyncClient):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self._log_lock = asyncio.Lock()

    async def computer_use(self, **tool_kwargs) -> Completion:
        # 1️⃣ Record pre‑action screenshot (non‑blocking)
        pre_uuid = await capture_screenshot_async(purpose="pre")

        # 2️⃣ Build the log entry (response fields will be filled later)
        entry = ActionLogEntry(
            tool_name="computer_use",
            arguments=tool_kwargs,
            screenshot_pre=pre_uuid,
        )
        start = datetime.datetime.utcnow()

        try:
            # 3️⃣ Forward the request to the real SDK
            resp = await super().computer_use(**tool_kwargs)
            # 4️⃣ Capture post‑action screenshot
            post_uuid = await capture_screenshot_async(purpose="post")
            entry.screenshot_post = post_uuid

            # 5️⃣ Populate response fields
            entry.response = resp.model_dump()
            entry.tokens_used = resp.usage.input_tokens + resp.usage.output_tokens
            entry.duration_ms = int((datetime.datetime.utcnow() - start).total_seconds() * 1000)

        except Exception as exc:
            entry.error = repr(exc)
            # Still try to capture a screenshot for the error state
            post_uuid = await capture_screenshot_async(purpose="error")
            entry.screenshot_post = post_uuid
            raise

        finally:
            # 6️⃣ Persist the entry (do not block the main loop)
            asyncio.create_task(append_log(entry))

        return resp

**Key points**

• The `ActionLogEntry` is immutable (`frozen=True`) so accidental mutation can’t corrupt the timeline.
• We use `aiofiles` for async file I/O – no blocking calls on the agent’s event loop.
• Screenshots are taken *before* and *after* the SDK call, plus on error, ensuring full visual coverage.

1.2 Persisting logs – SQLite alternative

If you prefer relational queries (e.g., filter by tool name or error flag), the same schema maps cleanly to SQLite via `sqlite-utils`.

pip install sqlite-utils

# file: sqlite_logger.py
import sqlite3
from pathlib import Path
from debug_client import ActionLogEntry

DB_PATH = Path("./logs/debug.db")
DB_PATH.parent.mkdir(parents=True, exist_ok=True)

def init_db():
    conn = sqlite3.connect(DB_PATH)
    cur = conn.cursor()
    cur.execute("""CREATE TABLE IF NOT EXISTS actions (
        id TEXT PRIMARY KEY,
        timestamp TEXT,
        tool_name TEXT,
        arguments TEXT,
        response TEXT,
        tokens_used INTEGER,
        duration_ms INTEGER,
        screenshot_pre TEXT,
        screenshot_post TEXT,
        error TEXT
    )""")
    conn.commit()
    conn.close()

async def append_sql(entry: ActionLogEntry):
    conn = sqlite3.connect(DB_PATH)
    cur = conn.cursor()
    cur.execute(
        """INSERT INTO actions VALUES (?,?,?,?,?,?,?,?,?,?)""",
        (
            entry.id,
            entry.timestamp.isoformat(),
            entry.tool_name,
            json.dumps(entry.arguments),
            json.dumps(entry.response) if entry.response else None,
            entry.tokens_used,
            entry.duration_ms,
            entry.screenshot_pre,
            entry.screenshot_post,
            entry.error,
        ),
    )
    conn.commit()
    conn.close()

Calling `init_db()` at startup gives you instant indexing for timeline inspection.

—

Step 2 – Automated, Non‑Blocking Screenshot Capture

Capturing the screen is I/O‑heavy. A naïve `PIL.ImageGrab.grab()` call blocks the event loop, causing the agent to miss its own timeout. The solution is to run the capture in a dedicated thread pool and immediately return a UUID.

# file: screenshot.py
# Requires: mss>=9.0, pillow>=10.2, aiofiles
import asyncio
import uuid
from pathlib import Path
from concurrent.futures import ThreadPoolExecutor

import mss
from PIL import Image

# Directory for screenshots; use WebP for 85% quality (~30 KB per 1080p frame)
SHOT_DIR = Path("./screenshots")
SHOT_DIR.mkdir(parents=True, exist_ok=True)

_executor = ThreadPoolExecutor(max_workers=2)

def _capture_raw(purpose: str) -> Path:
    """Runs in a background thread."""
    with mss.mss() as sct:
        monitor = sct.monitors[0]  # full virtual screen
        raw = sct.grab(monitor)
        img = Image.frombytes("RGB", raw.size, raw.rgb)
        uid = f"{uuid.uuid4()}_{purpose}.webp"
        out_path = SHOT_DIR / uid
        img.save(out_path, format="WEBP", quality=85)
        return out_path

async def capture_screenshot_async(purpose: str) -> str | None:
    """Schedule a non‑blocking capture. Returns the filename (or None on failure)."""
    loop = asyncio.get_running_loop()
    try:
        path = await loop.run_in_executor(_executor, _capture_raw, purpose)
        return str(path)
    except Exception as exc:
        # Never let a screenshot error bubble up to the agent
        print(f"[WARN] Screenshot ({purpose}) failed: {exc}")
        return None

**Why this works**

• `ThreadPoolExecutor` isolates the heavy pixel copy from the asyncio reactor.
• The function returns a **filename** that can be stored in the log entry.
• In case of failure we log a warning but continue –‑ the agent never sees a hard error.

—

Step 3 – Building the Replay Engine

The replay engine is a pure‑Python library that reads the log (JSONL or SQLite) and offers two APIs:

1. **`timeline()`** – an iterator that yields `ActionLogEntry` objects in order.
2. **`replay(entry_id)`** – a function that re‑executes the original tool call *in a sandbox* (no side effects) or simply displays the associated screenshots side‑by‑side.

3.1 Parsing the Log

# file: replay_engine.py
import json
from pathlib import Path
from typing import Iterator, List
from debug_client import ActionLogEntry

LOG_FILE = Path("./logs/action_log.jsonl")

def timeline() -> Iterator[ActionLogEntry]:
    """Yield log entries ordered by timestamp (fast for JSONL)."""
    with LOG_FILE.open("r") as f:
        for line in f:
            entry = ActionLogEntry.model_validate_json(line)
            yield entry

def load_entry(entry_id: str) -> ActionLogEntry | None:
    with LOG_FILE.open("r") as f:
        for line in f:
            entry = ActionLogEntry.model_validate_json(line)
            if entry.id == entry_id:
                return entry
    return None

3.2 Visual Replay

We provide a tiny HTML‑based viewer that can be embedded in the GUI. It loads the two WebP files and paints them next to each other, with a diff overlay if needed.

# file: replay_ui.py
import base64
from pathlib import Path
from IPython.display import HTML, display  # Works in Streamlit & Jupyter

def img_tag(path: Path) -> str:
    data = base64.b64encode(path.read_bytes()).decode()
    return f'<img src="data:image/webp;base64,{data}" style="max-width:48%;border:1px solid #ccc;">'

def show_replay(entry: ActionLogEntry):
    pre = Path(entry.screenshot_pre) if entry.screenshot_pre else None
    post = Path(entry.screenshot_post) if entry.screenshot_post else None
    html = "<div style='display:flex;gap:2%;'>"
    html += img_tag(pre) if pre and pre.exists() else "<div style='flex:1;'></div>"
    html += img_tag(post) if post and post.exists() else "<div style='flex:1;'></div>"
    html += "</div>"
    display(HTML(html))

In a Streamlit app you can simply call `st.components.v1.html(show_replay(entry), height=400)`.

3.3 Dealing with Non‑Determinism

Claude’s responses can change between replays because the underlying language model is stochastic. To keep the timeline meaningful we **record the `completion_id`** (available on the `Completion` object) and **display it in the UI**. If a replay diverges, you can highlight it with a red badge.

# Extend ActionLogEntry (add to schema)
completion_id: str | None = None

When persisting, set:

entry.completion_id = resp.id  # assuming `resp.id` holds the model's completion ID

During replay, compare `entry.completion_id` with a fresh call and flag mismatches.

—

Step 4 – The Debug GUI Frontend (Choose Your Framework)

You have two pragmatic routes. Both consume the same SQLite/JSONL backend, so you can swap later without touching the core.

Option A: Native Desktop with PyQt6

PyQt6 gives you a responsive, multi‑window UI that can run offline.

# file: gui_pyqt.py
# Requires: pyqt6>=6.6, sqlite-utils
import sys
from pathlib import Path
from PyQt6.QtWidgets import (
    QApplication, QMainWindow, QListWidget, QLabel,
    QVBoxLayout, QWidget, QPushButton, QHBoxLayout, QSplitter
)
from PyQt6.QtGui import QPixmap
from sqlite_utils import Database
from debug_client import ActionLogEntry

DB_PATH = Path("./logs/debug.db")
db = Database(DB_PATH)

class DebugWindow(QMainWindow):
    def __init__(self):
        super().__init__()
        self.setWindowTitle("Claude Computer‑Use Debugger")
        self.resize(1200, 800)

        # Left pane – timeline list
        self.log_list = QListWidget()
        self.log_list.itemClicked.connect(self.show_entry)

        # Right pane – details
        self.detail_lbl = QLabel("Select an action")
        self.detail_lbl.setWordWrap(True)

        # Screenshot area
        self.pre_img = QLabel()
        self.post_img = QLabel()
        img_layout = QHBoxLayout()
        img_layout.addWidget(self.pre