Claude AI GUI Agent: Build a VSCode Assistant (2026)

When the Claude 3.5 Sonnet **Computer Use API** finally let a language model move the mouse, click buttons and type into a terminal, my first thought was: *why are we still firing off CLI‑only bots that talk to us through a scrolling log?* I tried wiring Claude to a plain `subprocess.Popen` that ran `code –list‑extensions`. The output streamed into the console, the agent responded, and then it stalled—no progress bar, no way to cancel, and the user was left guessing whether the process was still alive.

That experience crystallised what a real‑world AI assistant for developers must look like: a **responsive GUI** that streams Claude’s thoughts, shows live VS Code feedback, and lets you intervene with a button click before any destructive command runs. In this guide we assemble exactly that—a desktop app that talks to Claude, inspects your VS Code workspace, and safely installs or re‑configures extensions, all from a chat‑style interface.

How to Build a Claude‑Powered VSCode Analysis Assistant with GUI (2026)

This guide shows how to build a Claude AI agent in 2026 that analyzes and automates your VSCode setup. You’ll create a desktop GUI (using Tauri or PyQt) that lets you chat with an agent. The agent uses the Claude API and tool calling to read your VSCode extensions, settings, and performance data, then can safely install or configure extensions for you, all through an interactive interface.

The Vision: A Self‑Analyzing IDE, Powered by Claude

From Static Analytics to Intelligent, Conversational Workflow

Static linters and extension recommendations have been around forever, but they lack context. Claude can ask *why* you need a particular language server, suggest alternatives, and even install them on the fly—provided we give it a safe way to interact with the IDE. The result is a **conversation** where you ask “Why is my TypeScript build slow?” and Claude replies, “Your `eslint` extension is outdated; shall I update it?” and you click *Yes*.

The Core Architecture: Agent, GUI, and IDE Extension Interface

+-------------------+          +-------------------+          +-------------------+
|   Tauri Frontend  | <--IPC--> |   Python Agent   | <--API--> |   Claude Service |
+-------------------+          +-------------------+          +-------------------+
        ^                                   ^                         ^
        |                                   |                         |
   User Input                        Tool Calls               Claude 3.5 Sonnet
        |                                   |                         |
        +-------------------+---------------+-------------------------+
                            |
                     VSCode CLI (code)
                            |
                     VSCode Settings / Extensions

The diagram above (generated with Mermaid) captures the three‑tier flow: the **frontend** renders a chat UI, forwards user prompts to the **agent** over Tauri’s command bridge, the agent decides which **tool** to invoke (e.g., `analyze_extension`), calls the Claude API (or runs a child process), and finally streams results back to the UI.

flowchart LR
    UI[Chat UI] -->|prompt| Agent
    Agent -->|tool call| Claude
    Agent -->|exec| VSCodeCLI
    VSCodeCLI -->|data| Agent
    Agent -->|stream| UI

—

Prerequisites & Tech Stack (2026)

Component	Why It Fits	Version
**Python 3.12**	Native `asyncio`, `subprocess` improvements, and typed `TypedDict` for tool schemas.	≥ 3.12
**Anthropic Python SDK**	First‑class support for tool calling and streaming.	0.15.0+
**Claude 3.5 Sonnet**	Best balance of cost, context length (200 k tokens), and tool‑use capabilities.	2026‑release
**Tauri 2.0**	Rust‑based, < 8 MB binary, seamless bridge between JS UI and Rust/Python backend.	2.0
**VS Code CLI (`code`)**	Official, cross‑platform way to list/install extensions, open workspaces.	≥ 1.86
**Node 20**	Required by Tauri’s dev server.	20.x
**Rust 1.73**	Needed for the Tauri builder.	1.73

**My take:** If you already have a Python‑centric workflow, stick to **PyQt6** for the UI. It removes the Rust jump, but you pay a few megabytes in binary size. For the lightest distribution, Tauri wins hands‑down.

Setting Up the Project Skeleton

# 1️⃣ Create a fresh repo
git init claude-vscode-agent && cd claude-vscode-agent

# 2️⃣ Python virtual env
python -m venv .venv
source .venv/bin/activate

# 3️⃣ Install SDKs
pip install "anthropic[async]>=0.15.0" aiohttp tqdm

# 4️⃣ Initialise Tauri (will scaffold Rust + JS asset folder)
cargo install create-tauri-app
create-tauri-app . --template vanilla

The `src-tauri` folder now contains the Rust bridge, while the `src` folder holds the HTML/JS UI (we’ll replace it with a minimal React‑lite chat later).

—

Architecting the System: Agent Brain, State, and GUI

Core Agent Loop with Tool Calling

Claude’s `tool` messages look like JSON schemas. We define three tools:

Tool	Purpose	Input Schema
`analyze_extension`	Scan `~/.vscode/extensions` and parse each `package.json`.	`{ “workspace_path”: “string” }`
`execute_command`	Run a VS Code CLI sub‑command safely.	`{ “command”: “string”, “args”: [“string”] }`
`generate_report`	Collate findings into markdown.	`{ “analysis_id”: “string” }`

# agent.py – v0.1 – Python 3.12
import os, json, asyncio, uuid, subprocess
from pathlib import Path
from typing import TypedDict, List
from anthropic import Anthropic, AsyncClient, HUMAN_PROMPT, AI_PROMPT

# ---------- Typed tool payloads ----------
class AnalyzePayload(TypedDict):
    workspace_path: str

class ExecPayload(TypedDict):
    command: str
    args: List[str]

class ReportPayload(TypedDict):
    analysis_id: str

The main loop streams Claude’s response while handling tool calls:

async def run_agent(user_prompt: str, state: dict) -> None:
    client: AsyncClient = Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
    messages = [{"role": "user", "content": user_prompt}]
    # Append persisted state (if any)
    if state.get("analysis_cache"):
        messages.append({
            "role": "assistant",
            "content": f"Previously cached data: {json.dumps(state['analysis_cache'])}"
        })

    async with client.messages.stream(
        model="claude-3-5-sonnet-202406",
        max_tokens=4096,
        temperature=0.0,
        system="You are a VSCode analysis assistant...",
        messages=messages,
        tool_choice="auto"
    ) as stream:
        async for event in stream:
            if event.type == "content_block_start":
                continue
            if event.type == "tool_use":
                tool_name = event.name
                tool_input = json.loads(event.input)
                result = await dispatch_tool(tool_name, tool_input, state)
                # Send tool result back to Claude
                await client.messages.create(
                    model="claude-3-5-sonnet-202406",
                    max_tokens=1024,
                    messages=[{
                        "role": "assistant",
                        "content": [
                            {"type": "tool_result", "tool_use_id": event.id,
                             "content": json.dumps(result)}
                        ]
                    }]
                )
            else:
                # Forward normal text chunks to the UI via Tauri bridge
                await forward_to_ui(event.text)

**State Management** – We keep a JSON file (`state.json`) next to the executable. It holds the `analysis_cache` map keyed by workspace hash, preventing duplicate reads of the same `package.json` files. The cache is refreshed only when the user explicitly asks for a *re‑scan*.

STATE_PATH = Path(__file__).parent / "state.json"

def load_state() -> dict:
    return json.loads(STATE_PATH.read_text()) if STATE_PATH.exists() else {}

def save_state(state: dict) -> None:
    STATE_PATH.write_text(json.dumps(state, indent=2))

Designing a Non‑Blocking GUI

Tauri’s Rust side runs on its own thread, while the Python agent lives in an `asyncio` subprocess. The bridge uses **JSON‑RPC over a Unix domain socket** (`tauri::api::process::Command`). The UI never blocks because each message is sent with a `Promise` that resolves when the Python process writes a newline‑terminated JSON line.

// src-tauri/src/main.rs – v2.0
use tauri::{CustomMenuItem, Manager, Runtime, State};

#[tauri::command]
async fn send_prompt(state: State<'_, PyBridge>, prompt: String) -> Result<(), String> {
    state
        .inner()
        .send_message(prompt)
        .await
        .map_err(|e| e.to_string())
}

On the JS side we expose a `window.__TAURI__.invoke(‘send_prompt’, {prompt})` call whenever the user hits **Enter**. The response streams back via an EventSource:

// src/index.js
const eventSource = new EventSource('http://localhost:3001/stream');
eventSource.onmessage = e => appendChatMessage('assistant', e.data);

The streaming endpoint is a tiny `aiohttp` server started alongside the agent:

# stream_server.py
from aiohttp import web

async def stream(request):
    resp = web.StreamResponse(status=200, reason='OK', headers={'Content-Type': 'text/event-stream'})
    await resp.prepare(request)
    async for line in ui_queue:  # asyncio.Queue filled by forward_to_ui()
        await resp.write(f"data: {line}\n\n".encode())
    return resp

—

Building the Communication Layer

Implementing Computer Use API for GUI Control

Claude’s tool messages can request *mouse clicks* or *keyboard input*. Tauri currently exposes a **`tauri-plugin-computer-use`** (still in beta for 2026) that forwards such commands to the OS. We wrap this in a safe Python wrapper:

# computer_use.py
import subprocess, shlex

def click(x: int, y: int) -> None:
    # Use `xdotool` on Linux, `osascript` on macOS, `powershell` on Windows
    if os.name == "posix":
        subprocess.run(shlex.split(f"xdotool mousemove {x} {y} click 1"), check=True)
    # ...platform branches omitted for brevity

The agent never calls `click` directly; Claude must output a `tool_use` of type `”computer_click”` which we map to `click()`.

Interacting with VSCode: Extension Host & Node.js Child Processes

Running `code –list-extensions` is inexpensive, but `code –install-extension` can trigger UI prompts. We sandbox those calls behind a **confirmation dialog** in the UI:

async def execute_command(payload: ExecPayload, state: dict) -> dict:
    cmd = [payload["command"], *payload["args"]]
    proc = await asyncio.create_subprocess_exec(
        *cmd,
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE,
    )
    try:
        out, err = await asyncio.wait_for(proc.communicate(), timeout=30)
    except asyncio.TimeoutError:
        proc.kill()
        return {"error": "command timed out"}
    return {"stdout": out.decode(), "stderr": err.decode(), "returncode": proc.returncode}

The UI shows a *Cancel* button that simply kills the child process via a `signal.SIGTERM` packet sent over the same IPC channel.

Reliable IPC Between GUI, Agent Core, and VSCode Subprocesses

We serialize every message as a newline‑delimited JSON object on a **Unix domain socket** (`/tmp/claude_vscode.sock`). The Rust side opens the socket once and reuses it; the Python side writes to it with `asyncio.open_unix_connection`.

# ipc.py
async def send_to_gui(payload: dict) -> None:
    writer, _ = await asyncio.open_unix_connection(path="/tmp/claude_vscode.sock")
    writer.write((json.dumps(payload) + "\n").encode())
    await writer.drain()
    writer.close()

Because the socket is **non‑blocking**, the UI receives updates even while a long‑running `code` command streams lines of output.

—

Implementing Core Agent Functions

Function 1: `analyze_extension` – Reading VSCode Configs

async def analyze_extension(payload: AnalyzePayload, state: dict) -> dict:
    workspace = Path(payload["workspace_path"]).expanduser()
    extensions_dir = workspace / ".vscode" / "extensions"
    if not extensions_dir.is_dir():
        return {"error": "extensions folder not found"}

    results = []
    for ext_path in extensions_dir.iterdir():
        manifest = ext_path / "package.json"
        if manifest.is_file():
            try:
                data = json.loads(manifest.read_text())
                results.append({
                    "id": data.get("name"),
                    "version": data.get("version"),
                    "contributes": data.get("contributes", {}),
                    "enabled": not data.get("disabled", False)
                })
            except json.JSONDecodeError:
                continue

    # Cache for future runs
    cache_key = str(hash(workspace))
    state.setdefault("analysis_cache", {})[cache_key] = results
    save_state(state)
    return {"extensions": results, "count": len(results)}

Function 2: `execute_command` – Safely Running VSCode CLI

Already shown above, but we add a **sanity check** to whitelist only `install-extension`, `uninstall-extension`, and `list-extensions`.

ALLOWED_CMDS = {"install-extension", "uninstall-extension", "list-extensions"}

if payload["command"] != "code" or payload["args"][0] not in ALLOWED_CMDS:
    return {"error": "disallowed command"}

Function 3: `generate_report` – Structuring Findings

def generate_report(payload: ReportPayload, state: dict) -> dict:
    cache = state.get("analysis_cache", {})
    analysis = cache.get(payload["analysis_id"])
    if not analysis:
        return {"error": "no cached analysis found"}

    lines