Claude AI GUI Agent: Build a Real‑Time Linux Copilot (2026)

When you let an LLM execute shell commands on a tiling window manager, the first thing that goes wrong is usually the UI freezing while the model waits for a tool response. You’ll see the overlay spinner spin forever, the hotkey you used to summon the assistant stops working, and any open terminal hangs because the main thread is blocked. The root cause isn’t “Claude is slow”—it’s a mismatched event loop that forces the GUI to wait on network I/O.

The fix is to separate the AI core from the Qt UI, drive everything with `asyncio`, and pipe tool calls through a `queue.Queue` that a dedicated `QThread` consumes. When the thread finishes a subprocess, it puts the result back on the asyncio queue, which the agent resumes without ever blocking the main Qt thread. The result is a buttery‑smooth overlay that lives in i3, Sway, or any other tiling WM and can answer you in real‑time without stealing focus from your work.

—

How to Build an AI Copilot for a Linux Tiling WM (2026)

To build an AI Copilot for a Linux tiling WM in 2026: use Python, PyQt6 for the GUI, and Anthropic’s Claude SDK with Tool Use for the AI core. Implement an event loop that connects the GUI, AI API calls, and system tools like subprocess, delivering assistance that lives in your WM without freezing apps.

Understanding the 2026 AI Copilot Architecture for Linux WMs

Agent Core vs. GUI Shell Separation

The agent is a pure‑async coroutine that talks to Claude, decides which tool to call, and awaits the tool’s result. The GUI shell is a thin PyQt6 overlay that only renders text, captures keystrokes, and forwards user messages to the agent via an `asyncio.Queue`. All heavy lifting—subprocess execution, window inspection, file I/O—happens in a **tool executor thread** that reads from a second queue, runs the request, and pushes the response back.

Event Loop: Bridging AI, GUI, and System Events

Qt (main thread)  <--->  asyncio event loop  <--->  QThread (tool executor)

• **Qt → asyncio**: `QTimer` fires every 10 ms, calling `asyncio.get_event_loop().call_soon_threadsafe` to drain the inbound message queue.
• **asyncio → QThread**: When the agent emits a `ToolCall`, it enqueues a `ToolRequest` object onto a `queue.Queue` that the executor thread blocks on.
• **QThread → asyncio**: After the subprocess finishes, the thread puts a `ToolResult` back onto an `asyncio.Queue`, waking the awaiting coroutine.

Key Architectural Principles for Latency & Stability

1. **Never block the Qt event loop** – all I/O is `await`‑ed or executed in the worker thread.
2. **Keep the Claude context window under 128 k tokens** – prune older messages when the session exceeds the limit.
3. **Graceful degradation** – if Claude returns a malformed tool spec, fall back to a user‑confirmation dialog instead of crashing.

—

graph LR
    A[Qt GUI Thread] -->|User Input| B[asyncio Core]
    B -->|ToolCall| C[QThread Executor]
    C -->|Subprocess| D[OS]
    D -->|Result| C
    C -->|ToolResult| B
    B -->|Response| A

Workspace Environment & Tool Stack Setup

Python with Asyncio & PyQt6 (not Tkinter/GTK)

Python 3.12 ships with built‑in `asyncio` improvements: `TaskGroup` and `timeout` helpers. PyQt6 provides native Wayland support and a high‑performance `QQuickOverlay` widget we’ll use as a floating panel.

# Create an isolated venv
python3.12 -m venv .venv
source .venv/bin/activate
pip install "pyqt6>=6.6" "anthropic[tool_use]>=0.7" psutil pydbus

Anthropic’s Claude 2026 SDK & Tool Use API

Claude 3.5 Sonnet introduced the **Tool Use** schema that returns a deterministic JSON payload (`type: “tool_use”`). The SDK automatically converts that payload to a `ToolResult` object when you call `client.completion(…, tool_use=True)`.

# anthropic_client.py
import os
from anthropic import Anthropic, AsyncClaude

# My take: keep a singleton client; re‑creating it per request adds ~30 ms latency.
client = Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
async_claude = AsyncClaude(client)  # wraps the sync client for asyncio

OS-level Access: Subprocess, File I/O, and WM Control

• **subprocess** – `asyncio.create_subprocess_shell` for non‑blocking shells.
• **psutil** – query process tree, find the active window’s PID.
• **pydbus** – talk to `org.kde.KWin` or `org.freedesktop.impl.portal.desktop` for window placement on Wayland.
• **i3‑msg / swaymsg** – control tiling layouts (focus, move, resize).

# wm_utils.py
import asyncio, shlex, json
from pathlib import Path

async def run_cmd(cmd: str) -> str:
    proc = await asyncio.create_subprocess_shell(
        cmd,
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE,
    )
    out, err = await proc.communicate()
    if proc.returncode != 0:
        raise RuntimeError(f"Command failed: {err.decode()}")
    return out.decode().strip()
    
async def i3_focus(window_id: str):
    await run_cmd(f"i3-msg [id={shlex.quote(window_id)}] focus")

Core Agent Implementation: Claude with Persistent State

Prompt Engineering for Long‑Running Agent Context

We prepend a system prompt that tells Claude it is a **desktop assistant** with a bounded toolbox. The prompt also includes a JSON schema for each tool, letting Claude generate strict tool calls.

SYSTEM_PROMPT = """You are a desktop copilot running on a Linux tiling WM.
You have access to the following tools only: file_search, exec, win_control.
Always respond with a JSON object matching the tool schema when you need to act.
If you are unsure, ask the user for clarification before calling any tool."""

State Management: Session, Tool History, and User Preferences

All messages are stored in a SQLite DB (`assistant_state.db`) with an expiring “session_id” column. When the session exceeds 120 minutes of inactivity, we start a fresh Claude conversation and archive the old one.

# state.py
import sqlite3, json, time
from pathlib import Path

DB_PATH = Path.home() / ".copilot" / "assistant_state.db"
DB_PATH.parent.mkdir(parents=True, exist_ok=True)

def init_db():
    con = sqlite3.connect(DB_PATH)
    cur = con.cursor()
    cur.execute("""CREATE TABLE IF NOT EXISTS messages(
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        session TEXT,
        role TEXT,
        content TEXT,
        ts REAL
    )""")
    con.commit()
    con.close()

def add_message(session: str, role: str, content: str):
    con = sqlite3.connect(DB_PATH)
    cur = con.cursor()
    cur.execute(
        "INSERT INTO messages (session, role, content, ts) VALUES (?,?,?,?)",
        (session, role, content, time.time()),
    )
    con.commit()
    con.close()

**My take:** SQLite is cheap, fast, and survives a hard reboot. Don’t over‑engineer with a full‑blown ORM—simple raw queries keep the latency sub‑millisecond.

Tool Implementation: File Search, CLI Execution, Window Control

Each tool is a coroutine that receives a dictionary, performs the action, and returns a JSON‑serializable response.

# tools.py
import asyncio, json, pathlib, re
from typing import Dict

async def tool_file_search(params: Dict) -> Dict:
    pattern = params.get("glob", "*")
    root = pathlib.Path(params.get("root", str(Path.home())))
    matches = [str(p) for p in root.rglob(pattern)][:10]
    return {"matches": matches