ComfyUI-AIAssistant

Let a local AI assistant see what is selected on your ComfyUI canvas — and, on explicit request, insert text into it.

Every browser tab publishes a bounded snapshot of the current node selection to the ComfyUI process over a same-origin WebSocket. The snapshot lives in RAM and is served to any local consumer over HTTP or MCP. One narrow write tool, set_widget_text, can insert text into a text-like widget of the selected node, guarded by concurrency checks.

No chat UI, no AI service calls, no workflow nodes, no runtime dependencies.

Highlights

• Selection context for agents — ask "how should I configure the selected node?" and get an answer grounded in the live widget values, input/output slots and connections.

• MCP server built in — one URL, no extra process: POST /mcp speaks a stateless subset of MCP Streamable HTTP (verified live with Claude Code and OpenCode).

• Multi-tab aware — every tab is a page in a bounded registry; the active page follows window focus and carries a label (AI-XXXX) shown at the top of the tab, so you and the agent always name the same tab.

• Read-only by design, one narrow carve-out — the only mutation the plugin can perform is writing text into one text-like widget of the selected node on the active page, on an explicit MCP request, guarded by revision and page-identity checks. On the canvas the edit behaves exactly like a manual widget edit: undo works, the workflow becomes dirty.

• Private by construction — RAM-only storage, no persistence, no telemetry, no outbound network requests, no payload logging; secret-like widget names are redacted before the snapshot leaves the page.

Related MCP server: ComfyUI MCP Server

Requirements

• ComfyUI with the current frontend (developed and verified against ComfyUI 0.33.3, frontend 1.50.6, aiohttp 3.14, Python 3.13).

• Python ≥ 3.10.

• No additional Python or Node packages.

Install

cd <ComfyUI>/custom_nodes
git clone https://github.com/alksef/ComfyUI-AIAssistant.git

Restart ComfyUI. Once the frontend loads, each tab shows its page label (for example AI-6DEF) at the top center of the window — that label identifies the tab in the context envelope below.

Connect an MCP client

The endpoint is http://127.0.0.1:8188/mcp — the ComfyUI port, Streamable HTTP transport, stateless. Adjust the host/port if your ComfyUI listens elsewhere.

Claude Code

claude mcp add --transport http comfyui http://127.0.0.1:8188/mcp

Then ask: "what is selected in ComfyUI?" — Claude reads it through get_selection.

OpenCode

In opencode.json:

{
  "mcp": {
    "comfyui": {
      "type": "remote",
      "url": "http://127.0.0.1:8188/mcp"
    }
  }
}

Any MCP client (or plain curl)

curl -s http://127.0.0.1:8188/mcp \
  -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call",
       "params":{"name":"get_selection","arguments":{}}}'

Supported methods: initialize, tools/list, tools/call, ping. Notifications get 202 Accepted; GET/DELETE get 405 Allow: POST; bodies are capped at 64 KiB. No sessions, no SSE — one JSON-RPC message per POST.

Tools

get_selection

Read-only, no arguments. Returns the context envelope (below) as compact JSON in content[0].text. An absent snapshot (available: false) is a normal result: no tab has published yet.

set_widget_text

Inserts text into widget of the currently selected node on the active page. All four parameters are required:

The revision and page checks make writes safe against a stale plan: if the user edited something or switched tabs between your read and your write, the call is refused with the current revision so you can re-read and retry.

Results (content[0].text, compact JSON):

• success — {"status":"applied","revision":<new>,"widget":<name>} when the page confirmed the edit within the bounded wait (≈3 s), or {"status":"queued",...} when the command was dispatched but no confirmation arrived in time;

• refusal (isError: true) — invalid params …, no selection available, no node selected, multiple nodes selected, unknown widget, widget is not text-like, no active page, active page changed, revision mismatch (the last two carry current_revision for a cheap retry).

Only text-like widgets (type text, customtext or string) of a singly selected node can be written. Nothing else mutates: no queue, no graph structure, no settings, no files.

HTTP API (for scripts and debugging)

All responses carry Cache-Control: no-store. Bodies are capped at 128 KiB.

• GET /ai-assistant/context — the context envelope. Poll this per question if your consumer is not an MCP client.

• POST /ai-assistant/context — anonymous snapshot channel: submit a normalized snapshot object (schema_version, captured_at, revision, workflow, selection) exactly as the frontend builds it; useful for testing consumers without a browser. Does not create registry pages.

• WS /ai-assistant/ws — the frontend channel (below).

Context envelope

{
  "schema_version": "comfyui.ai-assistant.context/1",
  "available": true,
  "received_at": "2026-08-29T12:00:00.123456+00:00",
  "snapshot": {
    "schema_version": "comfyui.ai-assistant.context/1",
    "captured_at": "2026-08-29T12:00:00.120000+00:00",
    "revision": 7,
    "workflow": {
      "id": "9",
      "title": "ace_step remix",
      "selected_count": 1,
      "selection_detail": "full",
      "selected_limit": 1,
      "selection_truncated": false
    },
    "selection": [
      {
        "id": "107",
        "comfyClass": "CLIPTextEncode",
        "title": "Song Tags",
        "type": "CLIPTextEncode",
        "mode": 0,
        "widgets": [
          { "name": "text", "type": "text", "value": "K-Pop Girl Group, …",
            "truncated": false, "redacted": false, "unsupported": false }
        ],
        "inputs": [
          { "name": "clip", "type": "CLIP",
            "link": { "link_id": "42", "origin_id": "11", "origin_slot": 0,
                      "target_id": "107", "target_slot": 0, "type": "CLIP" } }
        ],
        "outputs": [
          { "name": "CONDITIONING", "type": "CONDITIONING", "links": [] }
        ]
      }
    ]
  },
  "pages": [
    { "page_id": "6def0a1b-2c3d-4e5f-8a9b-0c1d2e3f4a5b",
      "page_label": "AI-6DEF", "workflow_name": "ace_step remix",
      "connected": true }
  ],
  "active_page": { "page_id": "6def0a1b-…", "page_label": "AI-6DEF", "connected": true }
}

Notes:

• selection_detail is full for a single selection, names_only for a multiple selection (bounded node names, no ids/params/widgets/links), none for an empty selection — an empty selection is a valid snapshot.

• revision grows monotonically per page on every published change; use it with page_id for the set_widget_text checks.

• Values are bounded and normalized: strings over 512 chars truncate to <truncated>, non-JSON shapes become <unsupported>, and widget names containing terms like password, token, api_key are reported with "<redacted>" values.

• active_page is the most recently focused connected tab; when it disconnects, the next remaining tab takes over immediately (socket liveness — no timeouts).

WebSocket channel (frontend ↔ server)

JSON text frames, one object per frame. The frontend sends register ({type, page_id}), snapshot ({type, page_id, snapshot} — the normalized snapshot object), activate ({type, page_id}, on focus and visibility change), and receives registered / accepted (with the revision) / activated acks, error frames that never close the socket, and command frames {type: "command", command_id, op: "set_widget_text", widget, text} carrying the write. The registry holds at most 8 pages; disconnected pages are evicted first.

Security and privacy boundary

• The plugin stores snapshots only in process memory and never writes them to disk, never logs them, never echoes them anywhere except to the asking consumer.

• The plugin makes no outbound network requests. The WebSocket is same-origin inbound; consumers pull context.

• Secret-like widget names are redacted in the frontend before the snapshot leaves the page; values never travel for them.

• The endpoints ride the ComfyUI port and are reachable wherever ComfyUI is reachable — including LAN exposure if you run ComfyUI that way. They expose exactly the bounded selection snapshot and the one guarded write; there is no authentication, so treat the port as trusted (default: localhost).

• The only mutation surface is the set_widget_text carve-out described above. AGENTS.md records this boundary for contributors.

Limitations

• Writes target text-like widgets only, a single selection only, the active page only.

• "queued" means dispatched without confirmation in the bounded wait — check the revision or re-read before retrying; a concurrent manual edit can also advance the revision and be reported as "applied".

• The MCP subset is stateless by design: no sessions, no SSE streams, no server-initiated notifications, no batching.

Development

python -m unittest discover -s tests -v   # backend (needs aiohttp)
ruff check ai_assistant tests             # lint
ruff format --check ai_assistant tests    # format
npm test                                  # frontend (node:test)

The backend core (pages, commands, mcp_protocol) is stdlib-only and fully testable offline; server.py is covered through aiohttp test utilities. Design decisions and their history live in docs/roadmaps/.

License

MIT — see LICENSE.