Desktop-MCP

Visible, supervised Windows desktop control, built on Windows-MCP.

An MCP client supplies the model and decides what to do. Desktop-MCP supplies the real mouse, keyboard, screenshots, and a local control window. It is not another AI model, a remote-desktop service, or a sandbox.

Instructions for the connected model

AGENT_GUIDE.md is the canonical operating explanation for agents. The server includes its contents directly in MCP initialization instructions and also serves it as desktop-mcp://guide. The guide ships inside the package; clients do not need access to this repository.

Loading an MCP server does not automatically load its repository's Markdown files. This connection-level guide covers the normal observe/act workflow, combined teaching/control, transcript listening and replies, visibility and local authorization. You can ask for the desktop task normally rather than having to restate its tool-call procedure. The client decides how it presents MCP instructions to the model; documentation is not a guarantee of model compliance.

After installing this update, reopen Desktop-MCP and reconnect existing MCP clients to receive the guide. New server connections read the packaged guide, so there is no separate hardcoded copy to keep synchronized.

Related MCP server: atomic-computer-mcp

What changes from stock Windows-MCP

• During automated input, a rounded black/grey arrow overlay follows real pointer movement. Pointer moves, including movement before clicks, accelerate and decelerate smoothly.

• Left, right, middle and extra mouse buttons; modifier-aware drags; native horizontal/vertical wheel input; named keys, chords, repeats and batch-scoped key/button holds.

• Fast Unicode typing without an artificial per-character speed limit or clipboard replacement. Long input stays cancellable.

• An Alt-Tab-accessible control window and a global Ctrl+Shift+H stop. Control starts stopped and can only be allowed/resumed locally.

• One shared local application for multiple Copilot sessions. Closing either main window with X quits the application, not just its taskbar window.

• Short serial input batches with a single final observation, instead of a model round trip for every key.

• Cropped, resized, efficiently encoded observations with frame IDs and server-side coordinate conversion. Bounded adaptive waits detect changes without continuously sending redundant screenshots.

• Optional local image files for clients whose native image reader works but whose MCP image-result forwarding does not.

• Teaching and control in one armed session: explain, circle a control, click the next tab, and keep explaining without changing modes.

• A draggable, Alt-Tab-accessible transcript with local pin and top/bottom docking. Type questions into its message box and receive agent replies in the same window. The main panel has a Transcript: On/Off toggle.

Requirements and installation

Use an interactive Windows 10/11 desktop and Python 3.14+. The package metadata, not old upstream installation guides, is authoritative.

In PowerShell, inside this checkout:

# If uv is missing:
python -m pip install --user uv

python -m uv sync --frozen --extra dev
.\.venv\Scripts\desktop-mcp.exe install-shortcut

UV can install the required Python interpreter into its managed environment. Dependencies remain in this project's .venv.

This project is local-first. Do not assume uvx desktop-mcp installs this fork: PyPI and MCP registry publication are not part of setting up the checkout. server.json is release metadata, not a publication receipt.

Open it from Windows Start

Press the Windows key and search for Desktop-MCP. The per-user shortcut opens the interface without a console, or reveals the existing instance. It does not arm desktop access.

install-shortcut above is a one-time setup step; a Git checkout or a Python console command does not automatically appear in Windows Start search. It creates only the Desktop-MCP shortcut and its application icon.

From the project folder you can also run:

.\.venv\Scripts\desktop-mcp.exe open

Opening the interface before Copilot is fine. Starting more Copilot sessions connects them to the same application rather than spawning competing windows and global-hotkey registrations.

Connect to Copilot CLI

From the project folder:

copilot.cmd mcp add desktop-mcp --timeout 45000 -- "$((Get-Location).Path)\.venv\Scripts\python.exe" -m desktop_mcp serve

For an npm-installed Copilot on Windows, use copilot.cmd: the PowerShell shim can consume -- and then misinterpret Python's -m. A native executable installation can use copilot with the same arguments.

Alternatively, use /mcp add inside Copilot and enter:

The equivalent configuration is:

{
  "mcpServers": {
    "desktop-mcp": {
      "type": "local",
      "command": "C:\\path\\Desktop-MCP\\.venv\\Scripts\\python.exe",
      "args": ["-m", "desktop_mcp", "serve"],
      "timeout": 45000,
      "tools": ["*"]
    }
  }
}

Use the virtual environment's absolute Python path so the connection does not depend on the client's working directory or PATH. serve is now a small stdio bridge. A single shared host owns the native windows and Ctrl+Shift+H; several Copilot sessions can connect without competing for that hotkey.

If loading fails, open Desktop-MCP from Start, then use Copilot's /mcp panel to reconnect the server (disable/enable it if needed). A session created before the configuration was installed may need a fresh Copilot session.

.\.venv\Scripts\desktop-mcp.exe doctor

doctor reports whether the shared host is running and its latest startup or explicit-quit state; it never starts or arms desktop control. The first cold startup can take longer than subsequent connections, so the example allows 45 seconds. Do not respond to a loading problem by disabling the local stop gate. Some MCP clients deliberately put their tools in a Windows job that forbids independent child applications. In that case, open Desktop-MCP from Start first, then connect Copilot. The bridge reports this explicitly rather than starting a shared host that would be killed when the first client exits.

Clients still use STDIO. The bridge uses a Windows named pipe restricted to the current account and interactive session; remote pipe clients are rejected. No TCP listener, firewall rule, administrator elevation, or login startup task is required. All connected clients share the same local Arm/Stop state; connect only clients you intend to give desktop access.

Start, stop and take over

The application and transcript open together. Desktop access starts stopped. Text conversation and the transcript toggle work even while desktop control is paused. Press Arm / Resume locally when you want screenshots or desktop actions. That single authorization enables guidance, observations and desktop input together; there is no Control/Teach selector. The panel minimizes so it does not intercept input; it remains reachable through Alt-Tab. If Windows activates the transcript instead of the target app during local minimization, the panel returns to the last non-Desktop-MCP window. It does not override a different app selected by the user. The panel compacts to fit the current monitor's work area without changing the physical pointer scale. Native accessibility text exposes takeover On/Off, arm-rejection details and current activity, rather than relying only on painted text.

Ctrl+Shift+H and the panel's Stop control revoke input, captures and annotations. Text conversation stays available so you can ask questions or explain a problem. Pending commands from the old generation stay cancelled even after you resume. Keys and buttons held by Desktop-MCP are released. The model has no Arm or Resume tool. X on either the control window or the instruction window quits Desktop-MCP: input is revoked, both windows/overlays and the hotkey are released, and the application and connected bridges exit. Use the ordinary minimize button if you want it to remain running.

After an explicit Quit, automatic MCP reconnects do not reopen the app behind your back. Open Desktop-MCP from Start, then reconnect it in /mcp. The new instance starts stopped. Closing a Copilot terminal alone leaves the desktop application available; disconnecting a client that used the desktop revokes access until local re-arming.

Pause on interruption stops an active automated input sequence when you use the mouse or keyboard. Moving while reading instructions or while the assistant waits for your cursor does not require another Arm click. The local checkbox can disable interruption pauses; the emergency hotkey always remains enabled. Physical clicks/keys still invalidate prior observations.

The boundary is this server. The hotkey does not terminate Copilot, revoke its shell tools, stop another MCP server, undo completed actions, or erase information already delivered to a model. Do not work around a stop using other tools. Normal Windows integrity restrictions still apply; input to an elevated or locked desktop may be refused.

Tool surface

Upstream PowerShell, registry, filesystem, process-killing and network-scraping tools are deliberately not registered. The retained python -m windows_mcp module is the upstream implementation, not an alternative supervised connection. Both installed console aliases, desktop-mcp and windows-mcp, launch the supervised entry point.

Explain and act in the same session

Arm once. The assistant can publish instructions, highlight a button, operate it, and publish the next explanation using ordinary tool calls. It can also wait for you to try a step before continuing. No mode switches or extra authorization are needed unless you stop or interrupt an active automated input sequence.

The transcript opens with the application. Its message box is separate from the read-only conversation history: Enter sends, Shift+Enter adds a line, and the Send button also works with input methods such as IME. Drag its title bar, use Top/Bottom to dock, or Pin to keep it above other windows. A model Transcript(action="back") request cannot override a local pin. Use Transcript: On/Off in the main panel to show or hide it without deleting messages or changing desktop permissions. The model can use Transcript(action="show") or "hide" directly; it does not need to Alt+Tab and click protected application controls. DesktopStatus.transcript reports enabled, actual visible, pending-message count and listener status.

Closing either main window quits the application. Minimize the instruction window instead when you just want it out of the way.

Have a conversation here

Start a task in Copilot with a prompt such as:

Use Desktop-MCP to teach me Blender. Put explanations and replies in the transcript, and keep using TranscriptRead to listen for my questions until I say we're done.

Then write in the transcript and press Enter or Send. The active agent receives the message through TranscriptRead, answers with Transcript(reply_to=...), and listens again. You do not need to return to the terminal for each question. Only one MCP session listens at a time, so unrelated Copilot tabs cannot both consume the same question. It can release its listener explicitly; disconnection releases it automatically. A silent listener lease expires after two minutes.

The status line distinguishes Agent listening, Awaiting reply and Queued. A queued message is retained until answered, not silently discarded. The app does not contain its own AI and cannot wake a completely idle/disconnected Copilot model. Keep the Copilot task active (Autopilot can do that); if nobody is listening, ask the intended Copilot session to use the transcript.

Chat is bounded and held in memory: up to 32 displayed entries and 32 unanswered messages, with 16,000 characters per message. A full queue reports an error and keeps your draft. Closing Desktop-MCP clears its local chat; the connected Copilot client may retain its normal session history.

An agent can publish a step, mark the relevant area, and wait for your pointer:

{"text": "Move your cursor over the Add menu.", "title": "Next step"}

Send that to Transcript; use Laser(bounds=[left,top,right,bottom], frame_id=...) to circle the area in an observed image, or Draw for persistent paths, rectangles and ellipses. Coordinates without frame_id are physical desktop pixels. These marks are separate click-through visual layers; they never move your real pointer or modify Blender. Erase and the local Clear ink button remove only Desktop-MCP marks.

The combined ink/laser/wait canvas is limited to 8,192 pixels per side and 16,777,216 pixels total. Oversized combinations are rejected before publication; erase older marks before guiding across widely separated monitors.

WaitForCursor automatically gives you a turn. Its radius is physical pixels; dwell requires continuously staying nearby. It returns reached, timeout, context_changed or input_changed; being nearby is not proof of a click or successful app action. A stop cancels the operation rather than returning a false success. During that bounded wait, automated input cannot take your pointer away; afterward input tools are available again if access is still armed. Marks disappear when their context becomes stale or control stops.

The transcript is not automatic mirroring of every Copilot CLI token. The model uses TranscriptRead and Transcript to receive and answer messages. Ink, laser, cursor and control windows are excluded from server screenshots so guidance does not feed back into the model's view of the application.

Use frames, not guessed coordinate math

Screenshot defaults to the active application. Use scope="desktop" for the full desktop, or supply an explicit physical-pixel region=[left,top,right,bottom]. Its response contains a real MCP image block, capture/image dimensions and a frame_id.

When clicking a point measured in that image:

{
  "loc": [340, 210],
  "frame_id": "<the Screenshot frame_id>"
}

The server converts image pixels using the actual image dimensions and capture origin, including negative monitor origins and independently rounded x/y scales. Do not multiply coordinates yourself when supplying frame_id.

Without frame_id, coordinates are explicitly physical virtual-desktop pixels. Frame references expire, are bounded in memory, and are rejected after input changes or relevant window/display geometry changes. A reference is not an eternal guarantee that an application has not redrawn its own contents. Teaching tools carry the input-revision ticket through coordinate mapping and annotation/wait authorization; a learner click cannot silently refresh an old frame.

Coordinate-bound batches guard the observed foreground window. If an action opens a new dialog or switches applications, use the returned fresh observation before deciding the next coordinate-based action.

Snapshot binds its accessibility tree and image to the same context and input revision. A switch or input change during the compound inspection is an error, not a tree from one window paired with another window's image.

Faster observations and actions

MCP is a request/response protocol, not a video stream into a model. The efficient loop is:

1. Observe at a decision point.

2. Send a short batch of already-understood actions.

3. Receive one fresh observation.

4. When waiting for rendering, use Screenshot(since=..., wait_for_change=...) instead of repeatedly asking the model to poll.

An unchanged image can be omitted while fresh frame metadata is returned, explicitly referencing the prior image. Use since only when the caller already has that image; omit it when starting a new agent/context.

The service adapts its polling interval within a bounded wait, briefly settles changed frames, and encodes only the observation it returns. Crop deliberately and tune max_dimension, encoding and quality instead of capturing a giant desktop for a small dialog. Timing and encoded-size metadata describe actual work; these choices do not remove model inference latency.

Automatic capture prefers the verified one-shot DXCAM path, then MSS/Pillow. Display access loss falls back instead of retrying DXCAM recovery indefinitely. An unverified DXCAM version is skipped until its recovery behavior is checked; normal native capture calls still depend on Windows returning promptly.

A short batch in an already-focused blank editor:

{
  "actions": [
    {"kind": "text", "text": "Hello from Desktop-MCP."},
    {"kind": "key", "keys": ["enter"]}
  ],
  "observe": true
}

Batch kinds also include move, click, drag, scroll, wait, key_down, key_up, button_down and button_up. A hold lasts only within that batch and is always released at its end. Use keys as mouse modifiers, for example {"kind":"drag","button":"middle","keys":["shift"],"start":[100,100],"loc":[300,200]}. Do not enter literal text while batch-held modifier keys are down.

Movement uses a minimum-jerk curve: zero initial/final velocity and acceleration, without overshooting a click target. Its default duration adapts to distance; an explicit positive duration can slow a demonstration. Text has no corresponding speed cap. Requests below 80 ms use an 80 ms pointer-motion minimum so approaches and drags still contain visible acceleration/deceleration steps. Any action that moves to loc requires a positive explicit duration; zero-length waits remain valid.

Do not automatically replay a failed input request. The error identifies how many complete steps ran; the current step can be partially applied. An error from the observation after a successful batch explicitly says the input already completed.

If your client cannot see MCP images

Successful MCP negotiation does not prove a client forwards image pixels to its model. Screenshot(export_image=true) returns both the normal image block and an image_path that a native image-reading tool can open. Omit since when requesting a full exported image.

For a client needing this regularly, set DESKTOP_MCP_IMAGE_FILES=true in its MCP environment configuration. Full observations then include a temporary file as well. This costs disk I/O and possibly another tool round trip, so it is a compatibility path, not the fastest default.

Images remain in memory by default. Explicit exports are private screen content on disk, retained among the latest 16 exports until server exit. They are never committed. A local server does not make Copilot/model processing offline; only show applications whose content you intend to share with your model service.

Development and documentation

See CLAUDE.md, SYSTEM_MEMORY.md, DECISIONS.md, agent-work.md, and TESTING.md for the relevant contracts and safe development workflow.

The preserved Windows engine is in src/windows_mcp. Supervision, the native interface, observation service and explicit MCP surface live in src/desktop_mcp. The repository preserves upstream history; upstream points to Windows-MCP and origin points to this fork.

MIT terms are in LICENSE.md. Bundled UIAutomation attribution and Apache 2.0 terms are preserved in THIRD-PARTY-NOTICES.md and LICENSE-UIAUTOMATION.txt.