desktop-touch-mcp
desktop-touch-mcp is a Windows-only MCP server that gives Claude eyes and hands on a Windows desktop via 46 specialized tools, optimized for LLM efficiency (minimal tokens and round-trips).
Note: All tools return
UnsupportedPlatformwhen running on Linux. Capabilities below apply when hosted on Windows 10/11 (64-bit).
Screen Capture & Vision
Screenshots with 3 detail levels:
image(~443 tokens),text(~100–300 tokens),meta(~20 tokens)Diff mode — only sends changed windows after first frame (60–80% token reduction)
Background window capture without stealing focus
OCR text extraction with click coordinates (Windows.Media.Ocr)
Full-page scroll capture with overlap detection
dotByDot=truefor pixel-perfect 1:1 coordinate clicking
Mouse & Keyboard Control
Move, click, drag with speed control and homing correction
Scroll in any direction; get cursor position
Image-local coordinate translation via
origin+scaleType text (clipboard bypass for non-ASCII/IME/CJK) and send key combos
forceFocusto bypass Windows foreground-stealing protection
Window Management
List all windows in Z-order; get active window info
Focus by partial title match; pin/unpin (always-on-top); dock to screen corners
UI Automation (UIA)
Extract full UI element trees as JSON with click coordinates
Click elements by name or automationId (no coordinates needed)
Set text field values directly; zoom/crop to specific elements
Automatic fallback from UIA to OCR for Chromium apps
Browser Automation via CDP (Chrome/Edge/Brave)
Launch, connect, navigate browsers via Chrome DevTools Protocol
Find/click elements by CSS selector; evaluate JavaScript; get DOM/outerHTML
Enumerate interactive elements with ARIA states
Extract SPA state (Next.js, Nuxt, Remix, Apollo, Redux, GitHub react-app)
Search DOM by text, regex, role, or ariaLabel with confidence ranking
Workspace & Macro Operations
Full workspace snapshot: all windows with thumbnails and UI summaries
Launch applications with auto-detection of new windows
run_macro: batch up to 50 sequential operations in a single API call
Event System & Terminal
Subscribe to, poll, and wait on desktop events (
wait_until)Read from and send input to terminals
Safety & Security
Emergency stop: move mouse to top-left corner (within 10px of 0,0)
Blocklists for dangerous executables and keyboard shortcuts
PowerShell injection protection; configurable allowlists for app launching
Provides browser automation capabilities through CDP (Chrome DevTools Protocol), enabling connection to Brave browser, finding elements via CSS selectors, clicking elements, evaluating JavaScript, and navigating pages without requiring Selenium or Playwright.
Supports CSS selector-based element location in browser automation, allowing precise identification of DOM elements in Chrome/Edge/Brave browsers for clicking and interaction through CDP integration.
Provides comprehensive browser automation through CDP (Chrome DevTools Protocol), enabling connection to Chrome, finding elements via CSS selectors, clicking elements, evaluating JavaScript, getting DOM content, and navigating pages without requiring Selenium or Playwright.
desktop-touch-mcp
Computer-use MCP server for Windows. Lets Claude, Cursor, or any MCP client see and operate your Windows 10/11 desktop — screenshots, UI Automation, Chrome CDP, keyboard / mouse, terminal — with semantic discover-then-act targeting that avoids pixel-coordinate guessing, and per-action perception guards that catch wrong-window typing before it happens.
npx -y @harusame64/desktop-touch-mcp32 tools, native Rust engine (UIA in 2 ms), zero-config PowerShell fallback, full CJK support, MIT licensed. Add the snippet above to your Claude / Cursor / VS Code Copilot config and Claude can drive Notepad, Excel, Chrome, Windows Terminal, and any other app on your machine.
Why this over pixel-clicking? Two ideas run through every tool: discover-then-act —
desktop_discoverreturns interactive entities with short-lived leases instead of raw coordinates, sodesktop_actoperates on what you mean, not where it was — and per-action perception guards that verify the target window's identity and bounds before input lands, catching wrong-window typing and stale-coordinate clicks before they happen.Under the hood: an 82× average speedup from the Rust native engine (UIA focus queries in 2 ms, SSE2-accelerated image diffing at 13–15×), with a transparent PowerShell fallback when the engine is absent. The npm launcher fetches only the GitHub Release tag matching the installed version and verifies the Windows runtime zip before extraction.
Features
⚡ High-performance Rust Native Core — The UIA bridge and image-diff engine are written in Rust (
napi-rs+windows-rs) and loaded as a native.nodeaddon. Direct COM calls from a dedicated MTA thread eliminate PowerShell process spawning —getFocusedElementcompletes in 2 ms (160× faster), andgetUiElementsreturns full trees in ~100 ms with a batch BFS algorithm that minimizes cross-process RPC. Image-diff operations use SSE2 SIMD for 13–15× throughput. When the native engine is unavailable, every function transparently falls back to PowerShell — zero config required.🎯 Set-of-Marks (SoM) visual fallback — Games, RDP sessions, and non-accessible Electron apps return clickable elements even when UIA is completely blind.
screenshot(detail="text")automatically detects UIA sparsity and activates a Hybrid Non-CDP pipeline: Rust-powered grayscale + bilinear upscale → Windows OCR → clustering → red bounding-box annotation with numbered badges ([1],[2]…). Two parallel representations returned: a visual PNG for spatial orientation and a semanticelements[]list withclickAtcoords — no CDP required.🔁 One-call confirmation on visual-only targets — On UIA-blind targets (Electron, PWAs, games, custom canvases, RDP windows),
desktop_actcan fold the post-action confirmation into its own response: an optionalroiCapturecarrying a PNG crop of just the region that changed plus a lease-less preview of the controls now visible there. The agent confirms what its click did and finds the next target without a separatedesktop_state+screenshot. On visual-only targets it is on by default for a visible change (returnCapture:"on-change"); passreturnCapture:"never"to suppress it, or"always"to force it. Never attached on structured targets (browser/CDP, UIA-rich native), wheredesktop_stateis cheaper and exact — so those responses are unchanged.🔐 Key Locker — the terminal autofills your SSH / sudo passwords — Save a credential once into the locker's own secure dialog (stored encrypted on your machine with Windows DPAPI; never shown to the assistant), then run
ssh/sudoin a console opened bykey_locker(action='launch_console')— the password is filled in automatically when the hidden prompt appears, with a per-fill confirmation prompt by default. See Key Locker.LLM-native design — Built around how LLMs think, not how humans click.
run_macrobatches multiple operations into a single API call;diffModesends only the windows that changed since the last frame. Minimal tokens, minimal round-trips.Reactive Perception Graph — Register a
lensIdfor a window or browser tab, pass it to action tools, and get guard-checkedpost.perceptionfeedback after each action. It reduces repeatedscreenshot/desktop_statecalls and prevents wrong-window typing or stale-coordinate clicks.Full CJK support — Uses Win32
GetWindowTextWfor window titles, avoiding nut-js garbling. IME bypass input supported for Japanese/Chinese/Korean environments.3-tier token reduction —
detail="image"(~443 tok) /detail="text"(~100–300 tok) /diffMode=true(~160 tok). Send pixels only when you actually need to see them.1:1 coordinate mode —
dotByDot=truecaptures at native resolution (WebP). Image pixel = screen coordinate — no scale math needed. Withorigin+scalepassed tomouse_click, the server converts coords for you — eliminating off-by-one / scale bugs.Browser capture data reduction —
grayscale=true(~50% size),dotByDotMaxDimension=1280(auto-scaled with coord preservation), andwindowTitle + regionsub-crops help exclude browser chrome and other irrelevant pixels. Typical reduction for heavy captures: 50–70%.Chromium smart fallback —
detail="text"on Chrome/Edge/Brave auto-skips UIA (prohibitively slow there) and runs Windows OCR.hints.chromiumGuard+hints.ocrFallbackFiredflag the path taken.UIA element extraction —
detail="text"returns button names andclickAtcoords as JSON. Claude can click the right element without ever looking at a screenshot.Auto-dock CLI —
window_dock(action='dock')snaps any window to a screen corner with always-on-top. SetDESKTOP_TOUCH_DOCK_TITLE='@parent'to auto-dock the terminal hosting Claude on MCP startup — the process-tree walker finds the right window regardless of title.Emergency stop (Failsafe) — Park the mouse in the top-left corner of the primary monitor (within 10px of 0,0) for 500ms to trigger the emergency stop.
Related MCP server: pywinauto-mcp
Requirements
OS | Windows 10 / 11 (64-bit) |
Node.js | v20+ recommended (tested on v22+) |
PowerShell | 5.1+ (bundled with Windows) — used only as fallback when the Rust native engine is unavailable |
Claude CLI |
|
Note: nut-js native bindings require the Visual C++ Redistributable. Download from Microsoft if not already installed.
Note (Key Locker): The credential helper Key Locker uses is an unsigned executable, so on some machines Windows SmartScreen or antivirus may show an "unknown publisher" warning the first time it runs. This is expected — the helper ships with desktop-touch-mcp and runs locally on your machine; you can allow it to proceed. Code signing is planned for a future release.
Installation
npx -y @harusame64/desktop-touch-mcpThe npm launcher resolves runtime strictly by npm package version. For package X.Y.Z, it fetches only GitHub Release tag vX.Y.Z, downloads desktop-touch-mcp-windows.zip, verifies its SHA256 digest, and only then expands it under %USERPROFILE%\.desktop-touch-mcp. Verified cached releases are reused on later runs.
Set DESKTOP_TOUCH_MCP_HOME to override the cache root directory.
On a shared or CI network? The first run reads the GitHub Releases API to locate the runtime zip. The anonymous limit is 60 requests/hour per IP, which a shared public address (CI runners, office NAT) can exhaust before your download even starts. Set
GITHUB_TOKEN(orGH_TOKEN) in the environment and the launcher authenticates the request, raising the limit to 5,000 requests/hour. No token is needed on an ordinary home connection.
Running the launcher from a source checkout? A source build's
bin/launcher.jscarries a placeholder integrity hash (sha256: "PENDING") instead of a finalized one. Rather than download and run an unverified runtime, the launcher fails closed — this guard stops an accidentally published or unfinalized launcher from silently starting unverified code. Published npm releases always ship a real SHA256, so end users never see this. If you are intentionally running the launcher from source, setDESKTOP_TOUCH_MCP_ALLOW_UNVERIFIED=1to skip integrity verification (development only).
Does your host give up before the launcher finishes? Some desktop hosts allow a plugin a fixed budget — 60 seconds is common — to become ready, and a launcher waiting on an unreachable GitHub can spend all of it. Two environment variables cover that case.
DESKTOP_TOUCH_MCP_FETCH_TIMEOUT_MS(default15000) bounds how long the launcher waits without hearing from GitHub. It applies to the release lookup and to the download; for the download it counts silence rather than total time, so a large runtime still installs over a slow connection. A value that is not a positive number of milliseconds is ignored with a warning.
DESKTOP_TOUCH_MCP_OFFLINE_FALLBACK=1lets the launcher start a release that is already installed when GitHub cannot be reached at all. It is off by default. GitHub is always contacted first, so a reachable network still re-downloads and repairs a damaged install; only a network failure reaches the fallback, which starts the copy of your version on disk — without re-verification — or, when that version was never installed, the newest older release that completed a verified install. Answers that are not network failures (a 404, the API rate limit, a mismatched integrity hash) still stop startup loudly. Leave it off unless a host timeout forces your hand: while it is set, a corrupted install of your current version is reused instead of being repaired.The two work together: with the fallback on, startup still waits out the timeout before falling back, so lower
DESKTOP_TOUCH_MCP_FETCH_TIMEOUT_MSif your host's budget is tight. Note also that a download which is still arriving, however slowly, is never interrupted — the fallback answers when the network has gone silent, not when it is merely slow.
Register with Claude CLI
Add to ~/.claude.json under mcpServers:
{
"mcpServers": {
"desktop-touch": {
"type": "stdio",
"command": "npx",
"args": ["-y", "@harusame64/desktop-touch-mcp"]
}
}
}No system prompt needed. The command reference is automatically injected into Claude via the MCP initialize response's instructions field.
Register with other clients (HTTP mode)
Clients that require an HTTP endpoint (GPT Desktop, VS Code Copilot, Cursor, etc.) can use the built-in Streamable HTTP transport:
npx -y @harusame64/desktop-touch-mcp --http
# or with a custom port:
npx -y @harusame64/desktop-touch-mcp --http --port 8080The server starts at http://127.0.0.1:23847/mcp (localhost only). Register the URL in your MCP client settings. A health check is available at http://127.0.0.1:<port>/health.
In HTTP mode the system tray icon shows the active URL and provides quick-copy and open-in-browser shortcuts.
Development install
git clone https://github.com/Harusame64/desktop-touch-mcp.git
cd desktop-touch-mcp
npm installBuild after install:
npm run buildFor a local checkout, register the built server directly:
{
"mcpServers": {
"desktop-touch": {
"type": "stdio",
"command": "node",
"args": ["D:/path/to/desktop-touch-mcp/dist/index.js"]
}
}
}Note: Replace
D:/path/to/desktop-touch-mcpwith the actual path where you cloned this repository.
Tools (32 Optimized Tools)
📖 Full Reference:
docs/system-overview.md— Exhaustive guide on parameters, return schemas, and coordinate math.
🌐 World-Graph V2 (Primary Path)
Tool | Description |
| Observe the desktop. Returns interactive entities with leases (UIA, CDP, Terminal, Visual SoM). |
| Perform actions (click, type, drag, select) on entities via lease validation. Returns semantic diffs — plus an optional |
👁️ Observation & State
Tool | Description |
| Lightweight check of focus, active window, cursor, and Auto-Perception attention signal. |
| Multi-mode capture: |
| Inspect and prune the on-disk screenshot cache behind the by-ref links: |
| Instant session orientation: all window thumbnails + UI summaries in one call. |
| Diagnostic check for native engine health and feature activation. |
⌨️ Input & Control
Tool | Description |
| Send keyboard input. Supports background input (WM_CHAR) and IME-safe clipboard bypass. |
| Precision coordinate-based interaction with homing and force-focus protection. |
| Multi-strategy: |
| Legacy UIA-based click by name/ID (fallback when entities are unavailable). |
🌐 Browser CDP (Chrome/Edge/Brave)
Tool | Description |
| Idempotent debug-mode launch and reliable navigation. |
| High-level DOM interaction stable across repaints and framework re-renders. |
| Deep inspection via |
| Semantic discovery, grep-like DOM search, and pixel-accurate coordinate lookup. |
🛠️ Utilities & Workflow
Tool | Description |
| Unified command execution: |
| Efficient server-side polling for window, focus, text, or URL state changes. |
| Window management: |
| Launch apps and auto-detect new HWNDs (supports localized titles). |
| Batch up to 50 operations into a single round-trip for maximum efficiency. |
| System-level text exchange and user alerts. |
| Manage credentials the terminal autofills for you (SSH key passphrases, sudo / login passwords). Secrets are entered once into the locker's own secure dialog and stored encrypted on this machine (Windows DPAPI); they are never shown to the assistant. |
📊 Office (Excel)
Tool | Description |
| Author and run Excel VBA macros via COM. |
Standard workflow (v1.0.0)
The v2 World-Graph surface (desktop_discover / desktop_act) is the recommended dispatch path. The four-call shape works for native apps, browsers, and terminals identically.
desktop_state → orient: focused window/element, modal, attention signal
desktop_discover → find actionable entities (returns lease + windows[])
desktop_act(lease, …) → act on entity (returns attention + post.perception)
desktop_state → confirm the world changed as expectedClicking — priority order:
browser_click(selector) → Chrome / Edge (CDP, stable across repaints)
desktop_act(lease, action='click') → native / dialog / visual (entity-based; use after desktop_discover)
click_element(name | automationId) → native UIA fallback if desktop_act returns ok:false
mouse_click(x, y, origin?, scale?) → pixel last resort; origin+scale from dotByDot screenshots onlyRecovery hints — read response.attention after every observation and response.warnings[] on desktop_discover / desktop_act. Common reasons:
lease_expired/lease_generation_mismatch/lease_digest_mismatch/entity_not_found→ re-calldesktop_discovermodal_blocking→response.blockingElement(when present) names the blocking modal; dismiss viaclick_element(name=blockingElement.name)then retryentity_outside_viewport→ the element moved off screen:scroll(action='to_element' | 'raw'), or re-calldesktop_discoverif its window moved or closedorigin_window_not_visible→ the element's window is minimised or hidden, so nothing is drawn where it was found:focus_window(windowTitle)to restore it, then re-calldesktop_discovercoordinate_outside_reachable_bounds→ the coordinate is not on any connected monitor. Coordinate-based mouse input (mouse_click/mouse_drag/scroll/browser_click, and the mouse route insidedesktop_act) now works on every monitor, including monitors placed left of or above the primary one, so this error normally means the coordinates are stale — the window moved or closed after they were read. Re-rundesktop_discoverand act on the new coordinates. If the server is running without its built-in Windows input module, mouse input falls back to the primary monitor only; the error message says so, and moving the window onto the primary monitor (or reinstalling the server) is the fixcursor_placement_blocked→ the coordinate is on a monitor, but the pointer could not be placed there, so nothing was clicked. This happens while another app confines the cursor to its own window (common in full-screen games), while a remote-desktop session is disconnected or locked, while another program keeps repositioning the pointer, or right after a monitor is added or removed. Leave the app holding the cursor, reconnect the session, or — after a monitor change — re-rundesktop_discover, then retry.click_elementacts through the accessibility API without moving the cursor and works meanwhileexecutor_failed→ fall back toclick_element/mouse_click/browser_click
Lease lifecycle:
Each
desktop_discoverresponse carriessoftExpiresAtMs(≈ 60 % of the TTL window). Past that timestamp the LLM should consider re-callingdesktop_discovereven though the lease is still technically valid —lease.expiresAtMsis the only correctness wall.TTL adapts to
viewmode (action/explore/debug), entity count, and response payload size. Cap is 60 s.Set
DESKTOP_TOUCH_DISABLE_FUKUWARAI_V2=1to fall back to the v1 tool surface (get_windows/get_ui_elements/set_element_value) for troubleshooting only — V2 is the recommended default.
Terminal command completion (until)
terminal(action='run') sends a command, waits for it to complete, and reads the
output in one call. How it decides "complete" is controlled by until:
Mode | Waits for | Best for |
| output to fall silent for | short interactive commands |
| a string/regex you expect in the output | long commands with a known final marker |
| the command to actually finish | when you need completion or the exit code |
Anchoring caveat (#384): a command whose final line has no trailing newline glues the marker to the next prompt with no line boundary (
printf X→Xuser@host:~$), so an end-anchoredpattern(X\s*\n/X$) can never bind. For completion usemode:'exit'; for content matching use a bare marker (no\n/$).mode:'pattern'also accepts an optionalquietMssettle fallback:until:{mode:'pattern', pattern, quietMs:1000}completes withreason:'quiet'(nomatchedPattern) once output is stable for that long without a match — instead of hanging untiltimeoutMs. It is opt-in (omitquietMsto keep waiting for the pattern; long commands with mid-run silent gaps are unaffected).
until:{mode:'exit'} — real completion + exit code
The heuristic modes can misfire on the common "append a sentinel" idiom
(some-task; echo DONE matched by DONE): the sentinel also shows up in the
echoed command line, and for multi-line commands there is no reliable way to
tell that echo apart from real output. mode:'exit' removes the guesswork — the
server appends its own completion marker whose printed form differs from its
typed form, so it never matches the echoed command (even for multi-line input),
and it returns the real process exit code:
terminal({
action: 'run',
windowTitle: 'pwsh',
input: 'npm run build',
until: { mode: 'exit', shell: 'powershell' },
})
// → completion: { reason: 'exited', exitCode: 0, elapsedMs: … }
// output: just the command's real output (the injected marker is stripped)Pass
shellexplicitly ('bash'or'powershell').shell:'auto'detects the shell from the terminal window, but it cannot see a shell running inside SSH or WSL — the window still looks like its local host — so for remote/nested sessions pass the remote side's shell (autootherwise warns and may pick the outer shell). A window whose process is genuinely unidentifiable (e.g. Windows Terminal) returnsExitModeShellAmbiguous.First-class shells:
bashandpowershell.cmd.exeis not supported yet (ExitModeShellUnsupported).Unsafe input is rejected up front (
ExitModeUnsafeInput) rather than hanging: a command ending mid-construct (unterminated quote, here-doc,$(…), a trailing\or PowerShell backtick).Exit mode controls its own delivery, so delivery-shaping
sendOptions(method/preferClipboard/pressEnter/chunkSize/pasteKey) are rejected withInvalidArgs; focus options remain accepted.
Key Locker (terminal credential autofill)
Running ssh user@host or sudo … normally stops at a hidden password prompt an
assistant can't safely type into. Key Locker stores your SSH key passphrases and
sudo / login passwords encrypted on your machine (Windows DPAPI, current user) and
fills them in automatically when a bound command reaches its prompt. The secret is
typed once into the locker's own secure dialog — it is never shown to the assistant
and never travels through the MCP channel.
// 1. Save the credential once — opens a secure dialog on your desktop
key_locker({ action:'save', uri:'ssh://user@host:22' })
// 2. Open an autofill-capable console (returns its paneId)
key_locker({ action:'launch_console' }) // → { paneId:'12345678', windowTitle:'…' }
// 3. Run the command through that pane — the password is filled at the prompt
terminal({ action:'send', paneId:'12345678', input:'ssh user@host' })Autofill only fires in a console opened by
launch_console— a pre-existing terminal is never autofilled. The console is a classic visible Windows console, so you can watch it and take over at any prompt yourself.Every autofill asks you to confirm by default; opt a binding out with
set_policy.list/status/forgetmanage saved credentials.terminalread/sendacceptpaneIdas an alternative towindowTitle— it targets that exact window even after ansshlogin renames its title.Supported binding URIs:
ssh://user@host:22,sudo://host/user,https-cred://host, and SSH key passphrases (sshkey:SHA256:…). Ansshsave needs the host key already inknown_hosts(connect to the host once first).Windows only. Disable the whole feature with
DESKTOP_TOUCH_DISABLE_KEY_LOCKER=1. The secure dialog is an unsigned helper executable — Windows SmartScreen may show an "unknown publisher" warning on first run (see the note under Requirements).
Browser CDP automation
For web automation, connect Chrome or Edge with the remote debugging port enabled — no Selenium or Playwright needed.
# Launch Chrome in CDP mode
chrome.exe --remote-debugging-port=9222 --user-data-dir=C:\tmp\cdpbrowser_open({launch:{}}) → spawn-if-needed Chrome in debug mode + list tabs (idempotent)
browser_open() → connect-only (fail if no CDP endpoint live)
browser_locate({selector:"#submit"}) → CSS selector → physical screen coords
browser_click({selector:"#submit"}) → find + click in one step (auto-focuses browser)
browser_eval({action:"js", expression:"document.title"}) → evaluate JS, returns result
browser_eval({action:"dom", selector:"#main", maxLength:5000}) → outerHTML, truncated to maxLength chars
browser_eval({action:"appState"}) → one-shot SPA state (Next/Nuxt/Remix/Apollo/GitHub react-app/Redux SSR)
browser_fill({selector:"#email", value:"user@example.com"}) → fill React/Vue/Svelte controlled input (state-safe)
browser_overview() → links/buttons/inputs + ARIA toggles + viewportPosition per element
browser_search({by:"text", pattern:"..."}) → grep DOM with confidence ranking
browser_navigate({url:"https://example.com"}) → navigate via CDP (no address bar interaction)For chained calls in the same tab, pass includeContext:false to omit the activeTab/readyState annotation (~150 tok/call saved). Boolean / object params accept the LLM-friendly string spellings ("true", "{}").
Coordinates returned by browser_locate account for the browser chrome (tab strip + address bar height) and devicePixelRatio, so they can be passed directly to mouse_click without any scaling.
Recommended web workflow:
browser_open({launch:{}}) → browser_eval({action:"dom"}) → browser_locate(selector) → browser_click(selector)Auto-dock CLI on startup
Keep Claude CLI visible while operating other apps full-screen. Set env vars in your MCP config and the docked window auto-snaps into place every MCP startup.
{
"mcpServers": {
"desktop-touch": {
"type": "stdio",
"command": "npx",
"args": ["-y", "@harusame64/desktop-touch-mcp"],
"env": {
"DESKTOP_TOUCH_DOCK_TITLE": "@parent",
"DESKTOP_TOUCH_DOCK_CORNER": "bottom-right",
"DESKTOP_TOUCH_DOCK_WIDTH": "480",
"DESKTOP_TOUCH_DOCK_HEIGHT": "360",
"DESKTOP_TOUCH_DOCK_PIN": "true"
}
}
}
}Env var | Default | Notes |
| (unset = off) |
|
|
|
|
|
| px ( |
|
| Always-on-top toggle |
| primary | Monitor id from |
|
| If true, multiply px values by |
|
| Screen-edge padding (px) |
|
| Max wait for the target window to appear |
Input routing gotcha: when a pinned window is active (e.g. Claude CLI),
keyboard(action='type')/keyboard(action='press')send keys to it, not the app you wanted to type into. Always callfocus_window(title=...)before keyboard operations, then verifyisActive=trueviascreenshot(detail='meta').
Screenshot cache (by-ref storage)
screenshot and the other visual results return a cheap screenshot://by-ref/{id} link to an image saved on disk instead of inlining the pixels every time, so routine look-act-confirm loops cost far fewer tokens. The cache bounds itself automatically and screenshot_query / screenshot_gc let you inspect and prune it. Tune the storage with:
Env var | Default | Notes |
| (per-user cache dir) | Pin the cache to a specific folder. If the default folder can't be created or written (e.g. corporate policy blocking new folders under your profile), the server auto-probes this → the runtime dir → an OS temp folder and uses the first writable one instead of giving up on the cache. |
|
| Keep at most this many captures in the cache. |
|
| Cap the total cache size on disk. |
| (off) | Drop captures older than this many milliseconds (opt-in). |
|
| Auto-trim the cache as new captures are saved. Set |
|
| Never auto-evict a capture younger than this (ms), so a by-ref link you were just handed survives long enough to open even when another AI/process on the same PC is also capturing. |
Multi-monitor screenshots
screenshot(displayId=…) and screenshot(region=…) capture any monitor, including one placed left of or above the primary — those have negative desktop coordinates, and you pass them exactly as screenshot(detail='meta') reports them. screenshot() with no region is the primary monitor, as it has always been.
A region that cannot be captured comes back as RegionOutsideCapturableBounds rather than a raw Windows error, and the message says which of three things happened. The region may be on no monitor at all, which usually means the coordinates went stale because the window moved or closed — take a fresh screenshot and use the new numbers. It may overlap a monitor but stretch past the edge of the screen area, in which case the coordinates are fine and the region is simply too big: ask for a smaller one, or capture the window itself with screenshot(windowTitle=…). Or this server may be limited to the primary monitor, which the message says outright — along with why, because that decides the fix: if an env override pinned it, screenshot(windowTitle=…) normally still works on every monitor, whereas if the built-in capture module is missing then window capture usually needs that same module and fails too, so move the window onto the primary monitor or reinstall the server. Whole-screen capture and single-window capture are separate parts of that module, though, and a server can end up with one but not the other — so rather than working it out from the cause, read the message: it says plainly whether screenshot(windowTitle=…) is available on this server.
If Windows returns no pixels at all — a locked screen, a UAC prompt, a disconnected remote-desktop session — you get CaptureBackendFailed; capturing the window itself with screenshot(windowTitle=…) usually still works, because it reads through a different Windows API.
Env var | Default | Notes |
| (unset = automatic) | Diagnostic override for the screen-capture path. Set to |
Auto Perception (always-on)
Phase 4 privatizes the explicit perception_* tool family — the v0.12 Auto
Perception layer attaches an attention signal to every desktop_state and
desktop_act response automatically. Action tools also auto-guard when given
a windowTitle. There is no longer a need to register / read / forget lenses
manually.
# desktop_state always returns the attention signal
desktop_state() → {focusedWindow, focusedElement, modal, attention:"ok", ...}
# Action tools auto-guard when windowTitle is given:
keyboard({action:"type", text:"hello", windowTitle:"Notepad"})
→ post.perception:{status:"ok"} // unsafe input blocked if guards fail
# When attention is dirty / stale / settling, refresh with desktop_state:
desktop_state() // re-evaluates attention via Auto PerceptionFor advanced pinned-target workflows, the lensId parameter remains on action
tools (keyboard, mouse_click, mouse_drag, click_element,
browser_click, browser_navigate, browser_eval, desktop_act). Omit
lensId for the normal Auto Perception path. The underlying registry, hot
target cache, and sensor loop are unchanged; only the explicit
perception_register / perception_read / perception_forget / perception_list
tools were retired.
Mouse homing correction
When Claude calls screenshot(detail='text') to read coordinates and then mouse_click seconds later, the target window may have moved. The homing system corrects this automatically.
Tier | How to enable | Latency | What it does |
1 | Always-on (if cache exists) | <1ms | Applies (dx, dy) offset when window moved |
2 | Pass | ~100ms | Auto-focuses window if it went behind another |
3 | Pass | 1–3s | UIA re-query for fresh coords on resize |
# Tier 1 only (automatic)
mouse_click(x=500, y=300)
# Tier 1 + 2: also bring window to front if hidden
mouse_click(x=500, y=300, windowTitle="Notepad")
# Tier 1 + 2 + 3: also re-query UIA if window resized
mouse_click(x=500, y=300, windowTitle="Notepad", elementName="Save")
# Traction control OFF — no correction
mouse_click(x=500, y=300, homing=false)The homing parameter is available on mouse_click, mouse_drag, and scroll. The cache is updated automatically on every screenshot(), desktop_discover(), focus_window(), and workspace_snapshot() call.
mouse_click image-local coords (origin + scale)
When you take a dotByDot screenshot with dotByDotMaxDimension, the response prints the origin and scale values. Instead of computing screen coords manually, copy them into mouse_click:
# Screenshot response:
# origin: (0, 120) | scale: 0.6667
# To click image pixel (ix, iy): mouse_click(x=ix, y=iy, origin={x:0, y:120}, scale=0.6667)
mouse_click(x=640, y=300, origin={x:0, y:120}, scale=0.6667, windowTitle="Chrome")
# Server converts: screen = (0 + 640/0.6667, 120 + 300/0.6667) = (960, 570)This eliminates a whole class of off-by-one and scale bugs. Without origin/scale, x/y remain absolute screen pixels (unchanged behavior).
screenshot key parameters
detail="image" — PNG/WebP pixels (default)
detail="text" — UIA element JSON + clickAt coords (no image, ~100–300 tok)
detail="meta" — Title + region only (cheapest, ~20 tok/window)
dotByDot=true — 1:1 WebP; image_px + origin = screen_px
dotByDotMaxDimension=N — cap longest edge (response includes scale for coord math)
grayscale=true — ~50% smaller for text-heavy captures (code/AWS console)
region={x,y,w,h} — with windowTitle: window-local coords (exclude browser chrome)
without: virtual screen coords
diffMode=true — I-frame first call, P-frame (changed windows only) after (~160 tok)
ocrFallback="auto" — detail='text' auto-fires Windows OCR on uiaSparse or emptyRecommended Chrome combo (50–70% data reduction):
screenshot(windowTitle="Chrome",
dotByDot=true, dotByDotMaxDimension=1280, grayscale=true,
region={x:0, y:120, width:1920, height:900}) # skip browser chromeRecommended workflow:
workspace_snapshot() → full orientation (resets diff buffer)
screenshot(detail="text", windowTitle=X) → get actionable[].clickAt coords
mouse_click(x, y) → click directly, no math needed
screenshot(diffMode=true) → check only what changed (~160 tok)Security
Emergency stop (Failsafe)
Park the mouse in the top-left corner of the primary monitor (within 10px of 0,0) for 500ms continuously to trigger the emergency stop.
The trigger corner is on the primary monitor only. Areas that used to trigger the stop in older versions (monitors left of or above the primary) no longer do; if the cursor dwells there, a one-time balloon notification points you to the right corner.
While a tool call is running: the server exits (exit code 1) — the runaway-automation brake. A balloon notification and a diagnostic log entry (with cursor coordinates) record why it stopped. Only a call that is actually mid-flight triggers the exit; in the rare case where that call finishes during the ~1 second the notification takes, the server stays up instead and a follow-up balloon corrects the first one.
While idle: the server stays up and refuses new tool calls until the cursor leaves the corner. Background credential autofill (
key_locker) is cancelled before any of its dialogs open — while you hold the corner, no credential prompt dialog appears and no credential is typed. It does not pick up again by itself: move the cursor away from the corner and run the command again.Per-tool check: runs before every tool handler. Background monitor: 500ms polling as a backup for long-running operations. Trigger radius: 10px.
DESKTOP_TOUCH_FAILSAFE_HOLD_MS— dwell time in ms before the stop fires (default500;0= fire immediately on corner entry).
Blocked operations
workspace_launch blocklist:
cmd.exe, powershell.exe, pwsh.exe, wscript.exe, cscript.exe, mshta.exe, regsvr32.exe, rundll32.exe, msiexec.exe, bash.exe, wsl.exe are blocked.
Script extensions (.bat, .ps1, .vbs, etc.) are rejected. Arguments containing ;, &, |, `, $(, ${ are also rejected.
keyboard(action='press') blocklist:
Win+R (Run dialog), Win+X (admin menu), Win+S (search), Win+L (lock screen) are blocked.
PowerShell injection protection
All -like patterns in the UIA bridge PowerShell fallback path are sanitized with escapeLike(), which escapes wildcard characters (*, ?, [, ]) before they reach PowerShell. When the Rust native engine is active, PowerShell is not invoked for UIA operations.
Allowlist for workspace_launch
Shell interpreters are blocked by default. To allow specific executables, create an allowlist file:
File locations (searched in order):
Path in
DESKTOP_TOUCH_ALLOWLISTenvironment variable~/.claude/desktop-touch-allowlist.jsondesktop-touch-allowlist.jsonin the server's working directory
Format:
{
"allowedExecutables": [
"pwsh.exe",
"C:\\Tools\\myapp.exe"
]
}Changes take effect immediately — no restart needed.
Mouse movement speed
All mouse tools (mouse_click, mouse_drag, scroll) accept an optional speed parameter:
Value | Behavior |
Omitted | Uses the configured default (see below) |
| Instant teleport — |
| Animated movement at N px/sec |
Default speed is 1500 px/sec. Change it permanently via the DESKTOP_TOUCH_MOUSE_SPEED environment variable:
{
"mcpServers": {
"desktop-touch": {
"type": "stdio",
"command": "npx",
"args": ["-y", "@harusame64/desktop-touch-mcp"],
"env": {
"DESKTOP_TOUCH_MOUSE_SPEED": "3000"
}
}
}
}Common values: 0 = teleport, 1500 = default gentle, 3000 = fast, 5000 = very fast.
Force-Focus (AttachThreadInput)
Windows foreground-stealing protection can prevent SetForegroundWindow from succeeding when another window (such as a pinned Claude CLI) is in the foreground. This causes subsequent keystrokes or clicks to land in the wrong window — a silent failure.
mouse_click, keyboard(action='type'), keyboard(action='press'), and terminal(action='send') all accept a forceFocus parameter that bypasses this protection using AttachThreadInput:
{
"name": "mouse_click",
"arguments": {
"x": 500,
"y": 300,
"windowTitle": "Google Chrome",
"forceFocus": true
}
}If the force attempt is refused despite AttachThreadInput, the response is ok:false with code: "ForegroundRestricted" (issue #202 unification — same shape as focus_window, keyboard, terminal_send, mouse_click). The action itself is suppressed so the keystrokes / click never land on the wrong window. Recover via focus_window's auto-escalate ladder before retrying. The legacy hints.warnings: ["ForceFocusRefused"] shape is no longer emitted.
Global default via environment variable:
{
"mcpServers": {
"desktop-touch": {
"env": {
"DESKTOP_TOUCH_FORCE_FOCUS": "1"
}
}
}
}Setting DESKTOP_TOUCH_FORCE_FOCUS=1 makes forceFocus: true the default for all four tools without changing each call.
Known tradeoffs:
During the ~10ms
AttachThreadInputwindow, key state and mouse capture are shared between the two threads. In rapid macro sequences this can cause a race condition (rare in practice).Disable
forceFocus(or unset the env var) when the user is manually operating another app to avoid unexpected focus shifts.
Auto Guard
Action tools (mouse_click, mouse_drag, keyboard(action='type'/'press'), click_element, desktop_act, browser_click, browser_navigate) automatically guard each action when you pass windowTitle / tabId:
Verifies target window identity (process restart / HWND replacement detected)
Confirms click coordinates are inside the target window rect
Returns
post.perception.statuson every response — including failures — so the LLM can recover without a screenshot
Keyboard writes must name a destination. keyboard(action='type'/'press'/'sequence') requires either windowTitle or hwnd. Without one there is no target to guard, and the keys would land on whatever window is foreground at that instant — including one you just clicked into yourself. Such a call is refused with code:"DestinationRequired" before any key is sent, and a windowTitle that is empty or only spaces counts as no target at all. A window that has no title can be addressed by hwnd, but only while it is already the foreground window — keyboard focus and guarding cannot target a titleless window yet, so bring it forward with focus_window first if it is not in front. Such a call also comes back with a warning saying the input was delivered unguarded.
Variable | Default | Meaning |
| (unset = required) | Set to |
| (unset = on) | Set to |
Disabling auto guard — set DESKTOP_TOUCH_AUTO_GUARD=0 to restore v0.11.12 behavior (no auto guard):
{
"mcpServers": {
"desktop-touch": {
"type": "stdio",
"command": "npx",
"args": ["-y", "@harusame64/desktop-touch-mcp"],
"env": {
"DESKTOP_TOUCH_AUTO_GUARD": "0"
}
}
}
}When auto guard is enabled (default), post.perception.status will be one of:
Status | Meaning |
| Guard passed — target verified |
|
|
| Multiple windows matched; use a more specific title |
| No window matched the given title |
| Window was replaced (process restart / HWND change) |
| A modal dialog is in the way — dismiss it, then retry |
| Click coordinates are outside the target window rect |
| The browser tab is still loading — wait, then retry |
| Use |
| A |
When unsafe_coordinates or identity_changed is returned, the response may include a suggestedFix.fixId. Pass that fixId to the relevant tool call to approve the recovery:
{ "name": "mouse_click", "arguments": { "fixId": "fix-..." } }
{ "name": "keyboard(action='type')", "arguments": { "fixId": "fix-...", "text": "hello" } }
{ "name": "click_element", "arguments": { "fixId": "fix-..." } }
{ "name": "browser_click", "arguments": { "fixId": "fix-..." } }The fix is one-shot and expires in 15 seconds. The server revalidates the target process identity before executing.
Diagnostic log
The server keeps an append-only log of events that never reach a tool response, at
%USERPROFILE%\.desktop-touch-mcp\logs\diagnostic.log (one JSON object per line). It records
crashes and slow calls, and — since the diagnostic log became the place to look when input lands in
the wrong place — how each windowTitle was resolved and where each write went:
a
resolverecord per title lookup: how many windows matched, which one was picked, the ones that lost, and a flag when the terminal process-name fallback fired because nothing matched by title;a
dispatch_sinkrecord per input dispatch —keyboard,terminal,scrollanddesktop_act's background writes: which channel was used, which window it was addressed to, and which window was in the foreground at that moment;a correlation id shared by all records from one tool call, so a resolution can be matched to the write it produced even when calls overlap.
If an input call ever seems to type into the wrong window, this is the file that says which window it picked and why. A record is written immediately before the write leaves the process, so a dispatch that is refused or fails first is not on record as having happened.
Variable | Default | Meaning |
| (unset = off) | Window titles and the titles you search for are recorded as a short hash plus their length, because a title can contain a file name, a mail subject, or a browser page title. Set to |
| (unset = on) | Set to |
| (per-user log dir) | Write the log somewhere else. |
Advanced response options
browser_eval Structured Mode
Pass withPerception: true to receive a structured JSON response with post.perception instead of raw text:
{ "name": "browser_eval", "arguments": { "expression": "document.title", "withPerception": true } }Returns { ok: true, result: "...", post: { perception: { status: "ok", ... } } }.
mouse_drag Cross-Window Guard
mouse_drag now guards both start and end coordinates. Drags that cross window boundaries (or reach the desktop wallpaper) are blocked by default. To allow intentional cross-window or range-selection drags:
{ "name": "mouse_drag", "arguments": { "startX": 100, "startY": 100, "endX": 900, "endY": 900, "allowCrossWindowDrag": true } }Performance (v0.15 — Rust Native Engine)
The Rust native engine (@harusame64/desktop-touch-engine) replaces PowerShell process spawning with direct COM calls over a persistent MTA thread. It loads automatically as a .node addon — no configuration needed.
UIA Benchmark (vs PowerShell baseline)
Function | Rust Native | PowerShell | Speedup |
| 2.2 ms | 366 ms | 163.9× |
| 106.5 ms | 346 ms | 3.3× |
Weighted average | ~82× |
Image Diff Benchmark (SSE2 SIMD)
Function | Rust (SSE2) | TypeScript | Speedup |
| 0.26 ms | 3.8 ms | ~15× |
| 0.09 ms | 1.2 ms | ~13× |
Architecture
Claude CLI / MCP Client
│ stdio or HTTP (MCP protocol)
▼
desktop-touch-mcp (TypeScript)
│
├── Rust Native Engine (.node addon) ← NEW in v0.15
│ ├── UIA: 13 functions via napi-rs + windows-rs 0.62
│ │ └── Dedicated COM thread (MTA) + batch BFS algorithm
│ └── Image: SSE2 SIMD pixel diff + perceptual hashing
│
└── PowerShell Fallback (automatic)
└── Activates transparently if .node is unavailableWhy getUiElements is 3.3× (not 160×)
The 160× speedup on getFocusedElement comes from eliminating PowerShell process startup (~200 ms) and .NET assembly loading. For getUiElements, the bottleneck shifts to the UIA provider inside the target application (e.g., Explorer) — it must enumerate its UI tree regardless of who asks. The Rust engine uses a batch BFS algorithm (FindAllBuildCache + TreeScope_Children) that minimizes cross-process RPC calls and supports maxElements early exit, making it dramatically faster on large trees (VS Code, browsers with 1000+ elements).
UI Operating Layer (V2)
Status: Default ON since v0.17.
desktop_discoveranddesktop_actare available out of the box.
V2 introduces two new tools that replace coordinate-based clicking with entity-based interaction:
Tool | Description |
| Observe a window or browser tab. Returns interactive entities with leases — no raw screen coordinates. Supports UIA (native), CDP (browser), terminal, and visual GPU lanes. |
| Interact with an entity returned by |
Clicking — priority order
When multiple tools could perform the same click, prefer them in this order:
browser_click(selector)— Chrome / Edge over CDP (stable across repaints)desktop_act(lease)— native windows, dialogs, visual-only targets (entity-based; use afterdesktop_discover)click_element(name | automationId)— native UIA fallback whendesktop_actreturnsok:falsemouse_click(x, y)— pixel-level last resort (origin+scalefromdotByDotscreenshots only)
Disabling V2 (kill switch)
To hide desktop_discover / desktop_act from the tool catalog, add the disable flag and restart:
{
"mcpServers": {
"desktop-touch": {
"type": "stdio",
"command": "npx",
"args": ["-y", "@harusame64/desktop-touch-mcp"],
"env": {
"DESKTOP_TOUCH_DISABLE_FUKUWARAI_V2": "1"
}
}
}
}All V1 tools continue to work without interruption — no reinstall required. Remove the env entry and restart to re-enable.
Flag semantics (exact-match: only the literal string "1" counts):
| V2 state |
unset / not | ON (default) |
| OFF (kill switch) |
Removed: DESKTOP_TOUCH_ENABLE_FUKUWARAI_V2
This was the opt-in switch in v0.16.x. V2 is on by default since v0.17, so the flag no longer has any effect and is safe to delete from your config. To turn V2 off, set DESKTOP_TOUCH_DISABLE_FUKUWARAI_V2=1.
Recovery when V2 fails
If desktop_act returns ok: false, read reason and follow the built-in recovery hints in the tool description. Common paths:
lease_expired/*_mismatch/entity_not_found→ re-calldesktop_discovermodal_blocking→response.blockingElement(when present) carries{ name, role, automationId? }; dismiss withclick_element(name=blockingElement.name), then retryentity_outside_viewport→ the element moved off screen:scroll/scroll(action='to_element')when it scrolled out of its own window, or re-calldesktop_discoverwhen the window itself moved or closedorigin_window_not_visible→focus_window(windowTitle)to restore the minimised / hidden window, then re-calldesktop_discovercoordinate_outside_reachable_bounds→ the target is on no connected monitor — usually stale coordinates: re-rundesktop_discover. (Without the built-in Windows input module, only the primary monitor is reachable; the message says so.)cursor_placement_blocked→ the pointer could not be placed there (an app is holding the cursor, or the session is not interactive), so nothing was clicked: free the cursor or reconnect the session, or useclick_element(UIA invoke, cursor-free)executor_failed→ fall back toclick_element/mouse_click/browser_click
For desktop_discover warnings (visual_provider_unavailable, visual_provider_warming, cdp_provider_failed, …), the coordinate-based tools (screenshot(detail='text'), click_element, mouse_click, terminal, …) remain available as an escape hatch.
Known limitations
Limitation | Detail | Workaround |
Games / video players may return black or hang in PrintWindow capture | DirectX fullscreen apps may not redraw under | Retry with |
UIA call overhead | ~2 ms (focus) / ~100 ms (tree) via Rust native engine; ~300 ms via PowerShell fallback | Rust engine loads automatically; |
Chrome / WinUI3 UIA elements are empty | Chromium exposes only limited UIA |
|
Chromium title-regex misses when sites rewrite | Guard relies on the | Title is treated as plain Chrome (UIA runs). OCR path is still reachable via |
| If Chrome is already running on the default profile without the flag, | Close Chrome first, then |
Layer buffer TTL | Buffer auto-clears after 90s of inactivity → next | After long waits, call |
| When | Call |
| Non-ASCII punctuation (em-dash | Always use |
| Setting | Use |
Token cost reference
Mode | Tokens | Use case |
| ~443 tok | General visual check |
| ~800 tok | Precise clicking (no coordinate math) |
| ~160 tok | Post-action diff |
| ~100–300 tok | UI interaction (no image) |
| ~2000 tok | Full session orientation |
Thanks
Huge thanks to everyone who tried a desktop-automation MCP server, filed issues, opened PRs, and shared what broke. Every bug report made the next release better. Thank you for building with me!
License
MIT
Available Tools
30 toolsbrowser_clickA
Click a DOM element in Chrome/Edge. Two ways to target: (1) selector — a CSS selector (combines browser_locate + mouse_click; stable across repaints); or (2) by-axis (semantic) — by:'text'|'regex'|'role'|'ariaLabel' + pattern, so you do not have to build a CSS selector for dynamic-class SPAs. by-axis resolves to a SINGLE actionable element (climbing to a clickable ancestor up to 3 levels, hit-testing for occlusion) and STOPS with code:'BrowserAmbiguousTarget' (candidates[] + next[] hints) when 2+ actionable elements match, or code:'BrowserNoActionableTarget' when matches exist but none is clickable — it never guesses. If the target is behind a modal dialog blocking the page, BOTH targeting modes STOP with code:'BrowserModalBlocking' (context.blockingElement {name, role}) instead of clicking through to the backdrop — dismiss the dialog (its close button or Escape) and retry; a plain navigation drawer does not count as blocking. Optionally add role to filter (by:'text',pattern:'Save',role:'button') and scope to narrow the search. Provide EITHER selector OR by+pattern (not both). Pass tabId+port so the server auto-guards (verifies tab readyState and identity) and returns post.perception.status. lensId is optional for advanced pinned-tab workflows. Caveats: selector mode fails if the element is outside the visible viewport — scroll it into view with browser_eval("document.querySelector('sel').scrollIntoView()") first (by-axis only resolves in-viewport actionable targets). hints.verifyDelivery:{status:'delivered'|'unverifiable', reason, observedSignals:{mutationCount,urlChanged,activeElementChanged}} reports the post-click observation in 2 values: 'delivered' fires only when mutationCount>0 OR urlChanged (activeElementChanged is recorded in observedSignals but intentionally NOT a delivery signal — plain clicks on focusable controls always update focus, treating that as 'delivered' would mask silent-fail regressions); 'unverifiable' reason ∈ {'iframe_context_mismatch','no_dom_mutation','probe_install_failed','probe_read_failed'}. CDP emits 2 values only (focus_only is a UIA-path concept, N/A here). BrowserClickNotDelivered is reserved-only (false-positive risk too high to emit) — degradation reads from 'unverifiable' status.
| Name | Required | Description | Default |
|---|---|---|---|
| by | No | Semantic axis to target by INSTEAD of a CSS selector: 'text' (visible text), 'regex', 'role' (ARIA/implicit role), 'ariaLabel'. Pair with pattern. Resolves to a SINGLE actionable element and STOPS with candidates when ambiguous. | |
| port | No | Chrome/Edge CDP remote debugging port. | |
| role | No | Optional ARIA/implicit-role filter AND-combined with by (e.g. by:'text', pattern:'Save', role:'button'). | |
| fixId | No | Approve a pending suggestedFix (one-shot, 15s TTL). Selector mode only. | |
| scope | No | Optional CSS selector to limit the by-axis search scope (disambiguation). | |
| tabId | No | Tab ID from browser_open. Omit to use the first page tab. | |
| lensId | No | Optional perception lens ID. Guards (target.identityStable) are evaluated before clicking, and a perception envelope is attached to post.perception on success. | |
| include | No | Optional response-shape opt-in. `['envelope']` returns the self-documenting envelope (`_version` / `data` / `as_of` / `confidence`). `['raw']` forces raw shape (overrides DESKTOP_TOUCH_ENVELOPE=1 server default). Default behaviour is raw shape (compat with existing clients). | |
| narrate | No | Narration level. rich includes UIA or browser state diff when supported. | minimal |
| pattern | No | Value matched against the chosen by axis (required when by is set). | |
| selector | No | CSS selector for the target element (e.g. '#submit', '.btn'). Provide EITHER selector OR by+pattern. | |
| caseSensitive | No | Case-sensitive matching for by:'text'/'regex' (default false). | |
| scrollIntoView | No | When true, if the target is outside the viewport, scroll it into view (centered) before clicking, instead of failing with ElementNotInViewport. Default false preserves the explicit scrollIntoView-then-retry workflow. Selector mode only (by-axis resolves only in-viewport actionable targets). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden and excels: details the by-axis resolution algorithm (climbing ancestors, hit-testing), error codes, modal blocking detection, verifyDelivery status with observedSignals, and disclaimers about delivery signals. It is remarkably transparent about internal behavior and edge cases.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with clear sections and front-loaded with the core purpose. However, it is somewhat verbose and could be trimmed without losing essential information. Still, the organization helps an agent parse it.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite 13 parameters and no output schema, the description covers all behavioral aspects: input modes, error handling, output parameters like verifyDelivery and post.perception, and linkages to other tools. An agent has sufficient information to use the tool correctly in diverse scenarios.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, but the description adds significant value beyond the schema: explains the semantics of by-axis parameters (e.g., how they resolve to actionable elements), the interaction between parameters (by+pattern vs selector), and detailed behavior of scrollIntoView and verifyDelivery. This greatly aids correct invocation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Click a DOM element in Chrome/Edge' and details two distinct targeting modes (selector and by-axis), distinguishing itself from siblings like browser_locate and mouse_click. It specifies the browser context and core action without ambiguity.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Extensive guidance on when to use each targeting mode, how to handle error codes (BrowserAmbiguousTarget, BrowserNoActionableTarget, BrowserModalBlocking), and explicit caveats about viewport scrolling. It provides alternatives like scrollIntoView and browser_eval, and contrasts with sibling tools implicitly.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
browser_evalA
Purpose: Inspect or operate on a browser tab via 3 actions: 'js' (evaluate JS), 'dom' (get HTML), 'appState' (extract SSR-injected SPA state). Details: action='js' — Run a JS expression. withPerception:true wraps in {ok, result, post}. action='dom' — Return outerHTML of selector (or document.body), truncated to maxLength. action='appState' — Scan Next/Nuxt/Remix/Apollo/GitHub/Redux SSR injected JSON; pass selectors to override defaults. Prefer: Use action='appState' BEFORE 'dom' or 'js' on SPAs where rendered HTML is sparse — single CDP call. Use 'dom' when 'appState' is empty and you need page structure. Use 'js' as the escape hatch for arbitrary scripting. Caveats: DOM nodes cannot be returned from action='js' directly (circular refs are serialized safely). React/Vue/Svelte controlled inputs cannot be set via element.value — use keyboard(action='type') / browser_fill instead. readyState is strictly checked; guard blocks if page is still loading. Typed errors: code:'BrowserNotConnected' on CDP disconnect (re-attach via browser_open); code:'AutoGuardBlocked' when the auto-guard refuses (e.g. page still loading) — the error message preserves the guard's 1-sentence recommended next step (most often wait_until({condition:'ready_state'}) or browser_eval readyState polling, then retry). Examples: browser_eval({action:'js', expression:'document.title'}) → page title browser_eval({action:'dom', selector:'#main', maxLength:5000}) → outerHTML browser_eval({action:'appState'}) → default SPA state probes
| Name | Required | Description | Default |
|---|---|---|---|
| port | No | Chrome/Edge CDP remote debugging port. | |
| tabId | No | Tab ID from browser_open. Omit to use the first page tab. | |
| action | Yes | Action selector — one of: js, dom, appState. Per-action required fields are enforced at call time (see the tool description); this flat schema lists every action's fields as optional. | |
| lensId | No | Optional perception lens ID. Guards (target.identityStable) are evaluated before eval. | |
| include | No | Optional response-shape opt-in. `['envelope']` returns the self-documenting envelope (`_version` / `data` / `as_of` / `confidence`). `['raw']` forces raw shape (overrides DESKTOP_TOUCH_ENVELOPE=1 server default). Default behaviour is raw shape (compat with existing clients). | |
| maxBytes | No | Max bytes per individual payload (default 4000). Larger payloads are truncated. | |
| selector | No | CSS selector for root element. Omit for document.body. | |
| maxLength | No | Max characters of HTML to return (default 10000). | |
| selectors | No | Custom probe selectors. Omit to use the default SPA framework set (__NEXT_DATA__ / __NUXT_DATA__ / __REMIX_CONTEXT__ / __APOLLO_STATE__ / window:__INITIAL_STATE__ etc.). Window globals must be prefixed with 'window:'. | |
| expression | No | JavaScript expression to evaluate. The server automatically wraps snippets in an async IIFE to avoid repeated const/let collisions. For multi-statement snippets, use an explicit final return value. Declarations (const/let/var) are scoped per snippet — use window.* / globalThis.* for persistence. A single eval is bounded by the CDP per-command timeout (~15s): do NOT write in-page polling loops here — use wait_until (element_matches / url_matches / ready_state) to wait for conditions instead. | |
| includeContext | No | When true, append activeTab and readyState context to the response. | |
| withPerception | No | When true, return structured JSON {ok, result, post} with post.perception attached. Default false preserves raw-text return. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Since no annotations are provided, the description fully discloses behaviors: actions, truncation, withPerception wrapping, readyState checks, error codes (BrowserNotConnected, AutoGuardBlocked), serialization limitations, and execution timeout. No contradictions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with clear sections (Purpose, Details, Prefer, Caveats, Examples) and front-loads the purpose. It is slightly verbose but every sentence adds value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the complexity (12 parameters, 3 actions), the description covers all aspects, including return structures, error types, and recommended usage patterns. Examples illustrate typical calls. No output schema exists, but the description adequately hints at return shapes.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, but the description adds significant context beyond the schema, such as the purpose of each action, the effect of withPerception, and the use of selectors for appState. It provides examples that clarify parameter usage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Inspect or operate on a browser tab via 3 actions'. It specifies each action (js, dom, appState) and differentiates from sibling tools by focusing on evaluation/scripting rather than navigation, clicks, etc.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The 'Prefer:' section explicitly tells when to use each action, e.g., 'Use action='appState' BEFORE 'dom' or 'js' on SPAs'. The 'Caveats' section specifies when not to use this tool (e.g., for controlled inputs) and provides error handling guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
browser_fillA
Fill a form input with a value via CDP — works on React/Vue/Svelte controlled inputs that reject browser_eval value assignment. Two ways to target: (1) selector — a CSS selector (use browser_overview / browser_locate to find one); or (2) by-axis (semantic) — by:'text'|'regex'|'role'|'ariaLabel' + pattern (e.g. by:'ariaLabel', pattern:'Email address', or by:'role', pattern:'textbox'), so you do not have to build a CSS selector. by-axis resolves to a SINGLE fillable element and STOPS with code:'BrowserAmbiguousTarget' (candidates[] + next[] hints) when 2+ match, or code:'BrowserNoActionableTarget' when the match is not a fillable input/textarea/contenteditable — it never guesses. Optionally add role to filter and scope to narrow. Provide EITHER selector OR by+pattern (not both). Use this over browser_eval when setting a controlled input's value via JS does not update framework state. Caveats: Requires browser_open (CDP active). actual in the response shows the element's value after fill; verify it matches the intended value. Typed errors: code:'BrowserFillNotDelivered' on post-fill value mismatch — note the false-positive case where a React controlled input's onChange transforms the value (delivery actually succeeded; hints.verifyDelivery.subReason:'controlled_input_transform' for that case; the actual value is authoritative).
| Name | Required | Description | Default |
|---|---|---|---|
| by | No | Semantic axis to target by INSTEAD of a CSS selector: 'text' (visible text), 'regex', 'role' (ARIA/implicit role), 'ariaLabel'. Pair with pattern. Resolves to a SINGLE actionable element and STOPS with candidates when ambiguous. | |
| port | No | Chrome/Edge CDP remote debugging port. | |
| role | No | Optional ARIA/implicit-role filter AND-combined with by (e.g. by:'text', pattern:'Save', role:'button'). | |
| scope | No | Optional CSS selector to limit the by-axis search scope (disambiguation). | |
| tabId | No | Tab ID from browser_open. Omit to use the first page tab. | |
| value | Yes | Text to fill into the input element | |
| include | No | Optional response-shape opt-in. `['envelope']` returns the self-documenting envelope (`_version` / `data` / `as_of` / `confidence`). `['raw']` forces raw shape (overrides DESKTOP_TOUCH_ENVELOPE=1 server default). Default behaviour is raw shape (compat with existing clients). | |
| pattern | No | Value matched against the chosen by axis (required when by is set). | |
| selector | No | CSS selector for the input element. Provide EITHER selector OR by+pattern. | |
| caseSensitive | No | Case-sensitive matching for by:'text'/'regex' (default false). | |
| includeContext | No | When true, append activeTab and readyState context to the response. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description fully discloses behavioral traits including CDP usage, controlled input handling, resolution logic, error codes (BrowserAmbiguousTarget, BrowserNoActionableTarget, BrowserFillNotDelivered), and the false-positive case for controlled input transforms. It also advises verifying actual value.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with bullet points and clear sections, front-loaded with main purpose, and every sentence adds value without being verbose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 11 parameters, no output schema, and no annotations, the description is thoroughly complete, covering targeting, error handling, usage, and caveats for all scenarios.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Although schema coverage is 100%, the description adds substantial meaning beyond the schema: explaining how by-axis targeting works, mutual exclusivity with selector, role filtering, scope narrowing, case sensitivity, port default, and error handling. It also integrates with browser_overview/locate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description begins with 'Fill a form input with a value via CDP' and specifies it works on React/Vue/Svelte controlled inputs, clearly distinguishing it from sibling tools like browser_eval.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly states when to use this tool over browser_eval and provides two targeting methods with clear caveats on ambiguity and error handling. It also notes the prerequisite of browser_open.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
browser_formA
Inspect all form fields (input, select, textarea, button) within a CSS-selector-specified container and return their name, type, id, current value, hint text, disabled/readOnly state, and associated label text (resolved via for[id], ancestor LABEL, aria-labelledby, aria-label in that order). Use this before browser_fill to discover exact field selectors and avoid accidentally targeting the wrong input (e.g. a global search bar). Caveats: Requires browser_open (CDP active). Hidden inputs (type=hidden) are excluded by default — set includeHidden:true if needed. Value text is truncated at 200 chars.
| Name | Required | Description | Default |
|---|---|---|---|
| port | No | Chrome/Edge CDP remote debugging port. | |
| tabId | No | Tab ID from browser_open. Omit to use the first page tab. | |
| include | No | Optional response-shape opt-in. `['envelope']` returns the self-documenting envelope (`_version` / `data` / `as_of` / `confidence`). `['raw']` forces raw shape (overrides DESKTOP_TOUCH_ENVELOPE=1 server default). Default behaviour is raw shape (compat with existing clients). | |
| selector | Yes | CSS selector for the form or container element to inspect (e.g. '#login-form', '.search-bar'). All input, select, textarea, and button descendants are returned. | |
| maxResults | No | Maximum number of form fields to return (default 100). | |
| includeHidden | No | When true, include hidden inputs (type=hidden). Default false to avoid CSRF-token / serialized-state clutter. | |
| includeContext | No | When true, append activeTab and readyState context to the response. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description carries full burden. It discloses prerequisites (CDP active), default behavior (hidden excluded), value truncation, and label resolution order. Lacks mention of performance or side effects, but covers key behavioral traits.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Description is concise, well-structured, and front-loaded with purpose, followed by usage guide and caveats. No redundant information; every sentence adds value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, description details what fields are returned but not the exact structure. Lacks error handling or empty-selector behavior. However, it covers prerequisites, parameter defaults, and integration with browser_fill, which is sufficient for most use cases.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, baseline 3. Description adds meaningful context: why includeHidden defaults to false (avoid clutter), order of label resolution, and that includeContext adds activeTab/readyState. Provides value beyond schema descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool inspects form fields within a CSS selector container, returning details like name, type, id, value, etc. It distinguishes from sibling tools by mentioning its use before browser_fill to discover exact field selectors.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly tells when to use ('Use this before browser_fill to discover exact field selectors') and provides caveats: requires browser_open, hidden inputs excluded by default, value truncation at 200 chars. Guides against accidental targeting of wrong inputs.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
browser_locateA
Find a DOM element by CSS selector and return its physical screen coordinates — compatible directly with mouse_click. Prefer browser_click to find+click in one step. Prefer browser_overview to discover selectors. Caveats: Coordinates are captured at call time; if the page reflows before mouse_click, coords may be stale. Typed errors: code:'BrowserNotConnected' (call browser_open first), code:'ElementNotFound' (selector did not match — re-discover via browser_overview / browser_search).
| Name | Required | Description | Default |
|---|---|---|---|
| port | No | Chrome/Edge CDP remote debugging port. | |
| tabId | No | Tab ID from browser_open. Omit to use the first page tab. | |
| include | No | Optional response-shape opt-in. `['envelope']` returns the self-documenting envelope (`_version` / `data` / `as_of` / `confidence`). `['raw']` forces raw shape (overrides DESKTOP_TOUCH_ENVELOPE=1 server default). Default behaviour is raw shape (compat with existing clients). | |
| selector | Yes | CSS selector for the target element (e.g. '#submit', '.btn', 'button[type=submit]'). | |
| includeContext | No | When true, append activeTab and readyState context to the response. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description carries full burden. It discloses capture-time coordinates, staleness risk, and typed errors. Could mention more about response shape, but sufficient for safe use.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences, front-loaded with purpose, zero wasted words. Caveats and error patterns are clearly separated.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema, but description states return of physical screen coordinates. More detail on return format (e.g., object with x,y) would improve completeness, but error types and compatibility hints are good.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, baseline 3. Description adds little beyond schema for parameters, but mentions coordinate compatibility with mouse_click, which is helpful context.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states action (find DOM element) and output (physical screen coordinates), explicitly distinguishing from sibling tools browser_click and browser_overview.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly suggests using browser_click for find+click in one step and browser_overview to discover selectors. Also includes caveat about stale coordinates after page reflow.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
browser_openA
Connect to Chrome/Edge running with --remote-debugging-port and return open tab IDs — required before all other browser_* tools. Pass launch:{} (or with overrides) to auto-spawn a debug-mode browser when no CDP endpoint is live (idempotent: an already-running endpoint is preferred). Returns tabs[] with id, url, title, active — pass tabId to browser_* tools to target a specific tab. Caveats: CDP connection is per-process; if Chrome restarts, call browser_open again to get fresh tab IDs. A Chrome session started without --remote-debugging-port cannot be taken over — close it first or use a separate userDataDir. If the CDP endpoint is unreachable and launch is omitted, returns ok:false (typically code:'BrowserNotConnected' when the fetch surfaces ECONNREFUSED, otherwise code:'ToolError' with error 'Cannot reach Chrome/Edge CDP...'); re-call with launch:{} (idempotent) to auto-spawn or start Chrome manually with --remote-debugging-port=9222.
| Name | Required | Description | Default |
|---|---|---|---|
| port | No | Chrome/Edge CDP remote debugging port. | |
| launch | No | If set, spawn a debug-mode browser when no CDP endpoint is live on the target port (idempotent: an already-running endpoint is preferred and the spawn step is skipped). Pass {} to use defaults (chrome, C:\tmp\cdp, no initial URL). Omit to perform pure connect. | |
| include | No | Optional response-shape opt-in. `['envelope']` returns the self-documenting envelope (`_version` / `data` / `as_of` / `confidence`). `['raw']` forces raw shape (overrides DESKTOP_TOUCH_ENVELOPE=1 server default). Default behaviour is raw shape (compat with existing clients). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Despite no annotations, the description fully discloses behavior: returns tabs array with specific fields, idempotent launch, error codes, killExisting warning data loss, and CDP connection per-process. No contradictions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Single paragraph but well-structured with clear logical flow. Slightly redundant on 'idempotent' but overall concise given the information density.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Covers all necessary aspects: prerequisite, input/output, errors, edge cases (browser restart, existing session, unreachable endpoint). References sibling tools. Complete for a setup tool with no output schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema already describes all parameters, but description adds context: explains launch parameter purpose, default behavior, example usage, and caveats for killExisting. Adds value beyond schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool connects to a debug-mode browser and returns open tab IDs, and it distinguishes from siblings by being the prerequisite for all other browser_* tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicit instructions on when to use (required before other browser tools), when to use launch:{} vs pure connect, and caveats (Chrome restart, cannot take over existing session without debug port). Also gives error handling guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
browser_overviewA
List all interactive elements (links, buttons, inputs, ARIA controls) on the current page with CSS selectors, visible text or value for inputs, and viewport status — use before browser_click to discover stable selectors, and prefer this over screenshot when verifying button/toggle state after submission (no image tokens, structured output). scope limits to a CSS subsection (e.g. '.sidebar'). Returns state (checked/pressed/selected/expanded) for ARIA custom controls. Also returns a modal: section — whether a true modal dialog is blocking the page (isModal + blocker {name, role} + the signals it was judged on); it is ALWAYS present (isModal:false when no modal), and a navigation drawer is NOT reported as a modal (only an aria-modal / alertdialog / native showModal dialog, or a backdrop-backed dialog that locks the page, is treated as modal). Caveats: Selectors are CDP-generated snapshots — re-call after page navigates or re-renders. Input text reflects the empty-field hint text when defined (takes priority over typed value) — use browser_eval('document.querySelector(sel).value') to read actual typed content. Typed errors: code:'BrowserNotConnected' (CDP not attached — call browser_open or browser_open({launch:{}})). Note: a non-matching scope CSS selector silently falls back to the full document (does not raise an error) — verify the selector via browser_eval if scoped enumeration is required.
| Name | Required | Description | Default |
|---|---|---|---|
| port | No | Chrome/Edge CDP remote debugging port. | |
| scope | No | CSS selector to limit the search scope (e.g. '.s-main-slot', '#nav-search-form'). Omit to scan the full page. | |
| tabId | No | Tab ID from browser_open. Omit to use the first page tab. | |
| types | No | Element types to include. Default 'all' returns links, buttons, and inputs. | |
| include | No | Optional response-shape opt-in. `['envelope']` returns the self-documenting envelope (`_version` / `data` / `as_of` / `confidence`). `['raw']` forces raw shape (overrides DESKTOP_TOUCH_ENVELOPE=1 server default). Default behaviour is raw shape (compat with existing clients). | |
| maxResults | No | Maximum number of elements to return (default 50). | |
| inViewportOnly | No | When true, only return elements currently visible in the viewport. | |
| includeContext | No | When true, append activeTab and readyState context to the response. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations, so description carries full burden. It details return state for ARIA controls, modal detection logic, caveats about CDP snapshots, input hint text behavior, error codes, and scope fallback. Exceptionally transparent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Well-structured with main purpose, usage guidance, modal section, and caveats. Slightly long but each sentence adds value. Good front-loading.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 8 parameters and no output schema, the description is highly comprehensive. Covers behavior, caveats, errors, modal detection, input nuance, and scope fallback. Return value format is inferred sufficiently.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, baseline 3. Description adds meaning beyond schema by explaining parameter behaviors like scope fallback, types default, and include envelope option, thus adding value.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool lists interactive elements with CSS selectors, text, and viewport status. It distinguishes from siblings by advising use before browser_click and preference over screenshot for state verification.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly advises when to use (before browser_click, instead of screenshot) and mentions scope limitation and modal detection. Lacks explicit 'when not to use' but context is clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
browser_searchA
Grep-like element search across the current page. by: 'text' (literal substring), 'regex', 'role', 'ariaLabel', 'selector' (CSS). Returns results[] sorted by confidence descending — pass results[0].selector to browser_click. Pagination via offset/maxResults. Caveats: Use browser_overview for broad discovery; use browser_search when you know specific text or role to target. Typed errors: code:'BrowserSearchNoResults' (broaden the by:'text' substring or relax the by:'regex' pattern; switch to browser_overview to enumerate selectors), code:'BrowserSearchTimeout' (reduce maxResults / narrow scope), code:'ScopeNotFound' (the scope CSS selector did not match — verify the selector or omit scope), code:'BrowserNotConnected' (call browser_open first, or browser_open({launch:{}}) to auto-spawn).
| Name | Required | Description | Default |
|---|---|---|---|
| by | Yes | Search axis: text/regex/role/ariaLabel/selector | |
| port | No | Chrome/Edge CDP remote debugging port. | |
| scope | No | CSS selector to limit the search scope. | |
| tabId | No | Tab ID from browser_open. Omit to use the first page tab. | |
| offset | No | Offset into the result set (default 0). | |
| include | No | Optional response-shape opt-in. `['envelope']` returns the self-documenting envelope (`_version` / `data` / `as_of` / `confidence`). `['raw']` forces raw shape (overrides DESKTOP_TOUCH_ENVELOPE=1 server default). Default behaviour is raw shape (compat with existing clients). | |
| pattern | Yes | Pattern to match against the chosen axis. | |
| maxResults | No | Max results returned (default 50). | |
| visibleOnly | No | Only visible elements (default true). Set false to include hidden ones with confidence penalty. | |
| caseSensitive | No | Case-sensitive matching for text/regex (default false). | |
| inViewportOnly | No | Only currently-in-viewport elements (default false). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description is transparent about result sorting ('confidence descending'), pagination, and options like visibleOnly and inViewportOnly with behavior notes. It mentions error codes and conditions. Lacks an explicit statement of non-destructiveness but implies it.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Compact and well-structured: purpose, axes, result usage, pagination, caveats, then error codes. Every sentence serves a purpose with no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a complex tool with 11 parameters and no output schema, the description covers core functionality, error handling, and usage context. Could include a sample result structure but sufficient given the chaining hint.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, baseline 3. Description adds value beyond schema by explaining search axes with examples (literal substring, CSS), result chaining, and parameter adjustments implied in error codes (e.g., 'broaden the by:text substring').
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states this is a 'Grep-like element search' with specific axes (text, regex, role, ariaLabel, selector) and explains result usage ('pass results[0].selector to browser_click'). It distinguishes from sibling browser_overview for broad discovery.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly advises when to use this tool ('when you know specific text or role to target') vs browser_overview for broad discovery. Also provides detailed error code remedies, guiding the agent on corrective actions (e.g., broaden substring, reduce maxResults).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
click_elementA
Invoke a UI element by name or automationId via UIA InvokePattern — no screen coordinates needed. The server auto-guards using windowTitle (verifies identity, foreground, modal) and returns post.perception.status. Prefer over mouse_click for buttons, menu items, and links in native Windows apps. Use desktop_discover first to discover automationIds. Pass fixId from a suggestedFix to re-target after window identity drift. lensId is optional for advanced pinned-lens use. Caveats: Typed errors: code:'InvokePatternNotSupported' — the control does not expose InvokePattern, fall back to mouse_click; code:'ElementDisabled' — the element is in a disabled state, re-check preconditions before retry; code:'GuardFailed' — read the perception envelope (attention / guard fields) and choose recovery (re-focus, wait, or pass the suggestedFix.fixId on the next call). Some custom controls do not expose InvokePattern at all; fall back to mouse_click for those.
| Name | Required | Description | Default |
|---|---|---|---|
| hwnd | No | Direct window handle ID (takes precedence over windowTitle). String to avoid 64-bit precision issues. | |
| name | No | Element name/label (partial match, case-insensitive) | |
| fixId | No | Approve a pending suggestedFix (one-shot, 15s TTL). | |
| lensId | No | Optional perception lens ID. Guards (safe.keyboardTarget, target.identityStable) are evaluated before clicking, and a perception envelope is attached to post.perception on success. | |
| include | No | Optional response-shape opt-in. `['envelope']` returns the self-documenting envelope (`_version` / `data` / `as_of` / `confidence`). `['raw']` forces raw shape (overrides DESKTOP_TOUCH_ENVELOPE=1 server default). Default behaviour is raw shape (compat with existing clients). | |
| narrate | No | Narration level. rich includes UIA or browser state diff when supported. | minimal |
| controlType | No | Control type filter, e.g. 'Button', 'MenuItem' | |
| windowTitle | Yes | Partial window title of the target window. Use '@active' for the current foreground window. | |
| automationId | No | Exact AutomationId of the element |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations, but description fully covers behavior: auto-guard with windowTitle, returns perception status, lists all typed errors with recovery actions. Mentions custom controls may not support InvokePattern.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Well-structured: main purpose, then usage advice, then caveats and error handling. No unnecessary repetition; every sentence adds value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, description adequately explains return (perception status, envelope). Covers prerequisites (desktop_discover), error recovery, and optional parameters. Complete for a complex tool with 9 parameters.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage 100%, but description adds rich context: explains fixId TTL, lensId guard evaluation, include options, narrate defaults, controlType filter, windowTitle '@active', and hwnd precedence. Goes beyond schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Clearly describes invoking a UI element via UIA InvokePattern without coordinates. Distinguishes from mouse_click and browser_click by specifying native Windows apps and fallback conditions.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly says prefer over mouse_click for buttons, menu items, links. Advises to use desktop_discover first for automationIds. Provides detailed fallback instructions for errors like InvokePatternNotSupported, ElementDisabled, GuardFailed.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
clipboardA
Read or write the Windows clipboard. action='read' returns current text content (empty string if non-text). action='write' replaces clipboard with given text and verifies delivery by reading the clipboard back and comparing the bytes (UTF-16LE) for exact equality. Caveats: Non-text clipboard payloads (images, files) return empty string on read. Reading text larger than about 8MB (roughly 4 million characters, far above anything this tool can write) is refused immediately rather than copied. Calls give up after 4s (read) / 5s (write) — the usual cause is that the application owning the clipboard has stopped responding, which blocks the clipboard for every application on the machine, so retrying does not help until it recovers or is closed. A write that gives up leaves the clipboard in an indeterminate state (it may still complete once that application recovers), unlike code:'ClipboardWriteNotDelivered' where the write is known not to have landed — re-read before relying on the contents. Overwrites existing clipboard content on write. action='write' delivery-verification failure returns code:'ClipboardWriteNotDelivered' — typical causes: a third-party clipboard manager intercepts SetClipboardData, DLP / endpoint protection blocks the payload, RDP / Citrix clipboard transcoding strips the text, or another process clears the clipboard between Set and the read-back. Recovery: retry the write, or fall back to keyboard(action='type', use_clipboard=false) for short text. On builds without the native addon (backend:'powershell') writes are additionally capped at about 12000 characters and return code:'ClipboardWriteTooLargeForFallback' above it. Diagnostics: every response reports backend:'native'|'powershell' — the implementation that served the call. A successful write adds postCloseChecked: whether the read that catches a clipboard manager swapping the payload actually ran (native uses a separate second read; powershell's single read-back is that read), plus postCloseSkipReason when it did not run. On backend:'native' only, a successful write also reports sequenceAfterWrite (a Windows clipboard sequence number, for diagnosis only — the delivery verdict is always the byte comparison) and, when the post-close read alone confirmed the write, inSessionReadable:false. Examples: clipboard({action:'write', text:'hello'}) → write+verify; clipboard({action:'read'}) → returns current text.
| Name | Required | Description | Default |
|---|---|---|---|
| text | No | Text to place on the clipboard | |
| action | Yes | Action selector — one of: read, write. Per-action required fields are enforced at call time (see the tool description); this flat schema lists every action's fields as optional. | |
| include | No | Optional response-shape opt-in. `['envelope']` returns the self-documenting envelope (`_version` / `data` / `as_of` / `confidence`). `['raw']` forces raw shape (overrides DESKTOP_TOUCH_ENVELOPE=1 server default). Default behaviour is raw shape (compat with existing clients). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must reveal all behavioral traits. It discloses timeouts (4s/5s), failure codes (ClipboardWriteNotDelivered), indeterminate state after a timed-out write, backend differences (native vs powershell), delivery verification, and diagnostic fields. This is exemplary transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Despite its length, the description is front-loaded with the core purpose and uses clear sections (Caveats, Recovery, Diagnostics) to structure dense information. Every sentence adds actionable detail, so the length is justified rather than wasteful.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, the description thoroughly covers return behavior (empty string for non-text), error conditions, environment variations, and diagnostics. It anticipates edge cases like clipboard managers and RDP transcoding, and offers recovery steps. No significant gaps remain.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, but the description adds critical semantics beyond it: the exact meaning of each action, the 8MB read limit, 12000-char fallback limit, and the delivery-verification process. It also explains the 'include' parameter's effect indirectly through response shape context, though not named.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with 'Read or write the Windows clipboard,' clearly specifying the verb and resource. It further distinguishes the two actions ('read' returns text; 'write' replaces and verifies) and avoids confusion with sibling tools like keyboard by focusing on clipboard interaction.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides explicit alternative guidance for failed writes: 'fall back to keyboard(action="type", use_clipboard=false) for short text.' It also warns when retrying is futile (unresponsive clipboard owner) and when reading non-text payloads yields empty strings, effectively defining when-not-to-use.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
desktop_stateA
Purpose: Read-only observation of the current desktop state. Returns focused window/element, modal flag, attention signal from Auto Perception. Phase 4 absorbs former get_active_window / get_cursor_position / get_screen_info / get_document_state via include* flags. Details: Always returns: focusedWindow (title, hwnd, processName), focusedElement (name, type, value, automationId), cursorPos {x,y}, cursorOverElement (name, type), cursorOverWindow, hasModal (boolean), pageState ('ready'|'loading'|'dialog'), attention, visibleWindows count. Optional fields (default off): includeCursor:true → cursor {x,y,monitorId} (richer than cursorPos). includeScreen:true → screen {virtualScreen, displays[], displayCount, primaryIndex}. includeDocument:true → document {url, title, readyState, selection, scroll, viewport} via CDP (silently omitted on non-Chromium foreground). includeSessionContext:true (or include:['sessionContext']) → sessionContext {origin, consoleSessionId, sessionLabel, sessionState, ownWinStation} for Terminal Services session classification (ADR-017, observability-only). Chromium: cursorOverElement is null (UIA sparse); focusedElement may fall back to CDP document.activeElement; hints.focusedElementSource reports which path produced the row ('view' = engine-perception latest_focus, 'uia' = direct UIA query, 'cdp' = document.activeElement). Does NOT enumerate descendants — use desktop_discover for actionable entity list and window list. Prefer: Use after each action to confirm state. Cheapest observation tool — cheaper than any screenshot. attention='ok' means safe to proceed; other values require recovery (see suggest[]). Set include* flags only when you need the extra data (each adds one syscall or CDP round-trip). Caveats: Cannot detect non-UIA elements (custom-drawn UIs, game overlays). hasModal only detects modal dialogs exposed via UIA — browser alert/confirm dialogs may not appear here. includeDocument requires browser_open (CDP active); silently omitted otherwise with hints.documentUnavailable.
| Name | Required | Description | Default |
|---|---|---|---|
| port | No | CDP port for includeDocument (default 9222). | |
| tabId | No | Optional CDP tab id for includeDocument; omit for the focused tab. | |
| include | No | Optional response-shape opt-in. `['envelope']` returns the self-documenting envelope (`_version` / `data` / `as_of` / `confidence`). `['raw']` forces raw shape (overrides DESKTOP_TOUCH_ENVELOPE=1 server default). Default behaviour is raw shape (compat with existing clients). | |
| includeCursor | No | When true, add a richer `cursor` field with monitor index alongside the lightweight `cursorPos`. Phase 4: absorbs former get_cursor_position. Default false. | |
| includeScreen | No | When true, add a `screen` field with all connected display info (resolution, position, DPI, scale). Phase 4: absorbs former get_screen_info. Default false. Use the displayId values returned here in screenshot / window_dock(action='dock'). | |
| includeDocument | No | When true, add a `document` field with the focused Chrome tab's url, title, readyState, selection, and scroll position via CDP. Phase 4: absorbs former get_document_state. Default false. Requires browser_open (CDP active); silently omitted on non-Chromium foreground. | |
| includeSessionContext | No | When true, add a `sessionContext` field with the Terminal Services session classification (origin, consoleSessionId, sessionLabel: 'console'|'rdp'|'other', sessionState: 'active'|'connected'|'disconnected'|'locked'|'unknown', ownWinStation). Default false. Equivalent to `include: ['sessionContext']`. Per ADR-017: observability-only — does not gate input. `sessionState: 'locked'` is a heuristic (active + foreground=null + previous sample within 60s saw a non-null foreground); treat it as a generic input-pause signal — it can also fire on secure-desktop transitions (UAC prompt, Credential UI), where the user-visible state is not strictly 'locked' but input is equally unavailable to this session. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description fully discloses behavior: always-returned fields, optional fields, Chromium-specific quirks (cursorOverElement null, fallback to CDP), limitations (non-UIA elements, modal detection), and sessionState heuristics. It is thoroughly transparent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is structured with Purpose, Details, Prefer, and Caveats sections, front-loading key info. While lengthy, every section adds essential context; slight wordiness is justified given the tool's complexity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite no output schema, the description thoroughly explains all return fields, optional behaviors, and edge cases (e.g., includeDocument silently omitted). It covers prerequisites (browser_open), limitations, and interaction with siblings, making it complete for agent decision-making.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
All parameters have schema descriptions, achieving 100% coverage. The description adds value by explaining absorbed tools (e.g., includeCursor absorbs get_cursor_position) and nuances (includeSessionContext's ADR-017 context). It enhances understanding beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Read-only observation of the current desktop state.' It lists specific return fields and distinguishes from siblings like desktop_discover and screenshot. The verb 'observe' and resource 'desktop state' are explicit.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit guidance: 'Use after each action to confirm state.' It notes cost (cheapest), mentions when to use optional flags, and suggests alternatives (desktop_discover for actionable lists). It also explains attention signals and conditions for recovery.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
excelA
Purpose: Author and run VBA macros against Excel via COM late binding (ADR-015). Headline differentiator against Claude for Excel which writes formulas but cannot run VBA.
Details: action='run_vba' authors a Sub in a fresh workbook, saves into the managed Trusted Location (%LOCALAPPDATA%\desktop-touch-mcp\trusted-vba), and Application.Run the macro. Requires HKCU AccessVBOM=1 + VBAWarnings=1 + a registered Trusted Location (all configured by node scripts/enable-access-vbom.mjs). Trust setup: Excel must restart after the CLI runs (values cached at process start). action='check_access_vbom' is a read-only preflight returning {trusted, lockedByPolicy, scope}.
Prefer: Run check_access_vbom first when a workflow depends on macro execution; the remediation hint pre-empts an opaque HRESULT 0x800a03ec failure inside run_vba.
Caveats: macroName MUST appear as Sub <name>(...) in code (else VbaMacroNotFound). VBA Editor UI is structurally bypassed — no UIA tree walk needed. Excel COM is STA: each call serialises through the bridge's worker thread, so long-running macros block subsequent excel() calls on the same MCP server.
Examples:
excel({action:'check_access_vbom'}) → {trusted:true, scope:'hkcu'}
excel({action:'run_vba', code:'Sub DesktopTouchAdHoc()\n Range("A1").Value = "Hello"\nEnd Sub'}) → {ok:true, workbookPath:'...\trusted-vba\dt_vba_.xlsm'}
excel({action:'run_vba', code:'Sub Demo()\n MsgBox "hi"\nEnd Sub', macroName:'Demo', visible:true}) → demo recording path
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description fully discloses behavioral traits: required registry keys (AccessVBOM=1, VBAWarnings=1), Trusted Location setup, need for Excel restart, COM STA serialization causing blocking, and the requirement that macroName must match the Sub name in code. Failure modes (HRESULT 0x800a03ec) are also noted.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with clear sections (Purpose, Details, Prefer, Caveats, Examples). It is comprehensive but each sentence adds necessary information, avoiding redundancy. Front-loading the purpose and distinction aids quick understanding.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (VBA execution, COM, permissions), the description covers all essential aspects: purpose, setup requirements, prevalidation, caveats about naming and blocking, and complete examples with return values. No gaps remain despite the lack of an output schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema is empty with additionalProperties: true, so it provides no parameter definitions. The description compensates by defining the key parameters (action, code, macroName, visible) and their meanings, including constraints like macroName must match Sub name. Examples show expected parameter combinations and their effects.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the purpose: author and run VBA macros against Excel via COM late binding. It distinguishes from 'Claude for Excel' which writes formulas but cannot run VBA. The two actions run_vba and check_access_vbom are explicitly defined, making the tool's functionality unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides direct guidance: 'Prefer: Run check_access_vbom first when a workflow depends on macro execution'. It explains the preflight check and remediation for failures. Examples illustrate correct usage for both actions, clarifying when to use each.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
focus_windowA
Bring a window to the foreground by partial title match (case-insensitive). Use when a tool does not accept a windowTitle param, or when you need to switch focus before a sequence of actions. Use chromeTabUrlContains to activate a specific Chrome/Edge tab by URL substring before focusing — only the active tab's title appears in the windows list. If CDP is unavailable, chromeTabUrlContains is silently skipped — check response.hints.warnings. Returns WindowNotFound if no match exists; call desktop_discover to see available titles. Caveats: On some apps focus may be immediately stolen back (modal dialogs, UAC prompts) — verify with desktop_state after focusing. Win11 foreground refusal (UIPI cross-elevation / admin-only target / call from a background process or service) returns code:'ForegroundRestricted' ok:false instead of silently failing — recover by switching to a tool that does not require foreground transfer: desktop_act / click_element use UIA InvokePattern (no foreground needed); keyboard BG path bypasses foreground for terminal-class targets only (Windows Terminal / cmd / PowerShell — keyboard with windowTitle on non-terminal apps still hits the same ForegroundRestricted refusal). browser_* tools target by tabId/selector, not windowTitle.
| Name | Required | Description | Default |
|---|---|---|---|
| title | Yes | Partial window title to search for (case-insensitive) | |
| cdpPort | No | CDP port for chromeTabUrlContains (default 9222) | |
| include | No | Optional response-shape opt-in. `['envelope']` returns the self-documenting envelope (`_version` / `data` / `as_of` / `confidence`). `['raw']` forces raw shape (overrides DESKTOP_TOUCH_ENVELOPE=1 server default). Default behaviour is raw shape (compat with existing clients). | |
| forceFocus | No | When set, use AttachThreadInput-based foreground escalation on the first attempt. When omitted (default), focus_window first tries the standard SetForegroundWindow path and auto-escalates to force-focus only if Win11 refused the default attempt (issue #197). Override env: DESKTOP_TOUCH_FORCE_FOCUS=1 sets the implicit default to true. If both default and force paths fail, focus_window now returns ok:false code:'ForegroundRestricted' instead of the previous silent ok:true with windowChanged:false. | |
| chromeTabUrlContains | No | When set, activate the Chrome/Edge tab whose URL contains this substring before focusing the window. Requires Chrome/Edge running with --remote-debugging-port (default 9222). Use this when the target is a Chrome tab that is not currently active — the active tab title is the only one visible in the window title list. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description carries full burden. It discloses case-insensitivity, partial match, CDP availability (silently skipped), focus stealing, Win11 foreground refusal with specific error code, and recovery options. No contradictions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Description is relatively long but every sentence adds value. Front-loaded with core purpose. Well-structured, no redundancy. Concisely covers all necessary aspects.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given complexity (5 params, no output schema), description covers usage, alternatives, failure modes, caveats, and error recovery. Complete for an AI agent to select and invoke correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%. Description adds context beyond schema: explains when to use chromeTabUrlContains ('when the target is a Chrome tab that is not currently active'), details forceFocus behavior (auto-escalation), and include parameter options. Adds meaning beyond schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description states 'Bring a window to the foreground by partial title match (case-insensitive)', which is a specific verb and resource. It distinguishes from siblings by mentioning when to use focus_window vs chromeTabUrlContains and other tools like desktop_act.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicit guidance: 'Use when a tool does not accept a windowTitle param, or when you need to switch focus before a sequence of actions.' Also details when to use chromeTabUrlContains and provides alternatives for recovery from ForegroundRestricted (desktop_act, click_element, keyboard).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
keyboardA
Purpose: Send keyboard input to a window: 'type' for text, 'press' for key combos, 'sequence' for atomic multi-step chords. Details: action='type' inserts text (auto-clipboard for non-ASCII, bypassing IME conversion). action='press' sends key combos like 'ctrl+c'/'alt+tab'. action='sequence' runs ordered steps in one keyboard lock — use for Alt+letter, letter mnemonic chains where intermediate tool calls would close the menu. windowTitle or hwnd is REQUIRED (blank/whitespace counts as neither) — the server focuses and auto-guards that window (identity, foreground, modal) first, and a call with neither stops with DestinationRequired before any key is sent. Use windowTitle:'@active' to aim at the foreground window on purpose; an hwnd naming a titleless window works only while that window is already foreground. DESKTOP_TOUCH_REQUIRE_DESTINATION=0 downgrades the stop to a warning. Prefer: Set lensId for perception guards. Use desktop_act({action:'setValue'}) for UIA ValuePattern text fields. Caveats: win+r/win+x/win+s/win+l blocked. action='type' does not handle CJK IME composition — use use_clipboard=true or desktop_act({action:'setValue'}); neither lands while an IME composition is pending — commit or cancel it first. hints.clipboard reports the backend and whether the clipboard was restored. Non-ASCII text (CJK / emoji / diacritics / smart-quote-class punctuation) auto-clipboards to prevent silent-drop and Chrome accelerator hijack; pass forceKeystrokes:true to disable. Background (PostMessage/WM_CHAR) auto-engages for terminal-class windows (Windows Terminal / cmd / PowerShell); DTM_BG_AUTO=1 enables globally. Foreground non-terminal type runs a per-chunk leash; user focus-steal mid-stream aborts with FocusLostDuringType + context.typed/remaining; pass abortOnFocusLoss:false to disable. BG type verifies WM_CHAR via UIA TextPattern read-back; mismatch returns BackgroundInputNotDelivered (see SUGGESTS for false-positive notes). BG press read-back is scoped to terminal-class + enter/tab/arrow; other combos return verifyDelivery:'unverifiable', failure returns BackgroundKeyNotDelivered. action='sequence' is FG-only (BG/foreground_flash schema-rejected); emits verifyDelivery:'focus_only'; mid-loop focus theft returns MenuFocusLostMidSequence + context.remaining: Step[]. Win11 FG refusal returns ForegroundRestricted — terminal-class targets auto-engage BG; non-terminal switch to desktop_act / click_element. Examples: keyboard({action:'type', text:'hello', windowTitle:'Untitled - Notepad'}) → text injected (guarded) keyboard({action:'type', text:'hello', windowTitle:'@active'}) → typed into the foreground window keyboard({action:'press', keys:'ctrl+c', windowTitle:'Untitled - Notepad'}) → copy keyboard({action:'press', keys:'escape', windowTitle:'Dialog'}) → dismiss dialog keyboard({action:'sequence', steps:[{keys:'alt+i', gapMs:100},{keys:'m'}], windowTitle:'Microsoft Visual Basic'}) → Insert > Module (atomic)
| Name | Required | Description | Default |
|---|---|---|---|
| hwnd | No | Direct window handle ID (takes precedence over windowTitle). Obtain from get_windows response (hwnd field). String type to avoid 64-bit precision issues. | |
| keys | No | Key combo string, e.g. 'ctrl+c', 'alt+tab', 'enter', 'ctrl+shift+s'. Note: win+r, win+x, win+s, win+l are blocked for security. | |
| text | No | The text to type (max 10,000 characters) | |
| fixId | No | Approve a pending suggestedFix (one-shot, 15s TTL). Pass the fixId returned by a previous failed keyboard(action='type') to re-attempt with guard-validated args. | |
| steps | No | Ordered list of key-press steps. Min 1, max 16. Total duration must not exceed 5000ms (excludes settleMs and focus acquisition). N=1 is allowed but inherits the sequence verification contract (hints.verifyDelivery.status='focus_only'); if you want the stricter keyboard:press contract, call keyboard({action:'press', keys}) directly (issue #278, matrix doc §3.1). | |
| action | Yes | Action selector — one of: type, press, sequence. Per-action required fields are enforced at call time (see the tool description); this flat schema lists every action's fields as optional. | |
| lensId | No | Optional perception lens ID. Guards (safe.keyboardTarget) are evaluated before typing, and a perception envelope is attached to post.perception on success. | |
| method | No | Input method. background = WM_CHAR PostMessage (no focus change); foreground = SendInput (current default); auto = pick automatically. | auto |
| include | No | Optional response-shape opt-in. `['envelope']` returns the self-documenting envelope (`_version` / `data` / `as_of` / `confidence`). `['raw']` forces raw shape (overrides DESKTOP_TOUCH_ENVELOPE=1 server default). Default behaviour is raw shape (compat with existing clients). | |
| narrate | No | Narration level. rich includes UIA or browser state diff when supported. | minimal |
| settleMs | No | Milliseconds to wait before checking post-action state. | |
| forceFocus | No | Bypass Windows foreground-stealing protection before focusing. | |
| replaceAll | No | When true, send Ctrl+A to select all existing text before typing. Equivalent to Ctrl+A → keyboard(action='type') in one call (requires field already focused). Default false. | |
| trackFocus | No | Detect if focus was stolen after the action. | |
| forceImeOff | No | Issue #245 系統②: when true, query the target window's IME open-status via Imm32 before typing; if ON, switch OFF for the duration of this call and restore the prior state in `finally`. Prevents silent romaji conversion when the user's Japanese IME is active but the LLM is typing ASCII commands. Requires `windowTitle` or `hwnd` (otherwise no target to query). Default false — existing use_clipboard auto-promotion still handles non-ASCII symbols transparently. No-op when the addon predates the IMM bridge (call proceeds with whatever IME state is in effect). | |
| windowTitle | No | Partial title of the window that should receive keyboard input. | |
| use_clipboard | No | If true, copy text to clipboard and paste with Ctrl+V instead of simulating keystrokes. Use this when typing URLs, paths, or ASCII text into apps with Japanese IME active — pasted text is not run through IME conversion. Note this does not help while an IME composition is already in progress: the paste keystroke is consumed by the IME and nothing is inserted, so commit or cancel the composition first. Your clipboard is replaced for the duration of the call and put back afterwards; hints.clipboard reports which backend served the paste and whether the restore ran. On builds without the native addon this path is capped at about 12000 characters and fails with code:'ClipboardWriteTooLargeForFallback' above it. Default false. | |
| forceKeystrokes | No | When true, always use keystroke mode even if text contains non-ASCII content (CJK, emoji, diacritics, em-dash, smart quotes, etc.) that would normally trigger auto-clipboard. Default false — auto-clipboard is enabled. | |
| abortOnFocusLoss | No | Focus Leash Phase B: when true, the foreground keystroke send is split into chunks (default 8 chars; override via DTM_LEASH_CHUNK_SIZE env) and the target window's foreground state is verified between chunks. If the user grabs focus mid-stream, the call aborts and returns FocusLostDuringType with context.typed (chars delivered to target) and context.remaining (unsent tail) so the caller can re-focus and retry the unsent portion. Default: true when windowTitle is provided, false otherwise. Has no effect on the clipboard path (atomic Ctrl+V) or the BG (WM_CHAR) path (HWND-targeted, foreground-independent). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description fully discloses behavioral nuances: auto-clipboard for non-ASCII, background PostMessage behavior, focus leash chunking, verification contracts, error codes like FocusLostDuringType and BackgroundInputNotDelivered, and platform-specific restrictions. It also details the handling of IME composition and clipboard restoration, leaving no major behavior undocumented.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Although lengthy, the description is well-organized with clear sections (Purpose, Details, Examples) and each sentence carries actionable information. The structure front-loads the core purpose and then systematically covers edge cases, methods, and examples, making it dense but appropriately sized for a complex tool.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers all aspects needed for correct invocation: parameter behaviors, error codes, verification modes, platform restrictions, and practical examples. Without an output schema, it still explains what results to expect (e.g., hints.verifyDelivery, context.typed/remaining) and how to handle failures, making it complete for an agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already covers all 19 parameters with descriptions, but the tool description adds significant extra meaning: for example, the interplay between use_clipboard and IME, the semantics of forceKeystrokes, abortOnFocusLoss, and forceImeOff, and how steps relate to sequence verification. This goes well beyond the schema's individual parameter descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Send keyboard input to a window' and enumerates three specific actions (type, press, sequence) with examples. It distinguishes itself from sibling tools by explicitly noting alternatives like desktop_act and click_element for different scenarios.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit guidance on when to use this tool versus alternatives, such as 'Use desktop_act({action:'setValue'}) for UIA ValuePattern text fields' and 'non-terminal switch to desktop_act / click_element'. It also explains when to use background vs foreground methods and cautions about IME and focus-loss scenarios.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
key_lockerA
Purpose: Manage credentials the terminal autofills for you (SSH key passphrases, sudo / login passwords). Secrets are entered once into the locker's own secure dialog and stored encrypted on this machine (Windows DPAPI, current user); they are NEVER shown to the assistant or sent to this tool.
Details: action='save' pre-seeds a credential for a binding URI (ssh://user@host:22, sudo://host/root, https-cred://host:443, sshkey:SHA256:…): it opens the locker's secure entry dialog and stores the secret; the first save also shows a one-time enable confirmation. action='list' shows saved bindings (metadata only, never secrets). action='forget' deletes a binding and its secret. action='set_policy' toggles per-binding autofill confirmation. action='status' reports whether the locker is enabled (consent) and how many bindings exist. action='launch_console' opens (or reuses) an autofill-capable anchored pane and returns {paneId, windowTitle} — by default a new tab in the user's current Windows Terminal window (host:'classic' opens a dedicated classic console window instead).
Prefer: Autofill is AUTOMATIC when a bound command triggers a credential prompt in the terminal — there is no manual fill action. But autofill ONLY fires in a pane opened by launch_console (a pre-existing terminal is never autofilled): to autofill, first launch_console, then run the ssh / sudo command with terminal({action:'run'|'send', paneId}) — pass the paneId field, not the windowTitle. Keep the returned paneId; there is no pane-listing action, but launch_console with fresh:false reuses the most-recent pane and returns its paneId again. Use save to enroll, list/status to inspect.
Caveats: Windows-only. The anchored pane defaults to a Windows Terminal tab (autofill and terminal reads operate while that tab is the ACTIVE tab — switching away pauses them safely); host:'classic' opens a dedicated classic console window instead, and is the retry when Windows Terminal is not installed (KeyLockerWtUnavailable). The human can also see and type into the pane. Enabling the locker (first save or launch_console) grants BOTH credential autofill AND the ability for the assistant to launch a locker-owned pane. Disable the whole feature with DESKTOP_TOUCH_DISABLE_KEY_LOCKER=1. An ssh save needs the host key already in known_hosts (connect once first). API-token / env-var credentials are not supported yet.
Examples:
key_locker({action:'status'}) → {consentAccepted:false, disabled:false, bindingCount:0}
key_locker({action:'save', uri:'sudo://buildbox/root'}) → opens the secure dialog → {captured:true}
key_locker({action:'list'}) → {bindings:[{displayUri:'sudo://buildbox/root', scheme:'sudo', …}]}
key_locker({action:'launch_console'}) → {paneId:'wt:31264:13322426700123', windowTitle:'dtm-locker-console-…'} → then terminal({action:'send', paneId:'wt:31264:13322426700123', input:'ssh user@host'})
key_locker({action:'launch_console', host:'classic'}) → {paneId:'12345678', windowTitle:'dtm-locker-console-…'} (dedicated classic console window)
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description fully covers behavioral traits: secrets are encrypted via Windows DPAPI, never shown to the assistant, autofill is automatic only in specific panes, fresh:false reuses pane, and the need for known_hosts. It also discloses caveats like Windows-only, classic console fallback, and disablement via environment variable.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but well-structured with headings (Purpose, Details, Prefer, Caveats, Examples). It front-loads the purpose and then organizes details logically. While comprehensive, minor trimming could improve conciseness without losing essential guidance.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema and only 0-parameter input schema, the description provides complete context: return value examples, detailed usage patterns, prerequisites, error conditions (e.g., KeyLockerWtUnavailable), and operational constraints. No gaps remain.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 0 defined properties with additionalProperties: true, so the description carries full burden. It explains the format of action, uri, and other fields with examples, providing meaning that the schema lacks entirely.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description explicitly states the purpose: 'Manage credentials the terminal autofills for you' and lists the specific credential types. It clearly distinguishes from siblings like terminal and clipboard by focusing on credential management and autofill, not general terminal interaction.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides detailed when-to-use and when-not-to-use guidance, e.g., 'Prefer: Autofill is AUTOMATIC' but 'only fires in a pane opened by launch_console'. It also gives alternatives like using save, list, and status for inspection, and prerequisites such as known_hosts for SSH. The extensive examples further clarify usage.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
mouse_clickA
Click at screen coordinates. Normally pass windowTitle so the server auto-guards the click (verifies target identity, foreground, coordinate is inside the target rect) and returns post.perception without a confirmation screenshot. origin+scale from dotByDot=true screenshots are converted to screen coords before guarding. doubleClick:true for double-click; tripleClick:true for triple-click (selects a full line of text). Prefer click_element (UIA) for native apps, prefer browser_click for Chrome. Examples: mouse_click({windowTitle:'Notepad', x:200, y:150}) // guarded — post.perception.status='ok'. mouse_click({x:100, y:100}) // unguarded — post.perception.status='unguarded'. If a guard failure returns a suggestedFix, pass its fixId to approve the fix: mouse_click({fixId:'fix-...'}) // one-shot, expires in 15s. lensId is optional and only for advanced pinned-target workflows; omit it for normal use. Caveats: origin+scale are meaningful ONLY with dotByDot=true screenshot responses. hints.verifyDelivery:{status:'delivered'|'focus_only'|'unverifiable', reason} reports the post-click observation in 3 values (focused-element shift, window-foreground change, or no signal). Win11 foreground refusal during the homing path (UIPI cross-elevation / admin-only target / call from a background process or service) returns code:'ForegroundRestricted' ok:false rather than landing the click on the wrong window — recover by switching to a tool that accepts windowTitle directly (click_element / desktop_act) — browser_* tools target by tabId/selector, not windowTitle. MouseClickNotDelivered is reserved-only (false-positive risk is too high to emit a typed code), so degradation is expressed via the 'unverifiable' status, not a separate error.
| Name | Required | Description | Default |
|---|---|---|---|
| x | Yes | X coordinate. Screen-absolute by default. When 'origin' is provided, treated as image-local (pixel position within the screenshot). | |
| y | Yes | Y coordinate. Screen-absolute by default. When 'origin' is provided, treated as image-local. | |
| hwnd | No | Direct window handle ID (takes precedence over windowTitle). Obtain from get_windows response (hwnd field). String type to avoid 64-bit precision issues. | |
| fixId | No | One-shot fix approval ID. If a previous mouse_click returned a suggestedFix, pass that fixId here to approve it. The server revalidates the fix and executes with corrected args. fixId expires in 15 seconds and can only be used once. | |
| scale | No | Scale factor from screenshot response (only when dotByDotMaxDimension caused a resize). Omit if the screenshot was 1:1. Only used when 'origin' is also provided. | |
| speed | No | Cursor movement speed in px/sec. 0 = instant. | |
| button | No | Mouse button to click | left |
| homing | No | Enable homing correction if the target window moved. | |
| lensId | No | Optional perception lens ID for advanced pinned-target workflows. When provided, guards are evaluated before clicking (safe.clickCoordinates, target.identityStable) and a perception envelope is attached to post.perception in the response. For normal use, omit lensId and pass windowTitle directly — Auto Perception handles tracking. | |
| origin | No | When set, (x,y) are image-local coords from a screenshot. Server converts to screen coords: screen_x = origin.x + x / (scale ?? 1), screen_y = origin.y + y / (scale ?? 1). Copy origin values directly from the screenshot response text. This eliminates manual coord math and prevents out-of-window clicks. | |
| include | No | Optional response-shape opt-in. `['envelope']` returns the self-documenting envelope (`_version` / `data` / `as_of` / `confidence`). `['raw']` forces raw shape (overrides DESKTOP_TOUCH_ENVELOPE=1 server default). Default behaviour is raw shape (compat with existing clients). | |
| narrate | No | Narration level. rich includes UIA or browser state diff when supported. | minimal |
| settleMs | No | Milliseconds to wait before checking post-action state. | |
| elementId | No | AutomationId of the UI element. | |
| forceFocus | No | Bypass Windows foreground-stealing protection before focusing. | |
| trackFocus | No | Detect if focus was stolen after the action. | |
| doubleClick | No | Whether to double-click | |
| elementName | No | Name or label of the UI element. | |
| tripleClick | No | Whether to triple-click (select a line of text). Takes precedence over doubleClick when both are true. | |
| windowTitle | No | Partial title of the target window. | |
| verifyDelivery | Yes | Parameter 'verifyDeliveryParam' from the Windows server schema. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description fully discloses behavioral traits: the guarding mechanism (verifies target identity, foreground, coordinate inside rect), return statuses ('ok', 'unguarded', 'unverifiable'), error conditions (ForegroundRestricted, MouseClickNotDelivered reserved), and verifyDelivery hint behavior. It also explains origin+scale conversion and lensId advanced workflow.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is lengthy but each sentence adds value. It is front-loaded with the core purpose and examples. However, it could be more structured (e.g., bullet points or sections) to improve readability. Slightly verbose but still effective.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 21 parameters, no output schema, the description covers behavioral aspects, output status, error handling, usage examples, and caveats. It addresses all likely agent questions, including edge cases like fixId expiration, origin+scale constraints, and Win11 restrictions. Highly complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, baseline 3. The description adds significant semantic value beyond the schema: explains how windowTitle triggers auto-guarding, how origin+scale convert image-local to screen coords, fixId expiry and one-shot nature, lensId purpose, and verifyDelivery status meanings. It also clarifies that doubleClick and tripleClick interactions (tripleClick precedence) and that speed=0 is instant.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Click at screen coordinates' and distinguishes guarded (with windowTitle) and unguarded clicks. It explicitly mentions double-click and triple-click behaviors. It also differentiates from sibling tools: 'Prefer click_element (UIA) for native apps, prefer browser_click for Chrome.'
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear when-to-use guidance: prefers click_element for UIA apps and browser_click for Chrome. It explains when to use windowTitle for auto-guarding vs unguarded clicks. It also covers fixId usage for suggested fixes, and warns about Win11 foreground restrictions and how to recover. Examples illustrate both guarded and unguarded scenarios.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
mouse_dragA
Click and drag from (startX, startY) to (endX, endY) holding the left mouse button — for sliders, drag-and-drop, canvas drawing, and window resizing. Pass windowTitle so the server auto-guards the start coordinate and returns post.perception. Examples: mouse_drag({windowTitle:'Notepad', startX:50, startY:50, endX:200, endY:200}). lensId is optional and only for advanced pinned-target workflows. Caveats: Left button only. Both start and endpoint are guarded. Cross-window and desktop drags are blocked by default — pass allowCrossWindowDrag:true to confirm intent; that refusal is code:'CrossWindowDragBlocked'. A drag starting in a tabbed application's tab strip returns code:'TabDragBlocked' — pass allowTabDrag:true when tearing off or rearranging a tab is intended. hints.verifyDelivery:{status:'delivered'|'focus_only'|'unverifiable', reason} reports the post-drop observation in the same 3-value shape as mouse_click. MouseDragNotDelivered is SUGGESTS-registered but reserved-only (not emitted) — degradation is expressed via the 'unverifiable' status rather than a typed code. Win11 foreground refusal (UIPI cross-elevation / admin-only target / call from a background process or service) returns code:'ForegroundRestricted' ok:false from the homing path.
| Name | Required | Description | Default |
|---|---|---|---|
| endX | Yes | ||
| endY | Yes | ||
| hwnd | No | Direct window handle ID (takes precedence over windowTitle). Obtain from get_windows response (hwnd field). String type to avoid 64-bit precision issues. | |
| speed | No | Cursor movement speed in px/sec. 0 = instant. | |
| homing | No | Enable homing correction if the target window moved. | |
| lensId | No | Optional perception lens ID. Guards and envelope same as mouse_click. | |
| startX | Yes | ||
| startY | Yes | ||
| include | No | Optional response-shape opt-in. `['envelope']` returns the self-documenting envelope (`_version` / `data` / `as_of` / `confidence`). `['raw']` forces raw shape (overrides DESKTOP_TOUCH_ENVELOPE=1 server default). Default behaviour is raw shape (compat with existing clients). | |
| narrate | No | Narration level. rich includes UIA or browser state diff when supported. | minimal |
| windowTitle | No | Partial title of the target window. | |
| allowTabDrag | No | When true, allow drags that start in the title-bar / tab-strip area of a tabbed app (Notepad, Terminal, Edge, Chrome, etc.). Default false — such drags are blocked because they detach the tab into a new window rather than moving the window. Pass true only when you intentionally want to rearrange or detach a tab. Note: active only when auto-guard is enabled (same scope as allowCrossWindowDrag). | |
| verifyDelivery | Yes | Parameter 'verifyDeliveryParam' from the Windows server schema. | |
| allowCrossWindowDrag | No | When true, allow dragging the endpoint into a different window or the desktop background. Default false — cross-window drags (including desktop/wallpaper) are blocked to prevent accidents. Pass true to confirm intent for deliberate cross-window or desktop-area drags. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description fully discloses critical behaviors: left-button only, start and endpoint guards, cross-window and tab drag restrictions with error codes, verifyDelivery status shape, and Win11 foreground refusal conditions. It even clarifies that MouseDragNotDelivered is reserved-only and not emitted, setting precise expectations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the core action and use cases, and every sentence provides valuable detail. However, it is a dense single paragraph without structural breaks, which could slightly reduce readability. Despite this, the length is justified by the tool's complexity and many behavioral caveats.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description is exceptionally complete for a complex input tool with no output schema. It covers primary purpose, specific use-cases, error codes, guarded behaviors, verification mechanisms, and example usage. It addresses the richness of the tool effectively, making it fully usable by an AI agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The description adds significant meaning beyond the schema by explaining the purpose of windowTitle ('auto-guards the start coordinate'), lensId ('advanced pinned-target workflows'), allowCrossWindowDrag, allowTabDrag, and verifyDelivery (delivery status shape). This complements the 71% schema coverage and clarifies otherwise ambiguous parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly identifies the tool as a mouse drag operation: 'Click and drag from (startX, startY) to (endX, endY) holding the left mouse button.' It also lists concrete use cases (sliders, drag-and-drop, canvas drawing, window resizing) and gives an example invocation, which distinguishes it from sibling tools like mouse_click and scroll.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly states when to use the tool ('for sliders, drag-and-drop, canvas drawing, and window resizing') and provides detailed caveats about blocked scenarios (cross-window drags, tab-strip drags) with specific opt-in flags. It also notes lensId is 'only for advanced pinned-target workflows,' giving clear guidance on when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
notification_showA
Show a Windows system tray balloon notification to alert the user. Use at the end of a long-running task so the user knows it finished without watching the screen. Caveats: toast の user reach は原理的に観測不能 (matrix §3.1 line 158 規範整合)。Focus Assist (Do Not Disturb) / Notifications-off setting / consent UI sink いずれも tool 側からは判別不能のため、successful response は常に hints.verifyDelivery を含む (status="unverifiable", reason="user_visible_side_effect_uninspectable", channel="win32_balloon_tip" — 全 double-quoted JSON literal)。caller は user 側の post-notification behavior (例: wait_until(focus_changes)) で間接観測することが望ましい。Uses System.Windows.Forms — no external modules needed.
| Name | Required | Description | Default |
|---|---|---|---|
| body | Yes | Notification body text | |
| title | Yes | Notification title | |
| include | No | Optional response-shape opt-in. `['envelope']` returns the self-documenting envelope (`_version` / `data` / `as_of` / `confidence`). `['raw']` forces raw shape (overrides DESKTOP_TOUCH_ENVELOPE=1 server default). Default behaviour is raw shape (compat with existing clients). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, but description fully discloses that hints.verifyDelivery is always included with unverifiable status, explains caveats about Focus Assist and DND, and mentions underlying technology (System.Windows.Forms).
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Contains Japanese text and technical references (e.g., 'matrix §3.1 line 158 規範整合') that may hinder clarity for an English-speaking AI; while informative, it could be more concise and better structured.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Comprehensive for a notification tool with no output schema: explains return behavior, limitations, and usage context, leaving no critical gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so baseline 3. Description adds context for the optional 'include' parameter but does not significantly enhance meaning beyond schema for required parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states 'Show a Windows system tray balloon notification to alert the user' with a specific verb and resource, and distinguishes from sibling tools by being the only notification tool.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly says 'Use at the end of a long-running task' and advises indirect observation via wait_until(focus_changes) for post-notification behavior, providing clear when-to-use and alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
run_macroA
Purpose: Execute multiple tools sequentially in one MCP call — eliminates round-trip latency for predictable multi-step workflows. Details: steps[] is an array of {tool, params} objects. Accepts all desktop-touch tools plus a special sleep pseudo-step: {tool:"sleep", params:{ms:N}} (max 10000ms per step). stop_on_error=true (default) halts on first failure. Max 50 steps. The LLM cannot inspect intermediate results during execution — all steps run to completion (or first error) before any output is returned. Prefer: Use for predictable fixed sequences (focus → sleep → type → screenshot). Do not use for conditional logic — return to the LLM between branches so it can inspect intermediate state. Caveats: If any step may fail conditionally (e.g. a dialog that may or may not appear), split the macro at that point. Each screenshot step within a macro incurs the same token cost as a standalone call. Examples: [{tool:'focus_window',params:{windowTitle:'Notepad'}},{tool:'sleep',params:{ms:300}},{tool:'keyboard',params:{action:'type',text:'Hello'}},{tool:'screenshot',params:{detail:'text',windowTitle:'Notepad'}}] [{tool:'browser_navigate',params:{url:'https://example.com'}},{tool:'wait_until',params:{condition:'element_matches',target:{by:'text',pattern:'Example Domain'}}}]
| Name | Required | Description | Default |
|---|---|---|---|
| steps | No | Ordered list of tool calls to execute sequentially (max 50 steps). | |
| include | No | Optional response-shape opt-in. `['envelope']` returns the self-documenting envelope (`_version` / `data` / `as_of` / `confidence`). `['raw']` forces raw shape (overrides DESKTOP_TOUCH_ENVELOPE=1 server default). Default behaviour is raw shape (compat with existing clients). | |
| stop_on_error | No | Stop execution on the first error (default true). Set false to collect all results. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Despite no annotations, the description discloses critical behavioral traits: the LLM cannot inspect intermediate results, all steps run to completion or first error, sleep has a max of 10000ms, stop_on_error defaults true, max 50 steps, and screenshots incur token cost. This fully compensates for missing annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with labeled sections (Purpose, Details, Prefer, Caveats, Examples) and front-loaded with the core purpose. Every sentence adds useful information; no redundancy or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity and lack of output schema, the description covers purpose, usage constraints, and side effects adequately. However, it does not explicitly state the shape of the return value (e.g., array of results or single result), which would improve completeness. Otherwise, it is thorough.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description adds value by explaining the sleep pseudo-step format, that params should match direct tool calls, and the optional 'include' parameter for response shaping. It does not add syntax details for each tool's params, but that would be excessive.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it 'executes multiple tools sequentially in one MCP call' to reduce latency, distinguishing it from sibling tools that perform single actions. The verb 'execute' and resource 'multiple tools' are specific, and the contrast with individual tools is evident.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicit guidance: 'Prefer: Use for predictable fixed sequences' and 'Do not use for conditional logic — return to the LLM between branches'. Also advises splitting macros at points of potential failure. Examples illustrate appropriate use cases.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
screenshotA
Purpose: Capture desktop, window, or region across detail levels (meta / text / image / som / ocr) and capture modes (normal / background). Details: detail='meta' (default) returns window titles+positions only (~20 tok/window, no image). detail='text' returns UIA actionable elements with clickAt coords, no image (~100-300 tok). detail='som' returns OCR-detected elements with IDs plus a Set-of-Marks annotated image delivered by-ref by default (bypasses UIA entirely). detail='ocr' returns Windows OCR words with screen-pixel clickAt coords (Phase 4: absorbs former screenshot_ocr — use when UIA is sparse and you want to force OCR unconditionally). detail='image' and detail='som' both return a cheap by-ref resource_link by default (no inline base64); pass confirmImage=true to also embed the inline image (the annotated bitmap for som). mode='background' captures hidden/minimised/occluded windows via PrintWindow (Phase 4: absorbs former screenshot_background) — pair with windowTitle/hwnd. dotByDot=true returns 1:1 pixel WebP; compute screen coords: screen_x = origin_x + image_x (or screen_x = origin_x + image_x / scale when dotByDotMaxDimension is set — scale printed in response). diffMode=true returns only changed windows after the first call (~160 tok). region={x,y,width,height} captures a sub-rectangle (Phase 4: absorbs former scope_element when paired with windowTitle/hwnd — discover element bounds via desktop_discover, then pass region here). Data reduction: grayscale=true (−50%), dotByDotMaxDimension=1280 (caps longest edge), windowTitle+region (sub-crop to exclude browser chrome — e.g. region={x:0, y:120, width:1920, height:900}). Prefer: Use meta to orient, text before clicking, dotByDot only when precise pixel coords are needed. Use detail='som' for native apps or games that do not expose UIA elements (UIA-Blind). Use detail='ocr' for OCR-only (skip UIA entirely). Use mode='background' when the target window must stay hidden or cannot be brought to foreground. Prefer browser_* tools for Chrome. Use diffMode after actions to confirm state changed. Only use image+confirmImage when text returned 0 actionable elements and visual inspection is genuinely required. Caveats: Default mode scales to maxDimension=768 — image pixels ≠ screen pixels; apply the scale formula before passing to mouse_click. Foreground detail='image' returns a by-ref resource_link by default; pass confirmImage=true to also receive inline pixels. diffMode requires a prior full-capture baseline (non-diff call or workspace_snapshot) — calling diffMode cold returns a full frame, not a diff. mode='background' requires windowTitle or hwnd, and only composes with detail in {'image','meta'} — detail='text'/'som'/'ocr' run only against foreground capture (the dispatcher rejects the conflicting combination). Passing mode='background' is itself the acknowledgement that image pixels are wanted, so confirmImage is NOT required for it (matches the former screenshot_background contract). fullContent=false enables legacy mode (faster but GPU windows may be black). detail='ocr' requires windowTitle or hwnd; first call may take ~1s (WinRT cold-start) and the matching OCR language pack must be installed. Examples: screenshot() → meta orientation of all windows screenshot({detail:'text', windowTitle:'Notepad'}) → clickable elements with coords screenshot({detail:'ocr', windowTitle:'PDF', ocrLanguage:'ja'}) → OCR words with screen-pixel coords screenshot({mode:'background', windowTitle:'Chrome', dotByDot:true, dotByDotMaxDimension:1280, grayscale:true}) → background-capture pixel-accurate Chrome screenshot({windowTitle:'Notepad', region:{x:0,y:120,width:600,height:400}}) → cropped sub-region (zoom into element after desktop_discover)
| Name | Required | Description | Default |
|---|---|---|---|
| hwnd | No | Direct window handle ID (takes precedence over windowTitle). Obtain from desktop_discover (windows[].hwnd). String type to avoid 64-bit precision issues. | |
| mode | No | Capture mode. 'normal' — default. Window-targeted captures (windowTitle / hwnd) use Win32 PrintWindow with automatic BitBlt fallback when PrintWindow returns no data or an all-black frame; the route used is reported in hints.captureSource. Fullscreen / displayId captures use BitBlt. 'background' — explicit Win32 PrintWindow capture, retained for back-compat and explicit selection. Requires windowTitle (or hwnd). Pair with fullContent for GPU-rendered apps. | normal |
| detail | No | Response detail level (omit to let the server pick a smart default): omitted — auto: 'image' when dotByDot/region/displayId is specified, else 'meta' 'meta' — window title + screen region only (~20 tok/window, cheapest) 'text' — UIA element tree as JSON with text values (~100-300 tok/window, no image) 'image' — actual screenshot pixels. Returns a cheap by-ref resource_link by default (no inline base64); pass confirmImage=true to ALSO embed the inline image. 'som' — Set-of-Marks elements + annotated image (bypasses UIA entirely). Returns the OCR elements[] plus a cheap by-ref resource_link by default (no inline base64); pass confirmImage=true to ALSO embed the annotated bitmap. 'ocr' — Windows OCR words with screen-pixel clickAt coords (Phase 4: absorbs former screenshot_ocr). Use when UIA returns no actionable elements (WinUI3 custom-drawn UIs, game overlays, PDF viewers). Note: detail='text' auto-falls back to OCR via ocrFallback='auto'; choose detail='ocr' only when forcing OCR unconditionally. | |
| region | No | Capture only this sub-region. Without windowTitle: virtual screen coordinates. With windowTitle: window-local coordinates — useful to exclude browser chrome (tabs/address bar). Example: windowTitle='Chrome', region={x:0, y:120, width:1920, height:900} skips the 120px browser chrome. | |
| include | No | Optional response-shape opt-in. `['envelope']` returns the self-documenting envelope (`_version` / `data` / `as_of` / `confidence`). `['raw']` forces raw shape (overrides DESKTOP_TOUCH_ENVELOPE=1 server default). Default behaviour is raw shape (compat with existing clients). | |
| diffMode | No | Layer diff mode — compares each window against the buffered previous frame. First call = full I-frame (all windows). Subsequent calls = only changed windows (P-frame). Implicitly enables dotByDot. Best used with windowTitle=undefined to snapshot all windows. | |
| dotByDot | No | 1:1 pixel mode — no scaling, WebP compression. Window captures include 'origin: (x,y)' so you can compute screen position: screen_x = origin_x + image_x. When dotByDotMaxDimension is also set, scale factor is included: screen_x = origin_x + image_x / scale. | |
| displayId | No | Capture a specific monitor (0 = primary). Use desktop_state({includeScreen:true}) to list displays. | |
| grayscale | No | Convert to grayscale before encoding. Reduces file size ~50% for text-heavy content (e.g. AWS console, code editors). Avoid when color is meaningful (charts, status indicators). | |
| fullContent | No | When mode='background', use PW_RENDERFULLCONTENT to capture GPU-rendered windows (Chrome, Electron, WinUI3). Default true. Set false for legacy mode (faster but GPU windows may appear black). Ignored unless mode='background'. | |
| ocrFallback | No | OCR fallback behaviour when detail='text'. 'auto' (default): fire Windows OCR if UIA returns 0 actionable elements OR hints.uiaSparse=true (UIA returned <5 elements, typical for Chrome). 'always': always augment actionable[] with OCR words. 'never': disable OCR entirely. | auto |
| ocrLanguage | No | BCP-47 language tag for the OCR engine (e.g. 'ja', 'en-US'). Auto-detects from system locale when omitted. Used when detail='text' (OCR fallback) or detail='ocr' (direct OCR). | |
| webpQuality | No | WebP quality when dotByDot=true or diffMode=true. 40=layout only, 60=general (default), 80=fine text. | |
| windowTitle | No | Capture only the window whose title contains this string. Use '@active' for the current foreground window. Prefer over full-screen when target window is known. | |
| confirmImage | No | Embed inline image pixels in the response. detail='image' now returns a cheap by-ref resource_link WITHOUT this flag (it is no longer blocked); confirmImage=true ADDITIONALLY embeds the inline image for immediate vision. detail='som' likewise returns its elements[] + a by-ref resource_link by default; confirmImage=true ADDITIONALLY inlines the annotated SoM bitmap. Prefer detail='text' / diffMode=true / dotByDot=true first — set confirmImage=true only when inline visual inspection is genuinely required. | |
| maxDimension | No | Max width or height in pixels (default 768). Use 1280 to read small text, code, or fine UI details. Ignored when dotByDot=true. | |
| preprocessPolicy | No | OCR preprocessing scale policy for detail='som' and OCR fallback paths. 'auto' (default): clamp scale to 1 on OOM (>8MP) or high-DPI (≥150%). 'aggressive': relaxes DPI clamp to 175%, preserving upscale on 150%-DPI monitors (e.g. Outlook PWA). Also auto-enables adaptive binarization. 'minimal': always scale=1 regardless of DPI/resolution. | auto |
| preprocessAdaptive | No | When true, apply Sauvola adaptive binarization after contrast stretch. Improves recognition of thin text on low-contrast or gradient backgrounds. Automatically enabled when preprocessPolicy='aggressive'. Requires Rust native engine; silently skipped otherwise. | |
| dotByDotMaxDimension | No | Cap the longest edge (pixels) when dotByDot=true. Reduces payload while preserving coordinate math. Example: 1280 on a 1920×1080 screen → scale≈0.667. Response includes scale factor: screen_x = origin_x + image_x / scale. Recommended for Chrome: dotByDot=true, dotByDotMaxDimension=1280, grayscale=true. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Despite no annotations, the description thoroughly discloses behavioral traits: default scaling, diffMode cold-start behavior, background capture constraints, OCR fallback logic, and coordinate computation. No contradictions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Long but well-structured with sections for purpose, details, prefer, caveats, and examples. Front-loaded with key info. Could be slightly more concise, but justified by tool complexity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema, but description covers return types (resource_link, inline image, elements) and important notes like the need for a baseline for diffMode. Addresses all key aspects for a complex tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description adds value with examples, inter-parameter dependencies (e.g., region with windowTitle), and business logic (e.g., confirmImage behavior).
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description explicitly states the tool captures desktop, window, or region with multiple detail levels and capture modes. It distinguishes from sibling tools like browser_* and screenshot_gc, and provides clear usage boundaries.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Extensive guidance on when to use each detail level, mode, and combination. Includes a 'Prefer' section with explicit recommendations and a 'Caveats' section detailing restrictions and prerequisites.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
screenshot_gcA
Reclaim disk space from cached screenshots by retention policy. By DEFAULT this is a dry run: it returns the captures that WOULD be deleted (candidates) plus a count/size of leftover orphan files, and deletes nothing. To actually delete, pass BOTH dryRun:false AND confirm:true. Retention caps (all optional): maxCount (keep newest N), maxTotalBytes (keep newest under a byte budget), maxAgeMs (delete older than). When you pass none, the cache's env defaults apply (newest 200 / 256 MiB). Scope to a single tag with tag (other tags are never touched); includeOrphans (default true) also reclaims leftover on-disk files with no index entry. The newest capture is always kept by the count/byte caps. Only ever touches files inside the screenshot cache — never any other path.
| Name | Required | Description | Default |
|---|---|---|---|
| tag | No | Limit deletion to captures under this tag (case-insensitive). Other tags are never touched. | |
| dryRun | No | Default true: only LIST what would be deleted, delete nothing. Set false (with confirm:true) to actually delete. | |
| confirm | No | Safety gate: deletion happens ONLY when dryRun:false AND confirm:true. Otherwise the call is forced to a dry run. | |
| include | No | Optional response-shape opt-in. `['envelope']` returns the self-documenting envelope (`_version` / `data` / `as_of` / `confidence`). `['raw']` forces raw shape (overrides DESKTOP_TOUCH_ENVELOPE=1 server default). Default behaviour is raw shape (compat with existing clients). | |
| maxAgeMs | No | Delete captures older than this many milliseconds (opt-in; can clear even the newest). | |
| maxCount | No | Keep only the newest N captures; delete the rest. The single newest is always kept. | |
| maxTotalBytes | No | Keep the newest captures under this total byte budget; delete older ones beyond it. The newest is always kept. | |
| includeOrphans | No | Default true: also reclaim leftover on-disk image files that are not tracked in the cache index (e.g. files left behind by a crash). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description fully bears burden. Discloses dry-run default, two-flag safety gate, retention cap behaviors (always keeps newest), and scope limitation to screenshot cache only. Thoroughly covers behavioral traits.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Single dense paragraph, front-loaded with key behavior and safety. Every sentence provides value; however, could benefit from clearer structure (e.g., bullet points for retention caps). Still highly concise.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Covers all 8 parameters, defaults, safety, scope, and return value in dry-run (candidates + orphan stats). No output schema, but description adequately explains what the call returns. Complete for a cleanup tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, but description adds value: explains default retention values (newest 200 / 256 MiB), safety interplay of dryRun/confirm, purpose of include (response shape), and includeOrphans default. Goes beyond schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states 'Reclaim disk space from cached screenshots by retention policy', specifying the action (reclaim) and resource (cached screenshots). It distinguishes from sibling tools like screenshot (capture) and screenshot_query (query) by focusing on garbage collection and cache cleanup.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly explains dry-run default and safety condition (dryRun:false + confirm:true for actual deletion). Provides context on when to use (disk space reclamation) and scope options (tag, includeOrphans, retention caps). Implicitly distinguishes from capture/query tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
screenshot_queryA
List screenshots already saved in the disk-cache WITHOUT re-reading any pixels. The screenshot tools return each capture as a cheap by-ref link (screenshot://by-ref/{captureId}); this lists what is in the cache — captureId + by-ref uri, dimensions, size in bytes, timestamp, and tag/window — so you can find and re-open a specific earlier capture, or check how much the cache holds before reclaiming space with screenshot_gc. The response also carries whole-cache totals (totalCaptures / totalBytes). Reading a capture's bytes still costs tokens, so open a by-ref link only when you actually need to inspect the pixels. Filter by tag (case-insensitive) / windowUuid / since / until; page with limit (default 50) and offset. Results are newest-first and never include a filesystem path.
| Name | Required | Description | Default |
|---|---|---|---|
| tag | No | Filter to captures stored under this tag (case-insensitive). Omit to list all. | |
| limit | No | Maximum rows to return, newest first (default 50, max 500). | |
| since | No | Only captures taken at/after this time (epoch milliseconds, inclusive). | |
| until | No | Only captures taken at/before this time (epoch milliseconds, inclusive). | |
| offset | No | Rows to skip from the newest end, for paging (default 0). | |
| include | No | Optional response-shape opt-in. `['envelope']` returns the self-documenting envelope (`_version` / `data` / `as_of` / `confidence`). `['raw']` forces raw shape (overrides DESKTOP_TOUCH_ENVELOPE=1 server default). Default behaviour is raw shape (compat with existing clients). | |
| windowUuid | No | Filter to captures of a specific window (the window's stable id). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description fully discloses behavior: non-destructive, lists returned fields (captureId, by-ref uri, dimensions, size, timestamp, tag/window, totals), notes ordering (newest-first), and warns about token costs for reading pixels. Also explains response shape options via 'include' parameter.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Five well-structured sentences: first states core purpose, then elaborates on return content, cost warning, filter/paging options, and ordering. No fluff, front-loaded with the main action.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 7 parameters (0 required) and no output schema, the description covers the tool's purpose, return shape (fields and totals), filtering, pagination, ordering, and token-cost warning. It adequately equips an agent to use the tool correctly without needing additional context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. Description adds extra context: defaults for limit (50), offsets, case-insensitivity for tag, and ordering (newest-first). While some info repeats schema, the description organizes and clarifies usage for pagination and filtering, adding meaningful value.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states the verb 'List' and resource 'screenshots saved in the disk-cache', emphasizing it is non-destructive and fast. It distinguishes from sibling tools like 'screenshot' and 'screenshot_gc' by describing its specific function.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides explicit use cases: 'so you can find and re-open a specific earlier capture, or check how much the cache holds before reclaiming space with screenshot_gc.' Also warns against unnecessary token cost when opening by-ref links. Differentiates from siblings and advises on when to use.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scrollA
Purpose: Scroll a window or page. 5 strategies via action: 'raw' (wheel notches), 'to_element' (UIA name/automationId or CSS selector), 'smart' (auto-detect target with multi-strategy fallback), 'capture' (full-page stitched image), 'read' (scroll+OCR+dedupe → stitched text). Details: action='raw': send raw mouse-wheel notches at (x,y) or current cursor, optional window focus. Scroll scale — UIA Tier 1 (ScrollPattern apps): empirically ≈1 text line per notch; amount:3 (default) ≈ 3 lines (small nudge), amount:10 ≈ 10 lines (~½ visible area). Legacy SendInput: each amount unit = 3 wheel ticks; ≈9 text lines per unit at Windows default (app/OS-setting dependent). action='to_element': scroll a named element into viewport (UIA or CDP). action='smart': handles nested scroll layers, virtualised lists, sticky-header occlusion. action='capture': stitches full-page images (caps at ~700KB raw); sizeReduced=true means downscaled. action='read': scrolls page-by-page, OCRs each viewport, deduplicates overlapping lines, returns stitched text; language auto-detected from OS locale if omitted. Prefer: Use action='to_element' or action='smart' for click target out-of-viewport recovery (entity_outside_viewport) — scrolling only helps when the target scrolled out of its own window; if desktop_act reported origin_window_not_visible, restore the window with focus_window and re-run desktop_discover instead. Use action='capture' for reading long pages as images. Use action='read' for extracting text from long native-app documents (PDF readers, text editors, terminals) where copy-paste is unavailable. For simple scroll without target, use action='raw'. Caveats: action='capture' returns stitched image — pixels do NOT match screen coords when sizeReduced=true, use for reading only, not mouse_click. action='smart' CDP path requires browser_open. action='to_element' native path requires element to implement UIA ScrollItemPattern. action='read' uses OCR (imperfect accuracy) and requires the window to be visible; for browser pages prefer browser_eval or browser_overview for accurate DOM text. action='raw' typed errors: code:'ScrollNotDelivered' on silent drop (overlay / non-scrollable / UIPI low-IL); already-at-boundary is success via pre/post-percent disambiguation. hints.verifyDelivery.{channel,reason} per ADR-018 §2.6 (Phase 1b: Tier 1 UIA dispatch for HWNDs exposing ScrollPattern; other apps use legacy SendInput). action='smart' typed errors: code:'OverflowHiddenAncestor' (retry with expandHidden:true), code:'VirtualScrollExhausted' (provide virtualIndex). Examples: scroll({action:'raw', direction:'down', amount:5, windowTitle:'Chrome'}) scroll({action:'to_element', name:'OK', windowTitle:'Dialog'}) scroll({action:'smart', target:'#create-release-btn'}) scroll({action:'capture', windowTitle:'Chrome', maxScrolls:10}) scroll({action:'read', windowTitle:'Acrobat', maxPages:15}) // OCR + dedupe long PDF
| Name | Required | Description | Default |
|---|---|---|---|
| x | No | X coordinate to scroll at (moves cursor there first) | |
| y | No | Y coordinate to scroll at | |
| hint | No | Scroll direction hint for binary-search (image path). Seeds lo/hi bounds to reduce attempts. | |
| hwnd | No | Direct window handle ID (takes precedence over windowTitle). | |
| name | No | Partial name/label of the element (UIA name match). Use for native app elements. At least one of name or selector must be provided. | |
| port | No | CDP port for Chrome path (default 9222) | |
| block | No | Vertical alignment after scroll — start/center/end/nearest (Chrome path only, default: center) | center |
| speed | No | Cursor movement speed in px/sec (0=teleport, omit=default) | |
| tabId | No | Tab ID (Chrome path only). Omit for first page tab. | |
| action | Yes | Action selector — one of: raw, to_element, smart, capture, read. Per-action required fields are enforced at call time (see the tool description); this flat schema lists every action's fields as optional. | |
| amount | No | Number of scroll notches (default 3). UIA-capable apps (Notepad, Explorer, WPF — Tier 1): empirically ≈1 text line per notch; amount:3 (default) ≈ 3 lines (small nudge), amount:10 ≈ 10 lines (~½ visible area). Legacy apps (SendInput path): each amount unit sends 3 wheel ticks; at Windows default 3 lines/tick that is ≈9 text lines per unit — distance varies by app/OS wheel-speed settings. | |
| homing | No | Apply window-movement homing correction to (x,y) before scrolling. Default true. | |
| inline | No | Vertical alignment after scroll (CDP path). Default: center. | center |
| target | No | CSS selector (Chrome/Edge) or partial UIA name (native apps). For CDP path, must be a valid CSS selector (starts with #, ., tag, or [ ). For UIA path, a partial name match against element Name property. | |
| include | No | Optional response-shape opt-in. `['envelope']` returns the self-documenting envelope (`_version` / `data` / `as_of` / `confidence`). `['raw']` forces raw shape (overrides DESKTOP_TOUCH_ENVELOPE=1 server default). Default behaviour is raw shape (compat with existing clients). | |
| language | No | OCR language code (e.g. 'ja', 'en', 'zh'). Omit to auto-detect from Windows system locale via Intl.DateTimeFormat().resolvedOptions().locale. Default: auto. | |
| maxDepth | No | Max number of ancestor scroll containers to walk. Default 3. | |
| maxPages | No | Maximum number of scroll steps / OCR pages (default 20, max 50). | |
| maxWidth | No | Max size of the short edge of the final image (default 1280). For 'down': caps the image width; height is unconstrained. For 'right': caps the image height; width is unconstrained. | |
| selector | No | CSS selector for the element (Chrome/Edge only). At least one of name or selector must be provided. | |
| strategy | No | auto (default): try CDP → UIA → image in order. cdp: Chrome/Edge only. uia: native Windows UIA. image: image + Win32 binary-search. | auto |
| direction | No | Scroll direction | |
| scrollKey | No | Key sent to scroll one page. PageDown (default): full-page scroll for most apps. Space: web/PDF readers. ArrowDown: line-by-line slow scroll. | PageDown |
| maxScrolls | No | Maximum scroll iterations before stopping (default 10, max 30) | |
| retryCount | No | Max scroll attempts (image path binary-search). Default 3, cap 4. | |
| windowTitle | No | Partial window title. When provided, the server focuses this window first. | |
| expandHidden | No | Temporarily set overflow:hidden ancestors to overflow:auto to unlock scroll. Mutates live CSS. | |
| virtualIndex | No | Target row index in a virtualised list (0-based). Enables direct TanStack/data-index seeking. | |
| virtualTotal | No | Total row count in a virtualised list. Required when virtualIndex is set. | |
| scrollDelayMs | No | Milliseconds to wait after each scroll for rendering to settle (default 400). Increase for slow/animated pages. | |
| verifyWithHash | No | Verify scroll effectiveness via perceptual hash comparison. Automatically enabled for image path. | |
| stopWhenNoChange | No | Stop automatically when two consecutive pages yield no new lines after deduplication (page-end detection). Default true. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description fully discloses behavioral traits: scroll amount interpretation (UIA vs legacy), error types (ScrollNotDelivered, OverflowHiddenAncestor, VirtualScrollExhausted), OCR language auto-detection, sizeReduced implications for capture, and that action='read' uses OCR with imperfect accuracy. No contradictions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but well-structured with clear sections (Purpose, Details, Prefer, Caveats, Examples) and front-loaded with a summary. While every sentence adds value, the length may be slightly verbose for a concise reference. Still, it is appropriately sized given the tool's complexity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (32 parameters, 5 actions) and no output schema, the description covers all necessary aspects: detailed action explanations, usage guidelines, caveats, error handling, and concrete examples. It leaves no significant gaps in understanding for the agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% so baseline is 3. The description adds significant value beyond the schema, such as empirical scroll amounts for 'amount' parameter and clarifying per-action applicability of parameters. However, some schema descriptions are already detailed, so the added value is notable but not exceptional.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Purpose: Scroll a window or page' and enumerates 5 distinct actions (raw, to_element, smart, capture, read) with specific use cases. It distinguishes itself from sibling tools by focusing exclusively on scrolling, mentioning alternatives like browser_eval or browser_overview for browser text extraction.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description includes a 'Prefer' section that explicitly tells when to use each action, such as using action='to_element' or action='smart' for out-of-viewport recovery, and caveats like not using action='capture' for mouse_click due to coordinate mismatch. It also advises against scrolling when origin_window_not_visible, suggesting focus_window instead.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
server_statusA
Return MCP server status. engine: native engine availability — uia: 'native' = Rust UIA addon (~2 ms focus / ~100 ms tree); 'powershell' = PS fallback (~366 ms focus). imageDiff: 'native' = Rust SSE2 SIMD (0.26 ms @ 1080p); 'typescript' = TS fallback (~3.8 ms). health: process diagnostic snapshot (issue #365) — uptimeSec, memory.{rssBytes,heapUsedBytes,heapTotalBytes}, cpu.{userUs,systemUs} (cumulative since startup), shutdown.{pending,graceMs,inflightCount} (pending=true means stdin EOF received and grace timer is running), lastRpc.{receivedAt(ISO),method} (last JSON-RPC request observed on stdio transport; HTTP transport is not tracked). Diagnostic metadata — do not surface unless the user asks about performance/troubleshooting. engine values are stable for the process lifetime; health values change per call.
| Name | Required | Description | Default |
|---|---|---|---|
| include | No | Optional response-shape opt-in. `['envelope']` returns the self-documenting envelope (`_version` / `data` / `as_of` / `confidence`). `['raw']` forces raw shape (overrides DESKTOP_TOUCH_ENVELOPE=1 server default). Default behaviour is raw shape (compat with existing clients). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so the description carries the full burden. It discloses that engine values are stable for the process lifetime and health values change per call. It also notes that diagnostic metadata should not be surfaced unless asked, showing awareness of implications.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but front-loaded with the main purpose. The detailed breakdown of fields is necessary given no output schema, but it could be more concise by grouping related information. Every sentence provides value, but the length affects readability.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite no output schema, the description thoroughly explains the return value structure for engine, imageDiff, and health fields, including units and examples. It also covers the optional parameter and its effects, making the tool's behavior fully understandable.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% for the single 'include' parameter. The description adds meaning by explaining the allowable values ('envelope', 'raw') and their effects, as well as default behavior. This goes beyond the schema's minimal description.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description starts with 'Return MCP server status' which clearly states the verb and resource. It then lists specific fields like engine, imageDiff, and health, distinguishing it from sibling tools that deal with browser, desktop, and other operations.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly states 'do not surface unless the user asks about performance/troubleshooting', providing clear when-to-use guidance. It does not give explicit when-not-to-use alternatives, but the context of siblings suggests no similar tool exists.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
terminalA
Purpose: Interact with a terminal window: read output, send input, or run+wait+read in one call. action='read' / action='send' absorb the formerly-standalone read/send tools (Phase 4).
Details: action='run' is the recommended high-level workflow: send command → wait until quiet/pattern/timeout → read output. The command text is passed as input (the legacy parameter name command is also accepted as a deprecated alias — see issue #245). Returns completion={reason, elapsedMs} first-class plus outputIntegrity:'ok'|'baseline_lost' so callers can detect when scrollback could not be anchored to the pre-send buffer. action='read' reads current text via UIA TextPattern (falls back to OCR); use sinceMarker for incremental diff. action='send' sends a command with focus management.
Prefer: action='run' for command execution + result. For long-running commands (test runners, builds, deploys) use until:{mode:'pattern', pattern:''} — the default quiet mode is tuned for short interactive commands and may complete prematurely on multi-second silent gaps mid-run. Use action='read'/'send' for fine-grained control or when you need to interleave other actions. read/send/run also accept a launch_console paneId (pass the paneId field, NOT windowTitle) to keep targeting a pane after its title changes.
Caveats: Do not screenshot the terminal — action='read' is cheaper and structured. action='run' supports completion reasons: quiet | pattern_matched | exited | timeout | window_closed | window_not_found | send_failed (send rejected on a live window — see warnings for the underlying error code). until:{mode:'exit', shell:'bash'|'powershell'} (issue #386) returns completion.exitCode + reason:'exited' via an echo-immune sentinel that works for multiline input that pattern mode cannot anchor; pass shell explicitly (auto fails as ExitModeShellAmbiguous on WT/conhost/SSH), cmd is unsupported (ExitModeShellUnsupported), open-construct input is rejected (ExitModeUnsafeInput). When outputIntegrity:'baseline_lost' is returned, output is forced to '' and readError.code='BaselineMarkerLost' is set: rerun with until:{mode:'pattern',...} or longer timeoutMs. action='run' may also emit warnings prefixed FileLockCollision: when output reveals an EBUSY/Windows-lock/EAGAIN-EDEADLK file collision (e.g. shell '>' redirect colliding with the script's own writer — issue #236). Default quietMs=1500 (issue #196); long silences require pattern mode. preferClipboard=true (send default) replaces the clipboard. Hidden-input prompts emit verifyDelivery.unverifiable (reason:'hidden_input_prompt') — use method:'foreground'. action='read' typed errors: TerminalWindowNotFound, TerminalTextPatternUnavailable (force source:'ocr'); stale sinceMarker → hints.terminalMarker.previousMatched:false on ok:true (omit sinceMarker). FG-path Win11 foreground refusal returns code:'ForegroundRestricted' — switch to method:'background' or DTM_BG_AUTO=1. BG path auto-engages only when the target class is ConsoleWindowClass (conhost: cmd / PowerShell / pwsh) OR env DTM_BG_AUTO=1. Windows Terminal (CASCADIA_HOSTING_WINDOW_CLASS) is EXCLUDED (issue #173): WT runs on WinUI/XAML and silently drops WM_CHAR, so the FG path is default — pass sendOptions:{method:'background'} only if your WT build accepts BG input.
Examples:
terminal({action:'run', windowTitle:'PowerShell', input:'npm test', until:{mode:'pattern', pattern:'Test Files'}}) → recommended for test runners; matches when vitest summary appears
terminal({action:'run', windowTitle:'pwsh', input:'ls'}) → quiet 1500ms wait, returns output (short interactive)
terminal({action:'run', windowTitle:'pwsh', command:'ls'}) → identical to the above; command is a deprecated alias of input (issue #245)
terminal({action:'read', windowTitle:'PowerShell', sinceMarker:'...'}) → incremental diff using the read action
terminal({action:'send', windowTitle:'PowerShell', input:'echo hello'}) → sends text + Enter using the send action
| Name | Required | Description | Default |
|---|---|---|---|
| input | No | Command to send (Enter is appended automatically). Either `input` or its deprecated alias `command` is required. | |
| until | No | ||
| action | Yes | Action selector — one of: read, send, run. Per-action required fields are enforced at call time (see the tool description); this flat schema lists every action's fields as optional. | |
| paneId | No | ||
| command | No | [Deprecated alias of `input`] Accepted for callers that mis-remember the parameter name; new code should use `input`. If both are set, `input` wins. | |
| include | No | Optional response-shape opt-in. `['envelope']` returns the self-documenting envelope (`_version` / `data` / `as_of` / `confidence`). `['raw']` forces raw shape (overrides DESKTOP_TOUCH_ENVELOPE=1 server default). Default behaviour is raw shape (compat with existing clients). | |
| timeoutMs | No | Hard timeout in ms (default 30s) | |
| readOptions | No | Extra options forwarded to terminal read (lines, source, ocrLanguage, etc.) | |
| sendOptions | No | Extra options forwarded to terminal send (method, chunkSize, etc.) | |
| windowTitle | No | Partial title of the terminal window (e.g. 'PowerShell', 'pwsh', 'WindowsTerminal'). Provide windowTitle OR paneId (paneId takes precedence). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are present, so the description carries full burden. It richly discloses behavioral traits: completion reasons (quiet, pattern_matched, exited, timeout, etc.), outputIntegrity values, baseline_lost behavior, file-lock warnings, default quietMs, platform-specific nuances (Windows Terminal vs conhost, FG/BG paths), and hidden-input prompt handling. This transparency far exceeds typical tool descriptions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely long but well-structured with clear sections (Purpose, Details, Prefer, Caveats, Examples) and front-loaded purpose. While every sentence is information-dense, some redundancy exists (e.g., command alias mentioned multiple times, multiple issue references). Still, for a tool with this complexity, the length is largely justified, though a trim would improve conciseness.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, the description thoroughly explains return values (completion={reason, elapsedMs}, outputIntegrity, error codes), behavioral details, and edge cases. Examples cover run/read/send with various options. The description is complete enough for an agent to select and invoke the tool correctly across diverse scenarios, including Windows Terminal limitations and error handling.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 80%, but the description adds substantial meaning: it explains the deprecated `command` alias, clarifies paneId vs windowTitle precedence, details until modes (pattern/exit/quiet) with examples, and describes readOptions/sendOptions forwarding. It also gives concrete input examples that map parameters to usage scenarios, going well beyond the schema's field-level descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb+resource: "Interact with a terminal window: read output, send input, or run+wait+read in one call." It clearly distinguishes three actions (read/send/run) and explicitly notes that this tool absorbs the formerly-standalone read/send tools, making its scope unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit guidance: "Prefer: action='run' for command execution + result" and "Use action='read'/'send' for fine-grained control or when you need to interleave other actions." It also instructs users to avoid screenshots of the terminal, directing them to read instead, and explains when to use pattern mode for long-running commands.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
wait_untilA
Purpose: Server-side poll for an observable condition — eliminates screenshot-polling loops when waiting for state changes. Details: condition selects what to watch: window_appears/window_disappears (target.windowTitle required), focus_changes (optional target.fromHwnd), element_appears/value_changes (target.windowTitle + target.elementName required, UIA; min 500ms interval), ready_state (target.windowTitle; visible + not minimized), terminal_output_contains (target.windowTitle + target.pattern required [+target.regex:true], needs terminal tools loaded), element_matches (target.by + target.pattern required, needs browser tools loaded), url_matches (target.pattern required [+target.regex:true]; matches the active tab's location.href via CDP — use for SPA route changes, redirects, OAuth flows). Returns {ok:true, elapsedMs, observed} on success, or WaitTimeout error with suggest hints. timeoutMs default 5000 (max 60000). Prefer: Use instead of run_macro({sleep:N}) + screenshot loops. Use terminal_output_contains to detect CLI command completion. Use element_matches for browser DOM readiness after navigation. Use url_matches when the URL is the most reliable signal (SPA routing / redirect cascades). Caveats: terminal_output_contains, element_matches, and url_matches require a browser CDP connection (open --remote-debugging-port=9222 first). element_appears/value_changes spawn a UIA process per poll — interval clamped to 500ms minimum. On elapsed-timeout the response is {ok:false, code:'WaitTimeout', error, suggest:[...]}; the suggest[] array lists three fixed actions: 'Increase timeoutMs', 'Verify the target is correct', 'Inspect intermediate state with screenshot(detail='meta')'. Non-timeout failures also occur — pre-poll validation and missing-hook errors classify as code:'ToolError' (read the descriptive error message), and CDP probe errors (url_matches / element_matches conditions) surface as code:'BrowserNotConnected' (re-attach via browser_open). Branch on code rather than assume WaitTimeout. Examples: wait_until({condition:'window_appears', target:{windowTitle:'Save As'}, timeoutMs:10000}) wait_until({condition:'terminal_output_contains', target:{windowTitle:'Terminal', pattern:'$ '}, timeoutMs:30000}) wait_until({condition:'element_matches', target:{by:'text', pattern:'Submit', scope:'#checkout-form'}}) wait_until({condition:'url_matches', target:{pattern:'/dashboard'}, timeoutMs:15000}) wait_until({condition:'url_matches', target:{pattern:'^https://app\\.example\\.com/orders/[0-9]+$', regex:true}})
| Name | Required | Description | Default |
|---|---|---|---|
| target | No | Target descriptor — fields used depend on condition. Accepts an object literal or a JSON-stringified object. | |
| include | No | Optional response-shape opt-in. `['envelope']` returns the self-documenting envelope (`_version` / `data` / `as_of` / `confidence`). `['raw']` forces raw shape (overrides DESKTOP_TOUCH_ENVELOPE=1 server default). Default behaviour is raw shape (compat with existing clients). | |
| condition | Yes | Condition to wait for. See per-condition target requirements. | |
| timeoutMs | No | Maximum time to wait (default 5000ms) | |
| intervalMs | No | Poll interval (default 200ms — terminal_output_contains uses 500 internally) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Without annotations, the description fully discloses behavior: server-side polling, condition-specific target requirements, return values (success and error shapes), timeout handling (WaitTimeout with suggestions), and non-timeout errors (ToolError, BrowserNotConnected). It also notes internal details like UIA process spawning and interval clamping.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but well-organized with clear sections (Purpose, Details, Prefer, Caveats, Examples). Each sentence adds necessary information. Slightly verbose due to error code details, but overall efficient and front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (9 conditions, multiple error modes, integration with other tools), the description is remarkably complete. It covers all essential aspects: required parameters, return shapes, error handling, and example invocations, leaving no major gaps for an AI agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, but the description adds substantial value beyond the schema. For each condition, it details required target fields and constraints (e.g., 'element_appears/value_changes require UIA; min 500ms interval'). It also explains the include parameter's envelope option and default behavior.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the purpose: 'Server-side poll for an observable condition — eliminates screenshot-polling loops.' It lists specific conditions like window_appears, terminal_output_contains, etc., and distinguishes this tool from alternatives like screenshot loops.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The 'Prefer:' section explicitly advises using this tool instead of run_macro with sleep and screenshot loops. It provides specific use cases for each condition (e.g., 'Use terminal_output_contains to detect CLI command completion') and mentions caveats like CDP requirements and error handling.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
window_dockA
Purpose: Decorate a window: pin (always-on-top), unpin, or dock (move + resize + optional pin). Details: action='pin' makes window always-on-top until unpin/duration_ms. action='unpin' removes always-on-top. action='dock' positions to corner with width/height (default 480×360 bottom-right) and optionally pins. Minimized windows are automatically restored before docking. Prefer: Use action='dock' for terminal/CLI window auto-positioning at session start. Use action='pin' alone when you only need always-on-top without moving or resizing. Caveats: Pin survives minimize/restore; explicit action='unpin' needed to release. Dock fails on elevated processes. Dock overrides any existing Win+Arrow snap arrangement. Examples: window_dock({action:'dock', title:'PowerShell', corner:'bottom-right', width:480, height:360}) window_dock({action:'pin', title:'Settings', duration_ms:5000}) window_dock({action:'unpin', title:'Settings'})
| Name | Required | Description | Default |
|---|---|---|---|
| pin | No | If true, set always-on-top so the docked window stays visible on top of other windows. Use window_dock(action='unpin') to remove the topmost flag later. Default true. | |
| title | No | Partial window title (case-insensitive) | |
| width | No | Window width in pixels after docking. Default 480. | |
| action | Yes | Action selector — one of: pin, unpin, dock. Per-action required fields are enforced at call time (see the tool description); this flat schema lists every action's fields as optional. | |
| corner | No | Screen corner to snap the window to. Default 'bottom-right'. | bottom-right |
| height | No | Window height in pixels after docking. Default 360. | |
| margin | No | Pixel padding between the window and the screen edge. Default 8. | |
| include | No | Optional response-shape opt-in. `['envelope']` returns the self-documenting envelope (`_version` / `data` / `as_of` / `confidence`). `['raw']` forces raw shape (overrides DESKTOP_TOUCH_ENVELOPE=1 server default). Default behaviour is raw shape (compat with existing clients). | |
| monitorId | No | Monitor to dock on (from desktop_state({includeScreen:true})). Omit for primary monitor. | |
| duration_ms | No | Auto-unpin after this many ms (0–60000). Omit to pin indefinitely. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Discloses important behaviors: pin survives minimize/restore, dock fails on elevated processes, dock overrides Win+Arrow snap, minimized windows restored before docking. Since no annotations are provided, the description carries the full burden, and it does a good job covering key behavioral traits. Lacks mention of return values, but no output schema exists.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Well-structured with clear sections (Purpose, Details, Prefer, Caveats, Examples). Every sentence adds value, and examples cover all three actions. No wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 10 parameters and no output schema, the description effectively explains the core functionality, defaults, and caveats. It covers the main use cases and limitations. The include parameter is not explained in the description, but its schema description is sufficient. Overall, it's quite complete for a complex tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description adds value beyond the schema by explaining inter-parameter dependencies (e.g., dock action's width/height defaults, pin and duration_ms relation) and provides examples that show typical parameter combinations. This enriches understanding.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: decorate a window by pinning, unpinning, or docking. It distinguishes between the three actions (pin, unpin, dock) and explains what each does, making it easy to select the right action.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides explicit guidance: 'Prefer: Use action='dock' for terminal/CLI window auto-positioning at session start. Use action='pin' alone when you only need always-on-top without moving or resizing.' This helps the agent choose the correct usage context. Also includes caveats like dock failing on elevated processes.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
workspace_launchA
Purpose: Launch an application and wait for its new window to appear, returning title, HWND, and PID. Details: Runs the command via ShellExecute, snapshots the window list before launch, then polls until a new HWND appears (compared by HWND, not title). Returns {windowTitle, hwnd, pid, elapsedMs}. Works for localized window titles (e.g. '電卓' for calc.exe) because detection is HWND-based, not title-based. timeoutMs default 10000. detach=true fires without waiting and returns no window info. Prefer: Use instead of run_macro({exec, sleep, desktop_discover}) combos. Follow with focus_window(windowTitle) to interact with the launched app. Caveats: Single-instance apps that reuse an existing window will not register as a new HWND — call desktop_discover first to check if the window is already open. detach=true returns immediately with no window title or hwnd. Examples: workspace_launch({command:'notepad.exe'}) → {windowTitle:'', hwnd:'...', pid:...} workspace_launch({command:'calc.exe', timeoutMs:15000})
| Name | Required | Description | Default |
|---|---|---|---|
| args | No | Command-line arguments (max 20). Shell metacharacters (; & | ` $() ${}) are not allowed. | |
| waitMs | No | Milliseconds to wait for the window to appear (default 2000) | |
| command | Yes | Executable name or full path (e.g. 'notepad.exe', 'calc.exe'). Shell interpreters (cmd.exe, powershell.exe, etc.) are blocked. | |
| include | No | Optional response-shape opt-in. `['envelope']` returns the self-documenting envelope (`_version` / `data` / `as_of` / `confidence`). `['raw']` forces raw shape (overrides DESKTOP_TOUCH_ENVELOPE=1 server default). Default behaviour is raw shape (compat with existing clients). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden and richly discloses behavior: HWND-based detection, polling mechanism, localized title handling, and detach effects. It explains the snapshot-then-poll logic and default timeout, leaving no ambiguity.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with clear sections (Purpose, Details, Prefer, Caveats, Examples). It is front-loaded with the main purpose and every sentence adds value without redundancy. Efficient use of space.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, the description fully explains the return shape and edge cases. It covers localization, single-instance apps, and detach behavior, making it complete for an AI agent to understand invocation and outcomes.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema has 100% coverage, but the description adds valuable context: explains command and args restrictions, default timeout (though slightly inconsistent with schema's waitMs), and includes examples. However, the minor discrepancy in default value (timeoutMs vs waitMs) prevents a perfect score.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Launch an application and wait for its new window to appear, returning title, HWND, and PID.' It specifies the verb (launch), resource (application), and return values, distinguishing from sibling tools like run_macro.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly recommends using this tool instead of run_macro combos and provides follow-up actions like focus_window. It also includes caveats for single-instance apps and detach behavior, offering clear when-to-use and when-not-to-use guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
workspace_snapshotA
Purpose: Orient fully in one call — returns display layouts, all window thumbnails (WebP), and per-window actionable element lists with clickAt coords. Details: uiSummary.actionable[] per window includes: action ('click'|'type'|'expand'|'select'), clickAt {x,y} (pass directly to mouse_click), value (current text for editable fields). Runs parallel internally; latency ≈ max(single screenshot), not N×screenshots. Also resets the diffMode buffer so subsequent screenshot(diffMode=true) returns only changes (P-frame). Prefer: Use at session start or after major workspace changes. Use screenshot(detail='meta') for cheap re-orientation within a session. Use screenshot(detail='text', windowTitle=X) for a single-window update. Caveats: Thumbnails are scaled, not 1:1 — use screenshot(dotByDot=true, windowTitle=X) for pixel-accurate coords on a specific window after snapshot. Also: this call resets the screenshot diff baseline (I-frame) and identity tracker as a side effect, so subsequent screenshot(diffMode=true) starts fresh from this snapshot. The reset is not currently exposed in causal/working memory — record an explicit 'workspace_snapshot' step if you need to track the reset point in your causal trail (ADR-010 §11 OQ carry-over for full visibility).
| Name | Required | Description | Default |
|---|---|---|---|
| include | No | Optional response-shape opt-in. `['envelope']` returns the self-documenting envelope (`_version` / `data` / `as_of` / `confidence`). `['raw']` forces raw shape (overrides DESKTOP_TOUCH_ENVELOPE=1 server default). Default behaviour is raw shape (compat with existing clients). | |
| includeUiSummary | No | Whether to include UI element summaries for each window | |
| thumbnailMaxDimension | No | Max size of per-window thumbnail images (default 400px) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden. It discloses internal parallelism ('Runs parallel internally; latency ≈ max(single screenshot)'), side effects ('resets the diffMode buffer', 'resets the identity tracker'), and caveats ('Thumbnails are scaled, not 1:1').
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with sections (Purpose, Details, Prefer, Caveats) and is front-loaded with the key purpose. It is slightly verbose but every sentence adds value; no waste.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, the description explains the return structure (uiSummary.actionable[] with clickAt, value) and behavior (thumbnails as WebP, scaled). It covers side effects and usage context. Minor gap: lacks explicit structure for display layouts.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so baseline is 3. The description does not add significant meaning beyond the schema; it mentions thumbnailMaxDimension as 'max size' which is already in the schema. The include parameter details are adequately covered in the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Orient fully in one call — returns display layouts, all window thumbnails (WebP), and per-window actionable element lists with clickAt coords.' It distinguishes from siblings by recommending alternatives like screenshot(detail='meta') for cheap re-orientation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The 'Prefer' section explicitly states when to use: 'Use at session start or after major workspace changes.' It also provides alternatives: 'Use screenshot(detail='meta') for cheap re-orientation within a session. Use screenshot(detail='text', windowTitle=X) for a single-window update.'
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
1 tool update
v1.14.3- Changed
keyboard1 field changed- changed
Input schema / properties / use_clipboard / descriptionPrevious value: -"If true, copy text to clipboard and paste with Ctrl+V instead of simulating keystrokes. Use this when typing URLs, paths, or ASCII text into apps with Japanese IME active — prevents IME from converting characters. Default false."New value: +"If true, copy text to clipboard and paste with Ctrl+V instead of simulating keystrokes. Use this when typing URLs, paths, or ASCII text into apps with Japanese IME active — pasted text is not run through IME conversion. Note this does not help while an IME composition is already in progress: the paste keystroke is consumed by the IME and nothing is inserted, so commit or cancel the composition first. Your clipboard is replaced for the duration of the call and put back afterwards; hints.clipboard reports which backend served the paste and whether the restore ran. On builds without the native addon this path is capped at about 12000 characters and fails with code:'ClipboardWriteTooLargeForFallback' above it. Default false."
1 tool update
v1.13.0- Changed
terminal2 fields changed- added
Input schema / properties / paneIdAdded value: +{ + "type": "string" +} - changed
Input schema / properties / windowTitle / descriptionPrevious value: -"Partial title of the terminal window (e.g. 'PowerShell', 'pwsh', 'WindowsTerminal')."New value: +"Partial title of the terminal window (e.g. 'PowerShell', 'pwsh', 'WindowsTerminal'). Provide windowTitle OR paneId (paneId takes precedence)."
20 tool updates
v1.12.0- Added
browser_click - Added
browser_eval - Added
browser_fill - Added
browser_form - Added
browser_locate - Added
browser_navigate - Added
browser_open - Added
browser_overview - Added
browser_search - Added
click_element - Added
clipboard - Added
desktop_state - Added
excel - Added
focus_window - Added
key_locker - Added
keyboard - Added
mouse_click - Changed
screenshot2 fields changed- changed
Input schema / properties / confirmImage / descriptionPrevious value: -"Must be true to receive image pixels when detail='image'. Without this flag, detail='image' is blocked and a guidance message is returned instead. Prefer detail='text' / diffMode=true / dotByDot=true first — only set confirmImage=true when visual inspection is genuinely required."New value: +"Embed inline image pixels in the response. detail='image' now returns a cheap by-ref resource_link WITHOUT this flag (it is no longer blocked); confirmImage=true ADDITIONALLY embeds the inline image for immediate vision. detail='som' likewise returns its elements[] + a by-ref resource_link by default; confirmImage=true ADDITIONALLY inlines the annotated SoM bitmap. Prefer detail='text' / diffMode=true / dotByDot=true first — set confirmImage=true only when inline visual inspection is genuinely required." - changed
Input schema / properties / detail / descriptionPrevious value: -"Response detail level (omit to let the server pick a smart default):\n omitted — auto: 'image' when dotByDot/region/displayId is specified, else 'meta'\n 'meta' — window title + screen region only (~20 tok/window, cheapest)\n 'text' — UIA element tree as JSON with text values (~100-300 tok/window, no image)\n 'image' — actual screenshot pixels. BLOCKED unless confirmImage=true is also passed.\n 'som' — Set-of-Marks image + OCR elements (bypasses UIA entirely). BLOCKED unless confirmImage=true is also passed.\n 'ocr' — Windows OCR words with screen-pixel clickAt coords (Phase 4: absorbs former screenshot_ocr). Use when UIA returns no actionable elements (WinUI3 custom-drawn UIs, game overlays, PDF viewers). Note: detail='text' auto-falls back to OCR via ocrFallback='auto'; choose detail='ocr' only when forcing OCR unconditionally."New value: +"Response detail level (omit to let the server pick a smart default):\n omitted — auto: 'image' when dotByDot/region/displayId is specified, else 'meta'\n 'meta' — window title + screen region only (~20 tok/window, cheapest)\n 'text' — UIA element tree as JSON with text values (~100-300 tok/window, no image)\n 'image' — actual screenshot pixels. Returns a cheap by-ref resource_link by default (no inline base64); pass confirmImage=true to ALSO embed the inline image.\n 'som' — Set-of-Marks elements + annotated image (bypasses UIA entirely). Returns the OCR elements[] plus a cheap by-ref resource_link by default (no inline base64); pass confirmImage=true to ALSO embed the annotated bitmap.\n 'ocr' — Windows OCR words with screen-pixel clickAt coords (Phase 4: absorbs former screenshot_ocr). Use when UIA returns no actionable elements (WinUI3 custom-drawn UIs, game overlays, PDF viewers). Note: detail='text' auto-falls back to OCR via ocrFallback='auto'; choose detail='ocr' only when forcing OCR unconditionally."
- Added
screenshot_gc - Added
screenshot_query
16 tool updates
v1.10.4- Removed
browser_click - Removed
browser_eval - Removed
browser_fill - Removed
browser_form - Removed
browser_locate - Removed
browser_navigate - Removed
browser_open - Removed
browser_overview - Removed
browser_search - Removed
click_element - Removed
clipboard - Removed
desktop_state - Removed
excel - Removed
focus_window - Removed
keyboard - Removed
mouse_click
1 tool update
v1.10.3- Changed
screenshot2 fields changed- removed
Input schema / properties / ocrLanguage / defaultRemoved value: -"ja" - changed
Input schema / properties / ocrLanguage / descriptionPrevious value: -"BCP-47 language tag for the OCR engine (e.g. 'ja', 'en-US'). Used when detail='text' (OCR fallback) or detail='ocr' (direct OCR)."New value: +"BCP-47 language tag for the OCR engine (e.g. 'ja', 'en-US'). Auto-detects from system locale when omitted. Used when detail='text' (OCR fallback) or detail='ocr' (direct OCR)."
27 tool updates
v1.9.2- Added
browser_click - Added
browser_eval - Added
browser_fill - Added
browser_form - Added
browser_locate - Added
browser_navigate - Added
browser_open - Added
browser_overview - Added
browser_search - Added
click_element - Added
clipboard - Added
desktop_state - Added
excel - Added
focus_window - Added
keyboard - Added
mouse_click - Added
mouse_drag - Added
notification_show - Added
run_macro - Added
screenshot - Added
scroll - Added
server_status - Added
terminal - Added
wait_until - Added
window_dock - Added
workspace_launch - Added
workspace_snapshot
27 tool updates
v1.8.0- Removed
browser_click - Removed
browser_eval - Removed
browser_fill - Removed
browser_form - Removed
browser_locate - Removed
browser_navigate - Removed
browser_open - Removed
browser_overview - Removed
browser_search - Removed
click_element - Removed
clipboard - Removed
desktop_state - Removed
excel - Removed
focus_window - Removed
keyboard - Removed
mouse_click - Removed
mouse_drag - Removed
notification_show - Removed
run_macro - Removed
screenshot - Removed
scroll - Removed
server_status - Removed
terminal - Removed
wait_until - Removed
window_dock - Removed
workspace_launch - Removed
workspace_snapshot
27 tool updates
v1.6.0- Added
browser_click - Added
browser_eval - Added
browser_fill - Added
browser_form - Added
browser_locate - Added
browser_navigate - Added
browser_open - Added
browser_overview - Added
browser_search - Added
click_element - Added
clipboard - Added
desktop_state - Added
excel - Added
focus_window - Added
keyboard - Added
mouse_click - Added
mouse_drag - Added
notification_show - Added
run_macro - Added
screenshot - Added
scroll - Added
server_status - Added
terminal - Added
wait_until - Added
window_dock - Added
workspace_launch - Added
workspace_snapshot
27 tool updates
v1.5.1- Removed
browser_click - Removed
browser_eval - Removed
browser_fill - Removed
browser_form - Removed
browser_locate - Removed
browser_navigate - Removed
browser_open - Removed
browser_overview - Removed
browser_search - Removed
click_element - Removed
clipboard - Removed
desktop_state - Removed
excel - Removed
focus_window - Removed
keyboard - Removed
mouse_click - Removed
mouse_drag - Removed
notification_show - Removed
run_macro - Removed
screenshot - Removed
scroll - Removed
server_status - Removed
terminal - Removed
wait_until - Removed
window_dock - Removed
workspace_launch - Removed
workspace_snapshot
27 tool updates
v1.5.0- Added
browser_click - Added
browser_eval - Added
browser_fill - Added
browser_form - Added
browser_locate - Added
browser_navigate - Added
browser_open - Added
browser_overview - Added
browser_search - Added
click_element - Added
clipboard - Added
desktop_state - Added
excel - Added
focus_window - Added
keyboard - Added
mouse_click - Added
mouse_drag - Added
notification_show - Added
run_macro - Added
screenshot - Added
scroll - Added
server_status - Added
terminal - Added
wait_until - Added
window_dock - Added
workspace_launch - Added
workspace_snapshot
TDQS
Tools are mostly distinct with detailed descriptions that clarify when to use each. Overlaps like browser_click, click_element, and mouse_click are disambiguated by target type (DOM, UIA, screen coords). However, the sheer number of similar actions could still confuse an agent, especially with multiple ways to achieve the same goal.
Tool names follow a clear verb_noun pattern with underscores (e.g., browser_click, mouse_drag, wait_until). Browser tools are consistently prefixed with 'browser_', but there is slight inconsistency with standalone names like 'clipboard' and 'desktop_state' lacking a verb prefix. Overall, naming is predictable and readable.
30 tools is on the higher end but reasonable given the broad scope (browser, desktop, terminal, window management, etc.). Some niche tools like screenshot_gc and key_locker could be merged, but each serves a specific need. The count feels slightly bloated but not excessive.
The toolset covers a wide range of actions for browser and native app automation, including click, type, scroll, screenshot, and window management. Minor gaps exist (e.g., no file operations, no direct text-to-speech), but for the intended domain it is sufficiently complete. The inclusion of terminal and Excel VBA adds depth.
Maintenance
Related MCP Connectors
A comprehensive Model Context Protocol (MCP) server that enables AI assistants to interact with yo…
MCP server for visual regression testing: triage a PR's UI diffs from your coding agent.
9118Personal assistant MCP server with search, execute, packages, jobs, secrets, and integrations.
MCP server for progressive tool usage at any scale (see https://klavis.ai)
Related MCP Servers
- AlicenseNot gradedqualityCmaintenanceGUI automation MCP server that enables AI agents to see and control the Windows desktop using a local Vision LLM (Ollama), supporting screenshot analysis, mouse/keyboard actions, and autonomous task execution.4MIT
- AlicenseNot gradedqualityAmaintenanceA Windows computer use agent — FastMCP server that gives AI assistants hands on the real desktop: windows, UI elements, mouse, keyboard, screenshots, OCR, shortcuts, dialogs, and outcome verification.35MIT
- AlicenseAqualityBmaintenanceAllow AI agents to see and control a real Windows PC you own: observe (UIA + screenshots), click/type/drag/scroll, launch apps, owner Live View. BYOH — your machine, your key.216Apache 2.0
- FlicenseAqualityAmaintenanceCross-platform desktop automation MCP server that lets AI agents capture screenshots, run OCR with UI-element classification, control mouse/keyboard, and launch programs on Linux, macOS, and Windows.20-
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/Harusame64/desktop-touch-mcp'
If you have feedback or need assistance with the MCP directory API, please join our Discord server