Skip to main content
Glama

Nova

A Computer Use implementation in Rust. Nova is a Model Context Protocol server that gives an LLM agent AX-first control of macOS and Windows: semantic UI reads/actions, screenshots, mouse, keyboard, scrolling, window/app introspection, OCR, and the clipboard — the "computer use" capability, built natively in Rust rather than wrapping a Python/JS automation stack.

Built directly on native platform APIs (macOS Accessibility/ScreenCaptureKit/ CoreGraphics and Windows UI Automation/Win32) — a single self-contained binary, no runtime to install. Connect it to any MCP client (Claude Desktop, an agent runtime, your own) over stdio or Streamable HTTP.

Tools

Tool

What it does

ax_read

Canonical ax:read: read semantic labels, text, values, roles, actions, state, and optional bounds through macOS Accessibility or Windows UIA, without a screenshot. Returns an ephemeral snapshot/node protocol and explicit coverage/status.

read_ui

Compatibility alias backed by the same ax_read traversal and cache generation.

ax_activate

Activate an exact actionable node from a fresh ax_read; rejects stale snapshot IDs and reports route=ax|uia|web_dom|element_center. Every attempt consumes its generation before provider dispatch.

screenshot

Capture the whole display or a single window= — use for layout, icons, colors, images, canvas, and visual verification after semantic/OCR paths.

zoom_region

Magnify a rectangle of the last screenshot at native resolution — reads small targets on surfaces with no Accessibility tree.

ocr

Recognize on-screen text with Apple Vision on macOS or Windows Media OCR on Windows. mode=auto uses Fast first with confidence-based Accurate fallback; mode=fast|accurate forces either policy. An optional strict roi={x,y,width,height} re-captures a rectangle from the current image through the native region path. Returns each line with a clickable center.

click_mark

Compatibility action for the latest numbered mark; prefer generation-safe ax_activate.

left_click / right_click / double_click / mouse_move / scroll

Pointer input in the pixel space of the last screenshot.

cursor_position

Read the cursor in OS-global logical coordinates; it is not converted into the last screenshot's pixel space.

type_text / key_combo

Keyboard input (full Unicode, incl. CJK + emoji).

list_windows / list_applications / open_application

Window & app introspection.

inspect_app

Optional macOS app capability discovery. Accepts an app name/bundle ID, or discovers running Chromium candidates when omitted; no caller-supplied port or permission prompt.

read_clipboard / write_clipboard

Clipboard access.

ax_click / ax_set_value / ax_focus

Drive controls by Accessibility role/label.

dump_ax

Read the raw AX/UIA tree for diagnostics and coverage debugging.

batch_actions

Run a sequence of input actions in one call.

wait

Pause for a specified number of seconds.

Related MCP server: computer-use

Requirements

  • macOS 14+ for the macOS desktop backend. The release archive is universal and runs on Apple Silicon and Intel Macs.

  • Windows x86_64 or ARM64 for the Windows desktop backend. GitHub Releases provide a native archive for each architecture.

  • On Windows, ocr uses installed Windows OCR language packs. Use nova --ocr-langs to inspect available languages; install the needed pack if recognition reports that it is unavailable.

  • Building on macOS requires the macOS 15 SDK / Xcode 16+ because of a transitive apple-metal build dependency; that is a build-time requirement, not Nova's minimum macOS runtime version.

  • On macOS, Screen Recording permission is required for screenshot, ocr, and list_windows; Accessibility is required for ax_read, semantic activation, and input.

macOS grants these permissions to the process it identifies as responsible for Nova. The managed nova mcp entrypoint and Bamboo plugin use the independent Nova.app on macOS. Legacy direct stdio/HTTP can use the host app, terminal, or directly launched binary as the permission subject. See Permissions & code signing.

Run

cargo run                      # stdio transport (default)
cargo run -- mcp                # managed MCP: Nova.app on macOS, stdio elsewhere
cargo run -- --http            # Streamable HTTP on 127.0.0.1:3100
cargo run -- --http --addr 127.0.0.1:8080

The Swift runtime that ScreenCaptureKit links is located via an LC_RPATH baked in by build.rs, so no DYLD_* environment variable is needed for cargo run/cargo test or the standalone binary.

Install

The supported public installation paths are a source build and the verified prebuilt archives on the v0.2.1 GitHub Release. That release provides:

Platform

Archive

macOS, Apple Silicon + Intel

nova-v<version>-universal-apple-darwin.tar.gz

Windows x86_64

nova-v<version>-x86_64-pc-windows-msvc.zip

Windows ARM64

nova-v<version>-aarch64-pc-windows-msvc.zip

Download the matching .sha256 file from the same release and verify the archive before extracting it. On macOS:

tar -xzf nova-v*-universal-apple-darwin.tar.gz
xattr -dr com.apple.quarantine ./nova  # only if Gatekeeper blocks the download
sudo install -m 0755 nova /usr/local/bin/nova

On Windows, extract the archive for the machine's architecture and invoke nova.exe directly or place its directory on PATH. The Windows binaries are not Authenticode-signed, so SmartScreen may warn on first run.

v0.2.1 predates the current AX-first tools (ax_read, read_ui, and ax_activate), the managed mcp command, and Nova.app. Its archives provide the earlier screenshot/mark/input tool set. Build the current source below when using this workflow; those features are not in the v0.2.1 binaries.

To build from source:

git clone https://github.com/bigduu/Nova.git
cd Nova
cargo build --release --locked

The result is target/release/nova on macOS or target/release/nova.exe on Windows. The macOS release binary is ad-hoc signed, not notarized.

Nova.app development preview

Releases cut from a revision containing the app packaging workflow also attach:

nova-v<version>-universal-apple-darwin-development-app.zip

This archive contains a universal Nova.app that runs Nova's per-user app service without a Dock icon. It gives Screen Recording and Accessibility a Nova application identity instead of making the MCP host (for example, Bodhi) the permission subject. Install and start it with:

shasum -a 256 -c nova-v*-universal-apple-darwin-development-app.zip.sha256
unzip nova-v*-universal-apple-darwin-development-app.zip
ditto Nova.app /Applications/Nova.app
open -gj -b com.zenith.nova

Install the app independently of Bodhi and the plugin's downloaded CLI. Keep it at /Applications/Nova.app (or ~/Applications/Nova.app), outside Bodhi.app and the plugin directory. The plugin still downloads the CLI archive and uses it only as the connector on macOS; installing/updating the plugin does not install or update Nova.app. Use a CLI and app built from the same current version.

Configure a stdio MCP client with nova mcp, as shown in Use it from an MCP client below. If no app archive has been published for the current code, build both macOS architectures, combine them into a universal binary, then assemble the app with package-development-app.sh, which requires a universal binary and the matching Cargo version as arguments.

WARNING

The app archive isDEVELOPMENT ONLY. It is ad-hoc signed, not Developer ID signed, not notarized, and not stapled. Gatekeeper can block it, and replacing it with a differently signed build can require granting TCC permissions again. The existing universal CLI .tar.gz remains the supported artifact consumed by Homebrew, npm, and Bamboo; the app .zip does not replace it.

Use it from an MCP client

Claude Desktop (or any stdio MCP client) — add Nova to the client's MCP config. Claude Desktop uses ~/Library/Application Support/Claude/claude_desktop_config.json on macOS and %APPDATA%\Claude\claude_desktop_config.json on Windows:

{
  "mcpServers": {
    "nova": { "command": "/absolute/path/to/nova", "args": ["mcp"] }
  }
}

mcp is the cross-platform managed entrypoint used by the Bamboo plugin. Windows and Linux headless builds serve ordinary stdio MCP. On macOS it only connects to the independent Nova.app, launching it through LaunchServices when needed. Install the app separately, open it once, and grant Accessibility to Nova; Screen Recording is needed for capture, OCR, and list_windows (the current preview may request it on app startup). The bundled executable can also be used as the connector:

{
  "mcpServers": {
    "nova": {
      "command": "/Applications/Nova.app/Contents/MacOS/nova",
      "args": ["mcp"]
    }
  }
}

The explicit --connect command remains supported and uses the same transport as macOS mcp. It carries MCP bytes over a private per-user Unix socket. The connector does not call desktop APIs or request macOS permissions; the app process owns the MCP handlers and TCC responsibility. The socket lives under /tmp/nova-app-<uid>/ with a mode-0700 directory, mode-0600 socket, and a same-UID peer check.

If Nova.app is unavailable, the managed command exits with installation and reconnection guidance. It never falls back to desktop operations inside the MCP host. NOVA_APP_SOCKET is for isolated development/tests; when set it disables automatic app launch. Unset it for the normal installed-app setup. Unbundled nova with no arguments still offers the legacy direct stdio mode.

When the app service closes its connection, the CLI connector finishes forwarding responses and exits even if the host keeps its stdin pipe open. Closing host stdin normally still half-closes the request stream and drains the service's final response, including its last buffered bytes. Standard stdout backpressure still applies: the host must keep reading responses. Reconnect only the Nova MCP server after replacing/restarting the service; Bodhi can stay open. No interrupted request is replayed and no replacement MCP session is created automatically.

This exit behavior is specific to the terminating CLI connector process. Its dedicated runtime is released after forwarding completes, and process exit reclaims an outstanding blocking stdin read. It does not make the connect_stdio library function's stdin cancellable inside a resident or embedded runtime.

Chrome DevTools MCP sidecar

For routine Chrome page automation and debugging, Nova can launch the official Chrome DevTools MCP next to the desktop server. This is a transparent stdio sidecar, not a second browser implementation inside Nova. It requires npm/npx, Node.js ^20.19.0, ^22.12.0, or >=23, and current stable Chrome (or newer). Nova pins the reviewed upstream package to chrome-devtools-mcp@1.8.0.

On macOS, a recommended two-server configuration is:

{
  "mcpServers": {
    "nova": {
      "command": "/Applications/Nova.app/Contents/MacOS/nova",
      "args": ["--connect"]
    },
    "nova-chrome-devtools": {
      "command": "/Applications/Nova.app/Contents/MacOS/nova",
      "args": ["chrome-devtools"]
    }
  }
}

For a standalone source/release binary, use the same binary path and ["chrome-devtools"]. If a GUI client cannot find npx, add "--npx", "/absolute/path/to/npx" after the subcommand.

The default launches a new temporary, isolated Chrome profile. Usage statistics, package update checks, CrUX URL lookups, and sensitive network headers are disabled/redacted by default. Requests made by attached DevTools targets can be guarded by repeating --allowed-url-pattern, for example:

"args": [
  "chrome-devtools",
  "--allowed-url-pattern", "https://example.com/*",
  "--allowed-url-pattern", "https://*.example.net/*"
]

URL allow patterns require Chrome 149+. They apply only to DevTools targets while the MCP server is attached and are not a complete network sandbox; use an OS/VM sandbox when full network isolation is required, as described by the upstream security policy.

To work with an already running signed-in Chrome profile instead, first open chrome://inspect/#remote-debugging in Chrome and enable remote debugging, then configure:

"args": ["chrome-devtools", "--profile", "existing"]

Automatic connection requires Chrome 144+. If several Chrome profiles are active, Chrome chooses the profile it considers the default; select and verify the connected pages before acting.

WARNING

Existing-profile mode can inspect and control every open window in the selected Chrome profile, including authenticated pages. Enable it only for a trusted local MCP client, and disable remote debugging when finished.

Use --enable-webmcp to expose upstream's experimental WebMCP tools. Nova adds Chrome's required --enable-features=WebMCP launch argument in isolated mode; for an existing profile, Chrome itself must already have been started with that feature enabled. WebMCP requires Chrome 150+. --expose-network-headers and --enable-performance-crux are explicit privacy opt-ins. The pinned 1.8.0 package does not support a --disable-javascript-evaluation option, so Nova does not advertise or pass it.

The sidecar and Nova's optional Secure Chrome Bridge serve different trust models: DevTools MCP is the broad, full-featured choice for normal browser automation, DOM/network inspection, and performance debugging; the Secure Chrome Bridge requires explicit per-page pairing and is preferable when least-privilege page scoping matters. Nova's desktop tools remain the path for browser chrome, native dialogs, and non-web UI.

Use the absolute path to the extracted release binary or the source-build output. On Windows, use an escaped executable path such as "C:\\absolute\\path\\nova.exe". Use the source-build output for the AX-first workflow below. If its directory is already on PATH, the command can be "nova".

Reconnect/reload the Nova MCP server in the client; Bodhi's main window can stay open. See Permissions & code signing for legacy direct-stdio and development-binary cases.

HTTP clients — run Nova as a server and connect over Streamable HTTP:

nova --http                       # 127.0.0.1:3100/mcp
nova --http --addr 127.0.0.1:8080 # custom loopback port

HTTP mode is currently a local transport: it keeps rmcp's default loopback Host allowlist and does not configure remote-access authentication. Binding all interfaces is not a supported LAN setup.

First calls (current source build). Call ax_read (optionally ax_read(window="<name>", mode="all")) for semantic content and controls, then ax_activate(snapshot_id, node_id) on an exact actionable node. Re-run ax_read after the action to verify semantic state. If AX/UIA coverage is absent or partial, use focused-window ocr for rendered text; use screenshot(window=...) / zoom_region only when pixels are necessary (layout, icon, color, image, canvas, or visual verification). All pointer tools use the pixel space of the most recent screenshot; cursor_position instead reports OS-global logical coordinates.

Permissions & code signing (macOS)

Inspect an application's interaction options

Use inspect_app when setting up an application or checking which interaction route is available. It is optional; ordinary native interaction still starts with ax_read.

{"app": "Slack"}

The selector accepts a running application's name or bundle identifier. Exact matches take priority over partial matches. Omit app to discover running Electron, Chromium, and CEF candidates, including applications with no discovered debugging connection. Names alone do not confirm a runtime: Nova checks known framework containers and their executable evidence. Unknown or unreadable bundles remain unknown.

The default result contains application identity, runtime, inspection status, the currently available native route, and a next step. Nova finds process-owned local connection candidates internally; callers do not need to find or supply ports. For diagnostics only, use:

{"app": "com.example.application", "details": true}

Detailed output includes bundle/runtime evidence, process start identities, endpoint provenance, and metadata verification. Nova checks the selected app's owned listeners, recognized debugging flags, and the exact DevToolsActivePort file only when a --user-data-dir flag evidences the profile location. It does not scan profile contents or return full arguments/environment. Programmatically enabled ports can be discovered through listener ownership even when a flag is absent from the OS argument list.

browser_endpoint_available means a metadata-only browser handshake succeeded; it does not attach browser tools, grant authorization, or verify the full Chrome DevTools MCP toolset. Native ax_read still uses Accessibility, and the result reports when that permission is needed. Node inspector endpoints, incompatible endpoints, stale evidence, and incomplete inspection remain distinct. No discovered port is not proof that debugging is disabled. Enablement and whether a particular application can support a restart-based change remain unknown until verified for that application.

Discovery does not launch, focus, quit, or restart applications, request permissions, modify bundles/arguments, or open a debugging service. Network requests stay on verified process-owned loopback sockets: /json/version, Browser.getVersion, and Target.getBrowserContexts only. There is no page enumeration, script evaluation, input, or Browser.close. HTTP proxies and redirects are disabled; advertised WebSockets must keep the same owned address and port. Ownership/start identity is checked before and after probing.

An investigation allows 8 seconds overall, 16 result apps, 32 processes per app, 4 helper generations, 8 endpoint probes per app, and 2 evidenced profiles. Each metadata probe has a 900 ms deadline; HTTP bodies and WebSocket messages are limited to 32 KiB, the WebSocket exchange to 128 KiB and 16 frames per reply. Framework lookup is limited to 64 entries in an app's Contents/Frameworks, plus at most four version directories in each recognized framework. Limits or unavailable evidence are reported as incomplete, rather than silently claiming that an application has no debugging support. Concurrent calls receive a busy result. Windows/Linux return an explicit unsupported result; their existing native tools are unchanged.

On macOS, resident desktop transports keep the process main run loop active, so applications launched or quit after the first inspection appear or disappear without restarting Nova or its MCP host. Relaunching an application triggers fresh process/start-time and endpoint ownership checks. This inventory refresh does not require Screen Recording or Accessibility permission. The mcp and --connect byte proxies return before this desktop event loop and bootstrap.

The automated tests use fake bundles, process records, and loopback services. The ignored own_listener_and_process_start_identity_match test inspects only its own process/listener. The ignored e2e_app_inspection acceptance test requires an explicitly prepared app with a dev.nova.acceptance.* bundle identifier and NOVA_TEST_APP_BUNDLE_ID; it never defaults to inspecting the user's running applications.

cargo test --test e2e_resident_app_inspection runs a separate macOS regression whose test binary owns the real process main thread. In one resident process it seeds discovery, launches a unique temporary AppKit app, checks appearance, quits it, checks disappearance, and checks a new process identity on relaunch. It repeats this with an internally allocated random listener that simulates the narrow CDP handshake; the fixture is not Chromium and is reported as an unknown runtime. It creates no windows and requests no permissions. Real Electron/Chromium lifecycle acceptance remains a separate check. The test-only --without-main-loop argument is a negative control that reproduces the old stale-inventory failure; it is expected to fail.

Permission ownership

The independent app transport is the preferred permission model: grant Screen Recording and Accessibility to Nova.app, then use nova mcp (or explicit nova --connect). The connector never initializes CoreGraphics or Accessibility, so Bamboo, Claude Desktop, and terminals no longer need Nova's desktop permissions.

Keep Nova.app installed independently and unchanged when upgrading Bodhi. The new Bodhi/plugin connector connects to the same app-owned service, so its own build/signing identity does not become Nova's permission subject. This is an architectural guarantee about where desktop calls execute; signed installation and real TCC upgrade acceptance remain separate release gates. Replacing Nova.app itself, changing its signature, or an OS permission decision can still require granting permissions again. The development preview is ad-hoc signed.

After granting Nova permissions in System Settings, retry the tool. If macOS requires a restart for the change, quit/reopen Nova.app, then reconnect only the Nova MCP server in the client. Keep Bodhi's main window open. The connector does not replay interrupted requests or automatically restore an MCP session after Nova exits. Do not remove/re-add Bodhi's grants to repair this managed Nova path.

Two details still matter for direct stdio/HTTP and source-development modes:

Grant the responsible process for the way Nova is launched. macOS TCC may attribute a child process to its responsible parent app. For legacy direct stdio MCP (an empty argument list), grant Claude Desktop, Bamboo, or the terminal/IDE that launches Nova. For a directly launched CLI/HTTP process, macOS may instead use the Nova binary. If granting the expected host does not work, add the installed nova binary (or target/release/nova) as a fallback under System Settings → Privacy & Security → Screen Recording and Accessibility.

Keep the identity of whichever process receives the grant stable. If Nova itself is the permission subject, cargo build produces an ad-hoc, linker-signed binary whose code-signing identity is a content hash (nova-<hash>). It changes every build, so a direct binary grant stops applying. Sign Nova with a stable self-signed identity when developing in that mode:

cargo build --release
./scripts/dev-codesign.sh --release   # re-sign after EVERY build

The first run creates a Zenith Nova Code Signing identity in your login keychain (click Always Allow once if codesign prompts) and signs the binary with a fixed identifier (com.zenith.nova). A direct Nova grant then survives rebuilds that are re-signed with the same certificate. Host-app grants likewise depend on the host keeping a stable signing identity.

Troubleshooting — screenshot fails with a "wedged" / "busy" capture error. All captures (and window enumeration) run in ONE shared per-user daemon (nova --capture-daemon, flock-elected, socket /tmp/nova-capture-<uid>-<hash>.sock), because replayd keys clients by executable path — two same-binary ScreenCaptureKit clients evict each other's XPC identity and wedge every new stream start. The daemon kills itself if a capture exceeds its 8s watchdog, and the client auto-recovers: kill+respawn the daemon, then (second failure) SIGKILL all nova capture processes and killall -9 replayd — wedges self-heal without manual action. If they don't: nova --selftest (probes ScreenCaptureKit in a sacrificial subprocess, then the daemon path) and read /tmp/nova-capture-worker.log (step trace) + /tmp/nova-capture-daemon.log (daemon stderr). Manual remedy = kill the processes holding streams (pkill -f -- --capture-daemon), NOT replayd: plain killall replayd is a no-op (replayd ignores SIGTERM), and even killall -9 replayd doesn't cure a wedge while a stream-holding client survives — it just reconnects and re-wedges the fresh replayd.

Coordinate grounding

A general LLM judging pixel coordinates off a downscaled screenshot is the main source of mis-clicks — so the primary path avoids pixels entirely.

  • ax_read first (no image) — returns actionable controls and non-actionable readable content in deterministic tree order. A successful macOS read requires Accessibility but does not contact ScreenCaptureKit. permission_denied means fix that grant; it is not an instruction to take a screenshot.

  • Fresh semantic action — call ax_activate with the returned snapshot and node IDs. Native AX/UIA and the browser DOM bridge are tried before a freshly revalidated element-center click. Stale generations fail closed; every activation attempt consumes its generation before provider dispatch, so read again after any result.

  • OCR second — when coverage is absent/partial and the missing information is rendered text, use focused-window OCR and its returned text center with left_click(..., source="ocr_center").

  • Screenshot/zoom last — use pixels for visual-only state or a surface with no semantic/text representation; coordinate clicks report route=visual_coordinate. Screenshot marks and click_mark remain available for compatibility.

When a screenshot is needed, all click/move/scroll tools work in the pixel space of the last screenshot — the server remembers that frame and maps clicks back to the real screen, so the model just "clicks what it sees":

  • screenshot(window: "<name>") — capture a single window (substring of its title or app name) instead of the whole display. Smaller, sharper image → less context and far less downscaling → better precision. Later clicks map into that window.

  • zoom_region(x, y, w, h) — magnify a rectangle of the last screenshot at native resolution (capturing only that rectangle). For reading small targets on surfaces that expose no Accessibility tree (canvas, games, custom views), where coordinates are the only option. A labeled coordinate grid is overlaid so the model reads positions straight off the axes.

Testing

The suite is split into fast, hermetic tests (run by default) and side-effecting end-to-end tests (opt-in, #[ignore]d).

Default — unit + hermetic integration tests

cargo test

Runs everything that has no side effects and needs no special permission:

  • unit tests for coordinate scaling, the key/char keystroke maps, combo parsing, batch (de)serialization, and MCP tool registration;

  • tests/e2e_interaction.rs — screenshot→logical coordinate mapping (via CGDisplay, no permission needed) and a non-destructive clipboard round-trip (snapshots and restores the clipboard).

This is what the macOS test job runs in CI (see .github/workflows/ci.yml; the workflow also has Windows cross-check and Linux headless jobs).

End-to-end tests (#[ignore]d)

These either post real input events (they move the cursor, click, scroll, or type into the focused window) or require Screen Recording permission, so they are excluded from cargo test and must be opted into. Run them on a desktop session where that's acceptable:

# all of them
cargo test -- --include-ignored

# or a single one
cargo test --test e2e_input mouse_move_roundtrips_through_cursor_position -- --ignored

Test (file)

What it does

Needs

semantic_snapshot_reads… (e2e_ax_read)

Resolves and reads a focused or NOVA_AX_WINDOW app through AX/UIA without pixel capture

Accessibility / logged-in UIA desktop

mouse_move_roundtrips… (e2e_input)

Moves the cursor, reads it back via cursor_position, asserts the position — restores the cursor

Accessibility

click_events_post… (e2e_input)

Left/right/double click on the empty desktop corner (Esc dismisses the menu)

Accessibility

scroll_events_post… (e2e_input)

Posts vertical scroll events

Accessibility

type_text_posts… (e2e_input)

Types into the focused window

Accessibility

open_application_launches… (e2e_input)

Launches/focuses System Settings

list_windows_returns… (e2e_input)

Enumerates on-screen windows

Screen Recording

e2e_capture_display_returns_valid_jpeg (e2e_screenshot)

Captures the display, checks the JPEG

Screen Recording

e2e_capture_dims_match_target_dims_contract (e2e_screenshot)

Asserts capture dims match the click-coordinate mapping

Screen Recording

e2e_window_screenshot_produces_view_frame (e2e_screenshot)

Captures a window and validates its view-frame metadata

Screen Recording

ocr_recognizes_text_on_the_display (e2e_ocr)

Runs Apple Vision OCR on a live capture; asserts text + in-bounds line centers

Screen Recording

daemon_* / client_* / concurrent_* (e2e_capture_worker)

Shared capture daemon: capture, kill→respawn recovery, concurrent clients, clean-error survival

Screen Recording

legacy_pipe_protocol_still_served (e2e_worker)

Old --capture-worker pipe protocol, proxied into the daemon

Screen Recording

stdio_server_completes_handshake_and_lists_tools (e2e_stdio)

Exercises the stdio (JSON-RPC) transport end-to-end

safari_opens_google_and_nova_reads_the_homepage (e2e_safari_google)

Launches Safari, opens Google, and reads the page through Nova

Network + Screen Recording + Accessibility

mouse_move_roundtrips… proves the macOS pointer post and cursor read-back round-trip using logical coordinates. The non-ignored interaction test covers screenshot→logical coordinate arithmetic, while the live screenshot tests cover captured-dimension contracts.

Run e2e_capture_worker single-threaded (-- --ignored --test-threads=1): the tests share one daemon/socket.

list_applications_returns_app_bundles (in e2e_input) is not ignored — it only reads Spotlight and is tolerant of a Spotlight-less CI host.

Lint & format

cargo fmt --all -- --check
cargo clippy --all-targets

Releasing (maintainers)

A version tag drives everything via .github/workflows/release.yml. The workflow resolves the tag once, verifies it against Cargo.toml and the event commit, and makes every source-building job check out that immutable commit. It builds and smoke-tests the universal macOS CLI and development-only Nova.app, creates the Release with those assets, then sequenced jobs attach native Windows x86_64/ARM64 archives and the Bamboo plugin bundle. The CLI .tar.gz name and checksum outputs stay unchanged for Homebrew, npm, and the Bamboo plugin manifest.

Run the hermetic release checks before tagging:

scripts/test-release-workflow.sh

The current crate version is already published as v0.2.1; bump it before creating the next release tag. Release tags must be protected from force updates; the workflow also serializes runs by tag and re-verifies the tag before its first upload. The Nova.app asset must remain labeled DEVELOPMENT ONLY until all production distribution gates are complete:

  • sign nested code and the outer app, in that order, with a Developer ID Application identity and the hardened runtime;

  • submit the distribution artifact to Apple's notary service and verify the accepted ticket;

  • staple the ticket to the app and validate it with codesign and spctl;

  • authenticate local MCP and Chrome bridge peers with macOS audit tokens and designated code requirements, rather than relying on same-UID sockets alone;

  • run the packaged native host and extension against a real Chrome install, including pairing, navigation revocation, stale snapshots, and disconnects;

  • pin third-party GitHub Actions by full commit SHA before treating the release workflow as a production supply-chain boundary;

  • smoke-test launch, upgrade, nova --connect, Screen Recording, and Accessibility grants on clean Apple Silicon and Intel macOS 14+ machines.

Do not describe the ad-hoc-signed app preview as a production-ready macOS app.

License

MIT © bigduu

Available Tools

23 tools
ax_clickA

Press a UI control directly through the OS accessibility tree (macOS Accessibility; Windows UI Automation is not yet implemented and returns a clear error) — no coordinates, no cursor movement, works in the background (the app need not be frontmost). query is a case-insensitive substring of the element's accessibility role or label/title (e.g. "Send", "Search"). Targets the last window-captured app (or the frontmost app). Only works for apps that expose an accessibility tree; if it returns "no element matching" (or the not-implemented error), fall back to screenshot + left_click.

ParametersJSON Schema
NameRequiredDescriptionDefault
queryYesCase-insensitive substring matching the target element's accessibility role or label/title (e.g. "Send", "Search", "AXButton").

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Describes behavior in detail: no cursor movement, background operation, case-insensitive substring matching, targets last-window-captured app, Windows UIA not implemented. No annotations provided, so description fully covers transparency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Efficient paragraph: first sentence core action, second explains query, third gives fallback. No redundant sentences.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Covers all needed info for invocation: purpose, query format, target application, limitations, and fallback. Adequate for an action tool with one parameter and no output schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Single parameter 'query' is described in schema; description adds examples and clarifies substring matching and case-insensitivity. With 100% schema coverage, description adds value via examples.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description states it presses a UI control via accessibility tree, without coordinates or cursor movement. Distinguishes from sibling tools like left_click by clarifying background operation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly says when to use (apps with accessibility tree) and gives fallback (screenshot + left_click on error). Provides clear context for selection.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

ax_focusA

Move keyboard focus to a control through the OS accessibility tree (macOS Accessibility; not yet implemented on Windows, where it returns a clear error). Background, no cursor. query matches the element's role/label. Targets the last window-captured app (or frontmost). Native-app accessibility only.

ParametersJSON Schema
NameRequiredDescriptionDefault
queryYesCase-insensitive substring matching the target element's accessibility role or label/title (e.g. "Send", "Search", "AXButton").

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Without annotations, describes key behaviors: background operation, no cursor movement, query matching, target selection (last window-captured app), and error handling on Windows. Missing success/failure details for macOS.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three concise sentences, each adding distinct value: action, platform constraints, query behavior. No redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no output schema, annotations, and one parameter, the description covers purpose, platform, target selection, and query matching. Could mention behavior on multiple matches, but adequate for a simple tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with clear description for 'query'. The tool description adds no new information about the parameter beyond what the schema already provides.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool moves keyboard focus via the OS accessibility tree, with specific verb 'move' and resource 'control'. It distinguishes from siblings like ax_click (click) and dump_ax (dump tree) by focusing solely on focus manipulation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides context on when to use: background, no cursor, native-app accessibility only, and platform limitations (macOS vs Windows). Does not explicitly state when not to use but implies constraints.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

ax_set_valueA

Set a control's value directly through the OS accessibility tree (macOS Accessibility; not yet implemented on Windows, where it returns a clear error) — e.g. fill a text field without focusing or typing. Background, no cursor. query matches the element's role/label; value is the text to set. Targets the last window-captured app (or frontmost). Native-app accessibility only.

ParametersJSON Schema
NameRequiredDescriptionDefault
queryYesCase-insensitive substring matching the target element's role or label.
valueYesThe value to set (e.g. the text to place into a field).

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Discloses key behavioral traits: background operation, no cursor, platform-specific behavior (macOS works, Windows returns error), and targeting the last window-captured app. No annotations are provided, so the description carries the full burden.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Four sentences, each providing essential information: purpose, usage context, parameter explanation, and target scope. No redundancy or extraneous detail.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Covers platform, behavior, and parameter details. Lacks explicit description of the return value, but for a mutation tool, this is generally acceptable given the simplicity of the tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema covers both parameters with descriptions. The description adds an example of use (filling a text field without focusing) and clarifies the matching logic (case-insensitive substring). This adds minor value beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states the tool sets a control's value via the OS accessibility tree, with a concrete example (filling a text field without focusing or typing). It distinguishes itself from siblings like type_text by emphasizing background operation and no cursor.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides context on when to use (background value setting without focusing) and platform limitations (macOS only, Windows returns error). However, it does not explicitly mention alternatives or when not to use.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

batch_actionsA

Execute a sequence of input actions (mouse_move, left_click, right_click, double_click, scroll, key_combo, type_text, wait) in one call to reduce round-trips. Coordinates are in screenshot space. Take a screenshot separately afterwards to observe the result.

ParametersJSON Schema
NameRequiredDescriptionDefault
actionsYesOrdered list of input actions to execute in a single call.

TDQS

A4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so description carries full burden. It notes coordinate space and need for separate screenshot, but does not mention execution order (sequential vs parallel), error handling, or return value. Adequate but not detailed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, no wasted words. Front-loaded with core purpose and key usage instructions.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema, but description adequately explains how to observe results (separate screenshot). Covers actions, coordinate space, and purpose. Slightly lacks details on limits or failure behavior, but sufficient for a batching tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with descriptions for each action type. Description adds minimal extra value beyond reaffirming coordinate space and ordering. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly states it executes a sequence of input actions to reduce round-trips, lists all supported actions, and distinguishes from sibling individual action tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Description implies use when multiple actions are needed without intermediate round-trips, and advises taking a screenshot afterward. Lacks explicit when-not-to-use guidance but context of siblings helps.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

click_markA

Activate an actionable element by the mark NUMBER shown in the most recent screenshot(marks=true) — the reliable way to click without guessing coordinates. Always background, no cursor movement: web-page content in a scriptable browser (Safari, Chrome, Arc, Edge, Brave, …) is clicked through the page's OWN JavaScript engine (an Accessibility press is a silent no-op on web content), native controls through the Accessibility tree; if neither applies it falls back to a click at the element's center. Numbers go stale when the UI changes, so take a fresh screenshot(marks=true) right before calling this. If the number is unknown, re-shoot with marks=true.

ParametersJSON Schema
NameRequiredDescriptionDefault
numberYesThe mark number [N] of the element to activate, as listed by the most recent `screenshot(marks=true)`.
backgroundNoDeliver the coordinate-click fallback in the background to the captured window's process (native apps only). The AX action is always background; this only affects the fallback. Default false = foreground.

TDQS

A4.6/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so description fully carries burden. Explains execution details: uses JS engine for web, AX tree for native, fallback to center click. Clarifies background behavior and staleness. No contradiction with parameters.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Well-structured with key info front-loaded. Slightly wordy but every sentence adds value. Could be trimmed slightly but still effective.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Covers all behavioral aspects: how click is performed, background option, staleness handling, fallback. No output schema needed. Agent has sufficient info to use correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, but description adds significant context: marks come from screenshot(marks=true), staleness warning, and explanation of background parameter's effect on fallback.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states it activates an element by mark number from a screenshot, contrasting with coordinate guessing. Differentiates from sibling click tools by using marks.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Tells when to use (with valid mark), when mark goes stale (take fresh screenshot), and what to do if number unknown (re-shoot). Does not explicitly name alternative tools but context implies superiority over coordinate clicks.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

cursor_positionA

Get the current mouse cursor position. Returns (x, y) in logical coordinates.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

A3.5/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, and the description does not disclose behavioral traits such as whether the operation is read-only, has side effects, or requires permissions. It only states it returns coordinates.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Very concise single sentence. Could be improved by elaborating on 'logical coordinates' or units, but currently efficient.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple, zero-parameter tool, the description adequately states purpose and return value. However, it does not explain 'logical coordinates' or any limitations, leaving minor gaps.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

No parameters exist, so schema coverage is 100%. The description adds return format information ('Returns (x, y) in logical coordinates'), which is appropriate for a zero-parameter tool. Baseline of 4 is justified.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool gets the current mouse cursor position and returns coordinates. The verb 'Get' and resource 'mouse cursor position' are specific, and it distinguishes from siblings like mouse_move (which sets position).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool versus alternatives (e.g., before clicking, after moving). No exclusions or context provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

double_clickB

Double-click at the given (x, y) coordinates (in screenshot space).

ParametersJSON Schema
NameRequiredDescriptionDefault
xYes
yYes
backgroundNoDeliver in the background to the captured window's process (native apps only; browsers/Electron ignore it). Default false = foreground.

TDQS

B3.1/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so description alone must convey behavioral traits. It only states the action without mentioning side effects, permissions, or whether it's safe/destructive. No indication of what happens after the double-click.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Single sentence, no filler. However, it is overly terse, lacking any structure or additional details that could fit without bloat.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no output schema and 3 parameters, the description is too sparse. It omits essential context like what triggers a double-click effect, whether a target application must be captured, or any return value.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Adds context that coordinates are in 'screenshot space', which is useful beyond the schema. However, with only 33% schema coverage, it does not fully compensate; the background parameter remains undescribed in the main description.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states the action (double-click) and the target (x,y coordinates in screenshot space). Distinguishes it from sibling tools like left_click and right_click by specifying 'double-click'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool vs alternatives like click_mark (which clicks on UI elements) or left_click. Lacks context about prerequisites or scenarios.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

dump_axA

DEBUG: dump the target app's Accessibility tree (roles, subroles, labels, actions, frames) as indented text — to diagnose why some elements are not marked. Targets the last window-captured app (or frontmost).

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

A4.2/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries full burden. It clarifies it is a DEBUG tool (suggesting read-only, non-destructive behavior) and explains the content of the dump (roles, subroles, etc.) and targeting behavior. However, it does not describe the output format (e.g., printed to console or returned as string) or any potential side effects, leaving some behavioral ambiguity.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence with a dash for additional context. It is front-loaded with 'DEBUG:' to signal intent, then states the action and purpose concisely. Every word is informative, and there is no redundant information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity (zero parameters, no output schema), the description is largely complete. It explains what, why, and targeting. However, it lacks explicit mention of whether the output is printed or returned, and the 'last window-captured app' prerequisite could be clearer. Still, it covers most necessary context.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has zero parameters, and the description does not need to explain any. It adds value by describing the tool's function and output, which is more than the bare schema provides. For a no-parameter tool, this is adequate and exceeds the baseline.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: dumping the accessibility tree of a target app as indented text, including specific components (roles, subroles, labels, actions, frames). It also provides the diagnostic intent (to find why elements are not marked). This distinguishes it from sibling tools that perform actions like clicking or focusing, as it is purely a debug/inspection tool.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description indicates when to use the tool: for debugging accessibility issues, specifically to diagnose why elements are not marked. It also specifies the target (last window-captured app or frontmost). However, it does not explicitly state when not to use it or mention alternative tools, making it slightly less than a perfect score.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

key_comboB

Simulate a key combination (e.g., "cmd+c", "shift+tab").

ParametersJSON Schema
NameRequiredDescriptionDefault
keyYes
backgroundNoDeliver in the background to the captured window's process (native apps only). Default false = foreground.

TDQS

B3.3/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, and the description does not disclose behavioral traits such as how modifiers are handled, whether it works on all applications, or any side effects. The short description leaves many behavioral details unspecified.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is very concise (one sentence) but sacrifices completeness. It could be improved by adding a brief usage hint without becoming excessively long.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 2 parameters and no output schema, the description provides the core purpose but lacks context on behavior, prerequisites, or when to prefer it over sibling tools. It is minimally adequate.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The description adds meaning for the 'key' parameter with examples, which is otherwise undocumented in the schema. The 'background' parameter is well-described in the schema, so the description covers gaps for half the parameters.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('simulate a key combination') and provides concrete examples ('cmd+c', 'shift+tab'). It distinguishes from sibling tools like 'type_text' which types character strings.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool over alternatives such as 'type_text' or other click tools. No prerequisites, when-not, or exclusion criteria are mentioned.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

left_clickC

Left-click at the given (x, y) coordinates (in screenshot space).

ParametersJSON Schema
NameRequiredDescriptionDefault
xYes
yYes
backgroundNoDeliver in the background to the captured window's process (native apps only; browsers/Electron ignore it). Default false = foreground.

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden. It only states 'left-click' and 'screenshot space', omitting details like whether the mouse moves, behavior for out-of-bounds coordinates, or effects of the background parameter (which is detailed in the schema but not the description).

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single sentence that is front-loaded with the action. It is efficient but too brief to provide needed context, given the complexity of coordinate-based clicks.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 3 parameters, no output schema, and no annotations, the description is inadequate. It omits crucial behavioral details such as coordinate system origin, click simulation behavior, and how background mode works. The schema provides more detail for the background parameter, but the tool description itself is incomplete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is low (33%). The description adds meaning to x and y by specifying coordinate space, but does not explain the background parameter or their types. This partially compensates for the low coverage but is not comprehensive.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states it performs a left-click at given coordinates and specifies 'in screenshot space', which adds context. It implicitly distinguishes from right_click and double_click via the name, but does not explicitly differentiate from ax_click or other click tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided on when to use this tool versus sibling tools like ax_click or click_mark. The description does not indicate prerequisites, context, or exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_applicationsA

List all installed applications on the system.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

A4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden. It only states the tool lists installed applications, revealing no side effects, security implications, or performance characteristics. For a read-only listing, this is minimally adequate but lacks detail.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single concise sentence that efficiently communicates the tool's purpose without unnecessary words or structure issues.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no parameters, no output schema, and low complexity, the description is mostly complete. However, it omits what format the list returns (e.g., names only, objects), which could be clarified. Still sufficient for a simple list command.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has no parameters, and the schema coverage is 100% (empty schema). The description adds no parameter information because none is needed. Baseline 4 is appropriate as the description does not need to compensate for missing schema details.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action 'list' and the resource 'all installed applications on the system', making the purpose immediately obvious. It is distinct from sibling tools like 'open_application' or 'list_windows'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No explicit guidance on when to use this tool versus alternatives. The description implies usage for retrieving installed apps, but does not mention when not to use it or suggest alternatives for related tasks.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_windowsA

List all visible windows across all applications.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

A3.6/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, and the description does not disclose behavioral traits like read-only nature, speed, or permissions. It only states the basic function.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence with no wasteful content, efficiently conveying the tool's purpose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a parameterless tool with no output schema, the description adequately covers the core function. However, it could mention that the tool returns a list of window identifiers or details.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

No parameters exist, and schema coverage is 100%. The description does not add extra semantics but is sufficient given the absence of parameters.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool lists all visible windows across all applications, with a specific verb and resource, distinguishing it from sibling tools like list_applications.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided on when to use this tool versus alternatives such as list_applications or click tools, nor any context on prerequisites or limitations.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

mouse_moveB

Move the mouse cursor to the given (x, y) coordinates (in screenshot space).

ParametersJSON Schema
NameRequiredDescriptionDefault
xYes
yYes

TDQS

B3.2/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, and the description only states the action without disclosing side effects (e.g., hover triggers, coordinate bounds). Full burden on description, but it's minimal.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Single sentence, no wasted words, directly conveys the action.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Adequate for a simple move action, but lacks return value info or errors. With no output schema or annotations, slightly incomplete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0% (no descriptions in schema), but the description merely restates the parameter names 'x, y coordinates' without adding meaning like units, range, or coordinate system details.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb 'move' and resource 'mouse cursor', specifying coordinates in screenshot space, distinguishing it from siblings like click, double_click, etc.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool vs alternatives like click_mark or left_click. Only implicit from the name and description.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

ocrA

Read on-screen TEXT via the platform OCR engine (Apple Vision on macOS, Windows.Media.Ocr on Windows). Captures the display (or window="") and returns the recognized text lines, each with a clickable center in the same pixel space as a screenshot — so you can both READ the text and click a line with left_click(x, y). Returns text only (no image), so it is a cheap, fast way to pull text off the screen. Best when you need to read or click TEXT on a surface where marks come back empty — canvas, games, image-rendered or custom-drawn views — or to grab a lot of text at once without parsing a screenshot. For native/web UI with an Accessibility tree, screenshot(marks=true) + click_mark is still more precise. Languages default to Simplified Chinese + English; pass languages=[...] (BCP-47) to override.

ParametersJSON Schema
NameRequiredDescriptionDefault
windowNoCapture only a single on-screen window (case-insensitive substring of its title or app name) instead of the whole display. Smaller, sharper image → better recognition of small text.
languagesNoBCP-47 language hints in priority order (e.g. ["zh-Hans", "en-US"]). Omitted, defaults to Simplified Chinese + English.

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations exist, so description fully discloses behavior: returns text lines with clickable centers, cheap/fast, captures display or window, defaults to zh-Hans+en, and details pixel-space coordinates. No hidden side effects.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with purpose and engine. Every sentence adds unique value; no filler. Structured with use cases and comparisons.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema, but description explains return format (text lines with clickable centers). Low parameter count fully covered. Distinguishes from siblings. Complete for an AI agent to use correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with descriptions for both parameters. Description adds context (window substring match, language defaults) and explains why a smaller window improves recognition, enhancing the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states 'read on-screen TEXT' specifying verb and resource, identifies OCR engine, and distinguishes from screenshot/click tools by emphasizing text-only output.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly describes when to use OCR (canvas, games, custom-drawn views) and when to prefer screenshot+click_mark (native/web UI with accessibility tree), providing clear alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

open_applicationA

Launch or focus an application by name (e.g., "Safari", "Slack").

ParametersJSON Schema
NameRequiredDescriptionDefault
appYes

TDQS

A3.5/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must fully disclose behavior. It only states 'Launch or focus an application' without explaining what happens if the app is not found, whether it always focuses or launches, or any other behavioral traits (e.g., permissions, error handling).

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, front-loaded sentence with zero waste. Every word is necessary and contributes to the purpose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity (1 required param, no output schema), the description is adequate but misses edge cases like missing applications or multiple instances. With no annotations, it leaves the agent without guidance on error states or return values.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so description must compensate. It adds meaning by stating 'by name' and giving examples, but lacks specifics on format (e.g., case sensitivity, exact vs. partial match). This provides basic guidance but is incomplete for precise usage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description uses specific verbs ('Launch or focus') and resource ('application'), with concrete examples ('Safari', 'Slack'). It clearly distinguishes from sibling tools like 'list_applications' and 'ax_click' which serve different purposes.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies the tool is for launching or focusing an app by name, but does not explicitly state when to use it vs. alternatives (e.g., using 'list_applications' to find the correct name first). No exclusions or alternative suggestions are provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

read_clipboardA

Read the current system clipboard contents as text.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the burden. 'Read' implies non-destructive, and the simplicity of the tool (no parameters) makes additional disclosure unnecessary. The description is adequate.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence with no wasted words. It is appropriately sized and front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple, parameterless tool with no output schema, the description fully covers the intended behavior. No additional context is required.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

There are zero parameters, so schema coverage is 100%. The description adds no parameter info, but none is needed. Baseline score of 4 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ('Read') and resource ('system clipboard contents'), and specifies the format ('as text'). It clearly distinguishes from the sibling 'write_clipboard'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

While the purpose is clear, the description does not provide explicit guidance on when to use this tool versus alternatives, such as noting it is safe for reading without side effects.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

right_clickB

Right-click at the given (x, y) coordinates (in screenshot space).

ParametersJSON Schema
NameRequiredDescriptionDefault
xYes
yYes
backgroundNoDeliver in the background to the captured window's process (native apps only; browsers/Electron ignore it). Default false = foreground.

TDQS

B3.3/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden. It only states that it performs a right-click at coordinates, without disclosing what happens if coordinates are invalid, whether it is foreground or background (though background parameter is in schema), or any other behavioral traits.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, clear sentence with 12 words. It is appropriately sized and front-loaded, containing no unnecessary information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple click tool with no output schema and limited annotations, the description is adequate but minimal. It misses details on return value, error handling, or context for use alongside siblings.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The description adds meaning to x and y by specifying 'in screenshot space', which is not in the schema. However, it does not elaborate on the background parameter, and schema coverage is only 33%, so the description only partially compensates.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action (right-click) and the required parameters (coordinates) along with the coordinate space (screenshot space), distinguishing it from sibling tools like left_click and double_click.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided on when to use this tool versus alternatives like click_mark or ax_click. The description lacks any context about prerequisites or scenarios.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

screenshotA

Capture the screen — the whole main display, or a single window with window="" — and return a base64 JPEG plus a text note with its pixel dimensions. ALL coordinate-taking tools (mouse_move, *_click, scroll) expect coordinates in THIS image's pixel space — origin (0,0) top-left, x right, y down — so read target positions directly off the returned image; subsequent clicks are mapped through it automatically. PREFER window="" (substring of its title or app name) over the whole display whenever you are working inside one app: a full-display shot is downscaled to ~1280px wide, so small UI (list rows, sidebar items, buttons) becomes only a few pixels — too small to read or click accurately. A window capture is larger and sharper, returns a smaller image (fewer pixels → less context, faster turn), and clicks map into the window automatically. marks is ON by default: it boxes+numbers actionable elements (needs Accessibility) and lists each as [N] — activate one with click_mark(number=N), the most reliable way to click with no coordinate guessing. A magenta coordinate grid (for reading x/y) is shown automatically when marks is off and hidden when it is on; pass grid=true to force both, or marks=false for pure coordinate mode. If a target is still too small to click, use zoom_region to magnify part of this image.

ParametersJSON Schema
NameRequiredDescriptionDefault
gridNoOverlay a labeled coordinate grid (magenta rules + pixel labels) for reading off click coordinates. Omitted by default it follows the AX-first rule: OFF when `marks` is on (you click by number, not coordinates), ON when `marks` is off (coordinate mode). Pass grid=true to force it on alongside marks, or grid=false to suppress it.
marksNoSet-of-Mark: number every actionable UI element (buttons, links, fields) and list each as `[N] role "label"`, so you can activate it with click_mark(number=N) — the most reliable targeting, no coordinate guessing. Defaults ON (AX-first). Needs Accessibility permission. Covers native controls AND web content — real links/buttons on semantic pages, and on div-rendered pages (e.g. webmail) the list rows are numbered too (their click lands via a coordinate at the row center). Only canvas/game- style surfaces with no AX come back empty. Pass marks=false for pure coordinate mode.
windowNoCapture only a single on-screen window instead of the whole display — a case-insensitive substring of the window title or app name (e.g. "Safari", "Settings"). Smaller, sharper image = less context and better click precision. Subsequent clicks map to this window automatically.

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, description fully discloses behaviors: downscaling of full display, coordinate system, default marks-on behavior, grid display logic, and limitations for canvas surfaces.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Well-structured with clear sections, but somewhat lengthy. Could be trimmed slightly without loss of essential information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given complexity (3 optional params, no output schema), description completely covers return format, coordinate mapping, defaults, and fallback suggestions.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema has 100% parameter descriptions, but description adds valuable context: default interaction between grid and marks, substring matching for window, and preference for window capture.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description explicitly states it captures screen or window and returns base64 JPEG with dimensions. It also explains how coordinates relate to the image, clearly distinguishing it from sibling tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides clear when-to-use guidance: prefer window capture when working inside an app, when to use marks vs grid, and when to fall back to zoom_region for small targets.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

scrollC

Scroll at the given (x, y) position. Positive lines = up, negative = down.

ParametersJSON Schema
NameRequiredDescriptionDefault
xYes
yYes
linesYes
backgroundNoDeliver in the background to the captured window's process (native apps only). Default false = foreground.

TDQS

C2.4/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided. The description only explains line direction but does not clarify how x and y coordinates interact with lines, nor the effect of the background parameter beyond the schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, no fluff, but could be more structured with clearer parameter relationships.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With 4 parameters (3 required) and no output schema, the description is incomplete. It fails to explain the role of x and y, which are critical for usage.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 25% (only background parameter described). The description adds meaning for 'lines' (positive=up, negative=down) but does not explain x and y, leaving ambiguity.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose3/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states it scrolls, but the phrase 'at the given (x, y) position' is ambiguous given the 'lines' parameter. It does not clearly distinguish from other scrolling approaches among sibling tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool versus alternatives like mouse_move or key_combo. The description lacks usage context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

type_textB

Type a string of text into the currently focused element.

ParametersJSON Schema
NameRequiredDescriptionDefault
textYes
backgroundNoDeliver in the background to the captured window's process (native apps only). Default false = foreground.

TDQS

B3.4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description discloses the basic behavior (typing text into the focused element) but fails to mention the effect of the 'background' parameter or handle edge cases (e.g., no focused element).

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single sentence, no waste. However, it is slightly too minimal for a tool with two parameters.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple tool, the description covers the core action but omits the 'background' parameter's role and usage distinctions from siblings like ax_set_value.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The description adds meaning to the 'text' parameter by indicating it's the string to type, but does not explain the 'background' parameter. With 50% schema coverage, partial compensation is provided.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Type a string of text') and the target ('into the currently focused element'), distinguishing it from siblings like key_combo or ax_set_value.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool versus alternatives such as key_combo or ax_set_value. The description is purely functional without contextual advice.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

waitB

Wait for a specified number of seconds before returning.

ParametersJSON Schema
NameRequiredDescriptionDefault
durationNo

TDQS

B3.4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries full burden. It implies synchronous blocking for the specified duration, which is the core behavior. However, it doesn't disclose that the tool may be blocking, has no side effects, or disallows negative durations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single sentence with 10 words is highly concise and front-loaded. Every word is necessary to convey the core functionality with no redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity (one parameter, no output schema), the description is largely complete. It could mention that the wait is blocking and that the tool returns after the duration, but for a trivial tool this is sufficient.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It mentions 'specified number of seconds,' which ties to the 'duration' parameter but adds no further detail (e.g., supports fractional seconds, precision limits). Baseline is low due to schema gap; the description provides minimal clarification.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool waits a specified number of seconds, using a verb ('wait') and resource ('seconds'). It's unambiguous and distinct from sibling tools, which are all actions (click, type, etc.). However, it could improve by explicitly noting it's a pause in automation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool versus alternatives. There are no exclusions or context, such as recommending it for delaying between steps or avoiding during time-sensitive operations.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

write_clipboardC

Write text to the system clipboard.

ParametersJSON Schema
NameRequiredDescriptionDefault
textYes
backgroundNoDeliver in the background to the captured window's process (native apps only). Default false = foreground.

TDQS

C2.2/5.0
Behavior1/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, and the description fails to disclose important behaviors like whether it overwrites the clipboard, permission requirements, or any side effects. This is a significant gap for a write tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single short sentence, which is concise but too minimal. It earns its place but lacks sufficient detail to be highly useful.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness1/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given two parameters, no output schema, and low schema coverage, the description is incomplete. It fails to explain parameter usage or return behavior, making it insufficient for correct tool invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 50% schema description coverage, the description does not add meaning beyond the schema. It does not explain the 'text' parameter or the 'background' parameter's behavior, which the schema partially describes.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool writes text to the clipboard, using a specific verb and resource. However, it does not differentiate from sibling tools like read_clipboard or type_text.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool versus alternatives such as read_clipboard or type_text. The tool writes to the clipboard, but no context or exclusions are provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

zoom_regionA

Zoom into a rectangle of the CURRENT image (the last screenshot's pixel space) and re-capture it at native resolution — a sharp, legible magnified view. Only that rectangle is captured (not the whole display), so it is also smaller and quicker to take than a full-display shot. Use it to read exact positions on surfaces that expose no marks (canvas/games, custom-rendered views) before clicking, or to stay scoped while working inside one area. Pass x, y, width, height in the current image's pixels (width,height > 0). The returned image becomes the new coordinate space, and clicks afterward map into the zoomed region automatically. marks defaults OFF (read coordinates off the overlaid grid); grid defaults ON. Take a screenshot first so there is an image to zoom into.

ParametersJSON Schema
NameRequiredDescriptionDefault
xYesLeft edge of the rectangle, in the CURRENT image's pixel space (the last screenshot's coordinates).
yYesTop edge of the rectangle, in the current image's pixel space.
gridNoOverlay a labeled coordinate grid. Defaults ON for a zoom (coordinate mode); pass grid=false to suppress it.
marksNoSet-of-Mark numbering. Defaults OFF for a zoom — the zoom is the tool for surfaces that expose no marks, so you read coordinates off the grid. Pass marks=true to also number any actionable elements inside the region.
widthYesWidth of the rectangle in current-image pixels. Must be > 0.
heightYesHeight of the rectangle in current-image pixels. Must be > 0.

TDQS

A5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description fully explains behavior: re-captures only the rectangle (smaller and quicker), the image becomes the new coordinate space, clicks map automatically, and defaults for marks (OFF) and grid (ON) are stated. No contradictions.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Every sentence is informative: purpose, benefits, use cases, parameter details, coordinate mapping, defaults. No redundancy. Well-structured with front-loaded key information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite no output schema, the description explains the outcome (new image as coordinate space) and side effects. Covers all 6 parameters' roles, defaults, and constraints. Complete for a tool with coordinate transformation complexity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, but the description adds context: explains pixel space coordinates, rectangle constraints (width, height > 0), defaults for grid and marks, and the coordinate transformation effect. This adds significant meaning beyond schema property descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool zooms into a rectangle of the current image and re-captures it at native resolution, distinguishing it from full-display screenshot tools. The verb 'zoom' and resource 'rectangle of the current image' are specific, and the description contrasts with a full-display shot.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly states when to use: to read exact positions on surfaces with no marks (canvas/games, custom-rendered views) or to stay scoped in one area. Implicitly advises against using when marks are available, and describes the coordinate remapping effect for subsequent clicks.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. 23 tool updatesv0.1.0
    • First observedax_click
    • First observedax_focus
    • First observedax_set_value
    • First observedbatch_actions
    • First observedclick_mark
    • First observedcursor_position
    • First observeddouble_click
    • First observeddump_ax
    • First observedkey_combo
    • First observedleft_click
    • First observedlist_applications
    • First observedlist_windows
    • First observedmouse_move
    • First observedocr
    • First observedopen_application
    • First observedread_clipboard
    • First observedright_click
    • First observedscreenshot
    • First observedscroll
    • First observedtype_text
    • First observedwait
    • First observedwrite_clipboard
    • First observedzoom_region

TDQS

A3.7/5.0
Disambiguation5/5

Each tool has a clearly distinct purpose, with detailed descriptions that prevent confusion between similar actions like ax_click, click_mark, and left_click.

Naming Consistency4/5

Tool names are descriptive but follow varying patterns (verb_noun, noun_noun, adjective_noun), with some prefixes like 'ax_' and 'list_'. Overall readable but not perfectly uniform.

Tool Count5/5

23 tools cover a comprehensive range of UI automation tasks (mouse, keyboard, accessibility, OCR, screenshots, clipboard, app management) without being excessive.

Completeness5/5

The tool set covers all essential UI interactions including clicking, typing, scrolling, accessibility, OCR, screenshots, clipboard, and app/window management. No significant gaps are apparent.

Maintenance

ActivityMaintained
ResponsivenessResponsive

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

  • A
    license
    Not graded
    quality
    C
    maintenance
    Standalone MCP server that gives AI agents full GUI control over macOS — screenshots, mouse, keyboard, apps, clipboard, and multi-display — with zero private dependencies.
    18
    MIT
  • A
    license
    Not graded
    quality
    D
    maintenance
    Enables AI agents to see and control your desktop with tools for screenshots, clicks, typing, and more, all locally on macOS and Windows.
    119
    MIT
  • A
    license
    A
    quality
    D
    maintenance
    Enables AI agents to control macOS desktop apps via screenshots, mouse clicks, keyboard input, accessibility queries, and AppleScript.
    11
    15
    MIT

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/bigduu/Nova'

If you have feedback or need assistance with the MCP directory API, please join our Discord server