macbot-mcp

An MCP server that gives AI agents hands on macOS — via the Accessibility API, AppleScript, and Quartz CGEvents.

The macOS counterpart to ahk-mcp. Same thesis: the accessibility tree already contains a machine-readable description of everything on screen. Screenshots throw that away and make the model re-derive it from pixels. Why?

Token cost per action: ~200-700 tokens (vs ~2000-3500 for screenshot-based).

Because the output is structured text, not images, any language model can drive it — including small open-source models with no vision capability. A 7B model can parse AXButton title="Save" @450,320 88x32 and call mac_click(x=490, y=336). It cannot interpret a screenshot. The accessibility approach makes computer use available to models that were previously locked out of it entirely.

How it works

macbot-mcp exposes 15 tools over MCP's stdio transport:

• Observation tools read the macOS accessibility tree, window properties, and browser URLs — returning structured text with element roles, names, values, and screen coordinates

• Action tools click, drag, scroll, type, and send keystrokes via Quartz CGEvents and AppleScript

• mac_run_applescript is the escape hatch — execute arbitrary AppleScript for anything the built-in tools don't cover

Every observation tool returns coordinates. Find a button with mac_ui_find, get its x,y position, and click it with mac_click — no screenshot needed.

Related MCP server: Automation MCP

Token cost comparison

A 20-step workflow: ~50k tokens with screenshots, ~8k with macbot. The structured output is also more reliable — the model doesn't guess where the "Save" button is when the accessibility tree says AXButton title="Save" @1043,672 88x32.

Installation

Prerequisites

• macOS 12+ (Monterey or later)

• Python 3.10+

Setup

git clone https://github.com/anomalous3/macbot-mcp.git
cd macbot-mcp

# Create a virtual environment
python3 -m venv .venv
# or: uv venv .venv

# Activate
source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt

macOS Permissions

macbot needs two permissions, granted to your terminal app (Terminal, kitty, iTerm2, etc.):

1. Accessibility — System Settings > Privacy & Security > Accessibility

2. Screen Recording — System Settings > Privacy & Security > Screen Recording

Without Accessibility, observation and input tools won't work. Without Screen Recording, mac_screenshot will fail.

Three gotchas:

• Granting is not enough — fully quit and reopen the terminal (Cmd-Q, not just a new tab/window). Running processes keep their old permission state.

• macOS 15+ periodically re-asks for Screen Recording. If screenshots worked yesterday and now fail with could not create image from display (and mac_get_windows returns []), respond to the re-consent prompt in System Settings, then restart the terminal.

• If you run Claude Code inside an IDE's integrated terminal (VS Code etc.), grant the permissions to the IDE, not Terminal.app.

Claude Code MCP configuration

Add to your ~/.claude.json:

{
  "mcpServers": {
    "macbot": {
      "command": "/path/to/macbot-mcp/.venv/bin/python3",
      "args": ["/path/to/macbot-mcp/server.py"],
      "env": {
        "MACBOT_SCREENSHOT_DIR": "/tmp/macbot"
      }
    }
  }
}

After adding the config, restart Claude Code. The tools appear with the mcp__macbot__ prefix.

Tool reference

Observation

Action

The focus race (why app= exists)

An agent's tool calls are separated by seconds of thinking time. If you call mac_focus_app in one tool call and mac_type_text in the next, the terminal running the agent frequently reclaims focus in the gap — and the text lands in the agent's own input box instead of the target app. Focus and action must happen inside a single call. That's what the app= parameter on mac_type_text and mac_key_press does: activate, wait until the app is verified frontmost, then act.

Escape hatch

Browser automation via the Accessibility API

Modern browsers expose their full UI through the macOS Accessibility API. macbot reads this directly — no browser extension, no WebDriver, no Playwright needed.

Read the tab bar with element coordinates:

> mac_ui_tree app="Firefox" max_depth=5 max_nodes=50

AXApplication title="Firefox"
  AXWindow title="GitHub - anomalous3/macbot-mcp" @36,30 1752x957
    AXGroup desc="GitHub - anomalous3/macbot-mcp" @36,30 1752x957
      AXToolbar desc="Browser tabs" @36,30 1752x44
        AXTabGroup (tab group) @190,30 1518x44
          AXRadioButton title="GitHub - anomalous3/macbot-mcp" value="True" @193,30 210x44
          AXRadioButton title="New Tab" @403,30 210x44

Find a specific element and get its click coordinates:

> mac_ui_find app="Firefox" name="Submit" role="AXButton"

{
  "match_count": 1,
  "window_titles": ["GitHub - anomalous3/macbot-mcp"],
  "matches": [
    {"role": "AXButton", "title": "Submit", "x": 450, "y": 320, "width": 80, "height": 32}
  ]
}

The window titles come back with every search, so an empty result tells you why it was empty — wrong page, still loading, or the window is gone.

Press it without coordinates — no mouse, no focus change:

> mac_ui_press app="Firefox" name="Submit" role="AXButton"

{"status": "ok", "pressed": {...}, "match_count": 1}

Get the URL without screenshots or clipboard tricks:

> mac_ui_url app="Firefox"

{"url": "https://github.com/anomalous3/macbot-mcp", "field": "Search with Google or enter address"}

Use Firefox. It exposes the richest accessibility tree of the major browsers — more element detail, better labeling, and more consistent structure than Chrome or Safari.

The coordinate system

All coordinates are absolute screen pixels in macOS logical coordinates (not Retina physical pixels). Origin (0,0) is the top-left of the primary display.

The @x,y WxH format in mac_ui_tree output gives position and size directly. To click the center of @450,320 80x32, click at (490, 336).

Works with everything

macbot works with any macOS application that implements the Accessibility API (which is most of them):

• Browsers — Firefox, Chrome, Safari (Firefox recommended for richest tree)

• Terminals — kitty, Terminal.app, iTerm2 (can read content, type commands)

• Editors — VS Code, Sublime Text, TextEdit

• System apps — Finder, System Settings, Activity Monitor

• Any app — if it has windows and controls, macbot can probably read and drive it

Configuration

Platform

macOS only. For Windows, see ahk-mcp. The approach is the same — read the accessibility tree, act via synthetic input — just different platform APIs.

License

MIT