Skip to main content
Glama

CI CodeQL Coverage Workflow npm GitHub stars License: MIT

OpenSSF Scorecard Dependabot npm provenance workflow npm weekly downloads Node >= 20 Smithery mcp-observatory MCP server All Contributors Gitee Stars Gitee Forks MCP Registry MCP Market MCP Hub China OpenTools Gitee

Secure the MCP servers you're building. MCP Observatory is the CI-native security tool for teams shipping custom MCP servers. Test during development, catch schema drift, simulate attacks, and generate compliance evidence — before agents depend on your servers.

Also available in Simplified Chinese.

Runtime enforcement: Use mcp-seatbelt to block dangerous MCP tool calls at runtime based on observatory scan results.

Get Started

Step-by-step setup guide · macOS, Windows, Linux · Node.js 20+ and npm (install Node.js LTS).

1. Run your first scan

Open Terminal or PowerShell in your project folder and paste:

npx -y @kryptosai/mcp-observatory@latest

This downloads Observatory and checks your first configured MCP server. With no configured server, it uses an included example. To try only that example, run npx -y @kryptosai/mcp-observatory@latest demo --example.

2. Read the result

The terminal shows the target, grade, findings, and saved receipt path. Failed checks are findings to review, not a request to pay. If connection failed, fix the startup command or credentials and rerun. A demo grade describes the example, not your own server.

3. Save one hosted snapshot free (optional)

In the same project folder, run:

npx -y @kryptosai/mcp-observatory@latest cloud upload

Open the verification link printed in the terminal, confirm the code, connect with GitHub, then return to the terminal. Wait for Upload complete and open the dashboard link. No separate account setup or payment method is needed. Free includes one target, the latest snapshot, seven-day retention, and ten uploads per month.

Local scans and local CI stay free. Individual Pro is $29/month when you need 10 targets, 90-day history, hosted CI ingestion, or hosted HTTP scans.

MCP Observatory uses jurisdiction-aware product telemetry after showing the collection notice. Review the exact fields with telemetry preview, check the current policy with telemetry status, or stop collection at any time with telemetry disable, MCP_OBSERVATORY_TELEMETRY=0, or DO_NOT_TRACK=1. See the complete privacy and telemetry notice.

npx -y @kryptosai/mcp-observatory@latest test npx -y @modelcontextprotocol/server-everything
- uses: KryptosAI/mcp-observatory/action@v1

If this repo is an MCP package, that tests npx -y <package.json name>. Otherwise set command:.

Or generate a full workflow: npx -y @kryptosai/mcp-observatory@latest setup-ci --all --command "npx -y my-mcp-server" --sarif

Install: npx -y @kryptosai/mcp-observatory@latest · brew tap kryptosai/mcp-observatory https://github.com/KryptosAI/mcp-observatory && brew install mcp-observatory · docker pull ghcr.io/kryptosai/mcp-observatory:latest · curl -fsSL https://raw.githubusercontent.com/KryptosAI/mcp-observatory/main/scripts/install.sh | sh

docker run --rm ghcr.io/kryptosai/mcp-observatory:latest demo
docker run --rm ghcr.io/kryptosai/mcp-observatory:latest test npx -y @modelcontextprotocol/server-everything

Scanning a server on the host needs network and, for local files, a volume mount (-v "$PWD:$PWD" -w "$PWD").

Agents: npx -y @kryptosai/mcp-observatory@latest serve

Default-deny connect: npx -y @kryptosai/mcp-observatory@latest protect rewrites every discovered client config (Cursor, Claude, VS Code, .mcp.json, …). Undo: protect --undo. HTTP/SSE servers are left alone.

Runtime deny-default: npx -y @kryptosai/mcp-observatory@latest enforce --start-proxy writes a Seatbelt policy from the scan and starts the proxy. Local scan stays free.

Proof: Safety Index. Upload one hosted snapshot free with cloud upload; Individual Pro is $29/month when you need retained history and hosted CI ingestion.

Related MCP server: SilentFail

Why MCP Observatory

MCP servers are becoming production dependencies. If agents rely on them, teams need a way to catch broken tools, unsafe schemas, schema drift, slow responses, and security footguns before those failures reach users.

Observatory gives maintainers and teams:

  • One-command CI setup with setup-ci --all

  • Profile-mapped audits with audit --profile nsa-mcp

  • MCP receipts that package target, evidence, verdict, action, and reproduction commands

  • MCP risk graphs that group servers by capability boundary, receipt state, CI posture, and recommended action

  • Action receipts that say allow, gate, rerun, quarantine, or escalate

  • GitHub PR comments for compatibility, drift, and security findings

  • GitHub Code Scanning SARIF for normalized MCP findings

  • Health score badges for public trust signals

  • Record/replay/verify workflows for regression testing

  • MCP server mode so agents can inspect other MCP servers directly

  • Production support path for one-user hosted history, CI ingestion, and owner-ready Release Gate remediation

See GitHub Code Scanning for MCP servers, MCP Receipts, Troubleshooting, Safety Methodology, MCP Server Safety Index, MCP Observatory Contributors, hosted client contract, repository boundary, open core boundary, and commercial support.

Self-Assessment

We scan ourselves with mcp-observatory on every release. See results →

For Security And Platform Teams

MCP servers are becoming part of the AI software supply chain. Agents need reliable, testable, auditable tools before those tools become dependencies in mission-critical workflows.

Whether you're shipping one MCP server or running a fleet, MCP Observatory gives you CI-native security scoring, attack simulation, schema drift detection, SARIF/HTML/Markdown reports, and GitHub Code Scanning — from your first npx command to production deployment. Local development stays free; teams with a near-term production approval decision can use the fixed-scope MCP Release Gate Pilot.

Production Support

Local OSS use stays free under MIT. Teams running MCP in production can use the MCP Release Gate Pilot for safe-mode evidence, SARIF/Code Scanning setup, CI rollout, private reporting, and owner-ready remediation notes. The fixed public entry offer is $15,000 for 1-3 critical MCP servers over ten business days; broader work is scoped after the release decision.

The open source repo is the portable evidence engine. Hosted authentication, retention, organization workflows, fleet coordination, and private intelligence stay outside the OSS package; see the repository boundary.

Run npx @kryptosai/mcp-observatory cloud, open a pilot request from the issue chooser, or see COMMERCIAL.md. Also see privacy, campaign attribution, and terms for production use.

How It Compares

Feature

mcp-observatory

Snyk agent-scan

Cisco mcp-scanner

agent-shield

MCP-native

Attack simulation

Schema drift detection

Record/replay/verify

Health scoring (0-100)

SARIF output

CI/CD native (setup-ci)

Public Safety Index

Runtime enforcement via mcp-seatbelt

Quick Start

Run a first check. No arguments starts demo (your servers, or the packaged local demo):

npx @kryptosai/mcp-observatory

Go deeper — also invoke safe tools to verify they actually run:

npx @kryptosai/mcp-observatory scan deep

Test a specific server:

npx @kryptosai/mcp-observatory test npx -y @modelcontextprotocol/server-everything

Add it to Claude Code as an MCP server:

claude mcp add mcp-observatory -- npx -y @kryptosai/mcp-observatory serve

Or add it manually to your config:

{
  "mcpServers": {
    "mcp-observatory": {
      "command": "npx",
      "args": ["-y", "@kryptosai/mcp-observatory", "serve"]
    }
  }
}

Commands

Command

What it does

scan

Auto-discover servers, check them, and run safe attack-readiness simulation by default

scan deep

Scan, run safe attack simulation, and also invoke safe tools to verify they execute

test <cmd> / test --target <file>

Test one server and emit an action receipt by command or target config

record <cmd>

Record a server session to a cassette file for offline replay

replay <cassette>

Replay a cassette offline — no live server needed

verify <cassette> <cmd>

Verify a live server still matches a recorded cassette

diff <base> <head>

Compare two run artifacts for regressions and schema drift

watch <config>

Watch a server for changes, alert on regressions

suggest

Detect your stack and recommend MCP servers from the registry

serve

Start as an MCP server for AI agents

lock

Snapshot MCP server schemas into a lock file

lock verify

Verify live servers match the lock file

history

Show health score trends for your MCP servers

setup-ci / init-ci

Create a GitHub Action and badge snippet for MCP compatibility/security checks

setup-ci --sarif

Generate a workflow that uploads normalized findings to GitHub Code Scanning

setup-ci --doctor

Inspect whether the repository has a complete CI adoption kit

telemetry status|enable|disable|identify|preview

Inspect or control product telemetry and deliberately provide a contact email

risk-graph --input <path>

Merge receipts and run artifacts into JSON, Markdown, and HTML MCP risk graphs

--no-attack-sim

Opt out of the default safe attack simulation on scan or test

ci-report

Generate CI report for GitHub issue creation

enterprise-report

Generate a static production/security report from run artifacts

score <cmd>

Score an MCP server's health (0-100)

badge <cmd>

Generate an SVG health score badge for README

cloud

Show the free hosted snapshot, Individual Pro, and Release Gate Pilot options

Run with no arguments to start demo. Use --menu for the interactive catalog.

What It Does

Check capabilities — connects to a server and verifies tools, prompts, and resources respond correctly.

Invoke tools — goes beyond listing. Actually calls safe tools (no required params / readOnlyHint) and reports which ones work and which ones crash.

npx @kryptosai/mcp-observatory scan deep

Detect schema drift — diffs two runs and surfaces added/removed fields, type changes, and breaking parameter changes.

npx @kryptosai/mcp-observatory diff run-a.json run-b.json

Recommend servers — scans your project for languages, frameworks, databases, and cloud providers, then cross-references the MCP registry to suggest servers you're missing.

npx @kryptosai/mcp-observatory suggest

Or ask your agent "what MCP servers should I add?" when running in MCP server mode.

Security scanning — analyzes tool schemas for dangerous patterns: shell injection surfaces, broad filesystem access, missing auth, and credential leakage in responses.

npx @kryptosai/mcp-observatory test --security npx -y my-mcp-server

Record / replay / verify — capture a live session, replay it offline in CI, and verify nothing changed. Like VCR for MCP.

# Record a session
npx @kryptosai/mcp-observatory record npx -y @modelcontextprotocol/server-everything

# Replay offline (no server needed)
npx @kryptosai/mcp-observatory replay .mcp-observatory/cassettes/latest.cassette.json

# Verify the live server still matches
npx @kryptosai/mcp-observatory verify cassette.json npx -y @modelcontextprotocol/server-everything

Watch for regressions — re-runs checks on an interval and alerts when something changes.

npx @kryptosai/mcp-observatory watch target.json

Scan locations

When you run scan, it looks for MCP configs in:

  • ~/.claude.json (Claude Code)

  • ~/Library/Application Support/Claude/claude_desktop_config.json (Claude Desktop, macOS)

  • %APPDATA%/Claude/claude_desktop_config.json (Claude Desktop, Windows)

  • .claude.json and .mcp.json (current directory)

Architecture

                    ┌─────────────────────────┐
                    │   MCP Observatory CLI    │
                    │  npx @kryptosai/mcp-     │
                    │     observatory scan     │
                    └───────────┬─────────────┘
                                │
                    ┌───────────▼─────────────┐
                    │   Config Discovery       │
                    │  (Claude, Cursor, etc.)  │
                    └───────────┬─────────────┘
                                │
              ┌─────────────────┼─────────────────┐
              ▼                 ▼                  ▼
    ┌─────────────────┐ ┌──────────────┐ ┌──────────────────┐
    │   Security Scan  │ │  Attack Sim  │ │  Schema Drift    │
    │  (shell, creds)  │ │ (tool poison)│ │  (version diff)  │
    └────────┬────────┘ └──────┬───────┘ └────────┬─────────┘
             │                 │                   │
             └─────────────────┼───────────────────┘
                               ▼
                    ┌─────────────────────┐
                    │   Health Score       │
                    │  (0-100 + verdict)   │
                    └──────────┬──────────┘
                               │
              ┌────────────────┼────────────────┐
              ▼                ▼                 ▼
    ┌──────────────┐  ┌──────────────┐  ┌──────────────┐
    │  SARIF       │  │  Markdown    │  │  CI Gateway  │
    │  (Code Scan) │  │  Report      │  │  (setup-ci)  │
    └──────────────┘  └──────────────┘  └──────────────┘

CI / GitHub Action

Add Observatory to your MCP server's CI pipeline:

npx @kryptosai/mcp-observatory setup-ci --all --command "npx -y my-mcp-server" --sarif --schedule weekly

Check the adoption kit:

npx @kryptosai/mcp-observatory setup-ci --doctor

Successful test, run, and single-target scan checks also offer to convert the passing result into a CI adoption kit. That automatic conversion enables SARIF/Code Scanning and weekly scheduled checks by default; pass --no-ci-sarif when you only want a conservative workflow without Code Scanning upload.

Or create the workflow manually:

# .github/workflows/observatory.yml
name: MCP Server Check
on: [pull_request]

permissions:
  contents: read

jobs:
  observatory:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: KryptosAI/mcp-observatory/action@v1
        with:
          command: npx -y my-mcp-server
          deep: true
          security: true
          comment-on-pr: false
          set-status: false

GitLab CI

Runs the same test scan on merge requests and on main, publishing the SARIF as a GitLab SAST report so findings appear in the merge request security widget. This matches what setup-ci --ci-provider gitlab-ci generates.

# .gitlab-ci.yml
mcp-observatory:
  image: node:22
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'
    - if: $CI_COMMIT_BRANCH == 'main'
  script:
    - npx @kryptosai/mcp-observatory test npx -y my-mcp-server --deep --security --sarif mcp-observatory.sarif
  artifacts:
    reports:
      sast: mcp-observatory.sarif

Or let the CLI write it for you:

npx @kryptosai/mcp-observatory setup-ci --ci-provider gitlab-ci --command "npx -y my-mcp-server"

Action inputs:

Input

Description

Default

command

Server command to test

(optional; omit to scan discovered servers)

target

Path to target config JSON

targets

Path to MCP config file for multi-server matrix scan

deep

Also invoke safe tools

false

security

Run security analysis

false

fail-on-regression

Fail the action on issues

true

fail-on-baseline-drift

Fail the action when baseline verification detects drift

true

comment-on-pr

Post report as PR comment. Requires pull-requests: write.

true

set-status

Set a commit status check (green/red) on the HEAD SHA. Requires statuses: write.

true

github-token

Token for PR comments and commit statuses

${{ github.token }}

The action can comment on PRs and set commit statuses when the workflow grants write permissions. setup-ci generates read-only third-party-friendly workflows by default and lets maintainers opt into comments/statuses later. init-ci remains available as a backward-compatible alias. See action/README.md for all options.

Production teams with a near-term MCP approval decision can use the fixed-scope MCP Release Gate Pilot: an approve, gate, or defer decision for 1–3 servers in ten business days. See COMMERCIAL.md or request a decision at mcp-observatory.com/release-gate-pilot.

Evidence badges for MCP Observatory

MCP server maintainers can add a public compatibility/security signal to their README:

[![MCP Observatory](https://img.shields.io/badge/MCP%20Observatory-enabled-2563eb)](https://github.com/KryptosAI/mcp-observatory)

Or generate a score badge from a live check:

npx @kryptosai/mcp-observatory badge npx -y my-mcp-server --output docs/mcp-health.svg

See the evidence distribution loop for the GitHub Action template, maintainer PR body, and badge rollout playbook. A badge is a public evidence signal, not a certification or endorsement.

Generate a pilot-ready production/security report from local run artifacts:

npx @kryptosai/mcp-observatory enterprise-report \
  --account "Your Company" \
  --format html \
  --output observatory-enterprise-report.html

For clearer internal account attribution in CI, set:

MCP_OBSERVATORY_ORG=your-company.com

To deliberately associate a contact email and optional contact-channel slug with private telemetry analytics, run mcp-observatory telemetry identify --email <address> [--channel <slug>]. Setting an organization or identifying a contact does not authorize outreach.

Testing Feishu/Lark integrations? See the Feishu/Lark MCP guide.

Lock Files

$ npx @kryptosai/mcp-observatory lock              # Snapshot all server schemas
$ npx @kryptosai/mcp-observatory lock verify        # Verify no drift since last lock

Lock files are the package-lock for AI tools: commit the MCP contract, then make every tool, schema, prompt, or resource drift visible in CI. See MCP lock files.

Trend Tracking

$ npx @kryptosai/mcp-observatory history            # Show health trends over time

Nightly Scans

$ npx @kryptosai/mcp-observatory ci-report          # Generate regression report for CI

MCP Server Mode

No other testing tool is itself an MCP server. Add Observatory as a server and your AI agent can autonomously test, diagnose, and monitor your other MCP servers.

claude mcp add mcp-observatory -- npx -y @kryptosai/mcp-observatory serve

Your agent gets 10 tools:

Tool

When to use it

scan

Check if all your configured MCP servers are healthy

check_server

Test a specific server before installing or after updating

score_server

Get a quick health score and grade for a server

record

Capture a baseline of a working server for future comparison

replay

Test against a recorded session — no live server needed

verify

Confirm a server update didn't break anything

watch

Check a server and see what changed since the last check

diff_runs

Find regressions between two check results

get_last_run

Retrieve previous check results for a server

suggest_servers

Discover MCP servers that match your project stack

An AI tool that checks other AI tools. It is a tool testing tools that serve tools.

Security

The MCP server runs inside AI hosts where an LLM chooses which tools to call. To prevent prompt-injection attacks:

  • Command allowlist: Only npx, node, python, python3, uvx, docker, deno, bun are permitted as base executables. The CLI has no restrictions.

  • Path validation: File-reading tools are constrained to the runs/cassettes directories.

  • No arbitrary execution: Use the CLI for unrestricted commands.

CLI vs MCP: Intentional Differences

Feature

CLI

MCP Server

Why

watch

Polling loop

Single check + diff

Request/response doesn't support long-polling

Interactive menu

Arrow-key navigation

Not available

MCP has no interactive UI

Color output

--no-color flag

Always plain text

MCP returns structured content

report

Renders saved artifacts

Not available

Agents read artifacts directly

serve

Starts MCP server

N/A

Is the MCP server

run

Reads target config files

Inline params

MCP tools accept params directly

get_last_run

Not available (use ls + diff)

Available

Convenience for agents

Compatibility

Works with any MCP server that uses standard transports:

Transport

Examples

Adapter

stdio (most servers)

filesystem, memory, context7, brave-search, sentry, notion, stripe

local-process

HTTP/SSE (remote)

Cloudflare, Exa, Tavily

http

Docker

All @modelcontextprotocol/server-* images

local-process via docker run -i

Servers needing API keys work via env in the target config. Python servers work via uvx. See the full compatibility matrix for tested servers and known issues.

Target config files

For more control (env vars, metadata, custom timeout):

{
  "targetId": "filesystem-server",
  "adapter": "local-process",
  "command": "npx",
  "args": ["-y", "@modelcontextprotocol/server-filesystem", "."],
  "timeoutMs": 15000,
  "skipInvoke": false
}
npx @kryptosai/mcp-observatory run --target ./target.json

HTTP / SSE targets

{
  "targetId": "my-remote-server",
  "adapter": "http",
  "url": "https://mcp.example.com/mcp",
  "authToken": "${MCP_SERVER_TOKEN}",
  "headers": {
    "X-Api-Key": "$MCP_SERVER_API_KEY"
  },
  "timeoutMs": 15000
}

Target configs support ${VAR}, $VAR, and env:VAR references in authToken, headers, and local-process env values.

How It Compares

Feature

Observatory

mcp-recorder

MCPBench

mcp-jest

Auto-discover servers

Check capabilities

Invoke tools

Schema drift detection

Record / replay

Verify against cassette

Response snapshot diffs

Benchmarking / latency

Jest integration

Works as MCP server

Each tool has strengths. Observatory focuses on regression detection and CI-friendly workflows. mcp-recorder is great as a transparent proxy. MCPBench is the go-to for performance benchmarking. mcp-jest is ideal if you're already in a Jest workflow.

Prior Art

The record/replay/verify pattern is inspired by:

  • VCR (Ruby) — pioneered cassette-based HTTP record/replay

  • Polly.js (Netflix) — HTTP interaction recording for JavaScript

  • mcp-recorder — MCP-specific traffic recording proxy

  • MCPBench — MCP server benchmarking

  • mcp-jest — Jest-style testing for MCP servers

Limitations

  • Servers requiring interactive OAuth (e.g., Google Drive) need pre-authentication before Observatory can connect

  • Custom WebSocket transports (e.g., BrowserTools MCP) are not supported

  • A few servers time out or close before init — see known issues and compatibility

Works with mcp-seatbelt

Scan before you trust. Then run npx -y @kryptosai/mcp-observatory@latest enforce --start-proxy — Observatory writes a deny-default mcp-seatbelt policy and starts the proxy. Observatory validates; seatbelt enforces.

Works with agent-obs

Secure your servers with Observatory. Trace your agents with agent-obs — an open-source agent execution tracer that records every tool call, computes A-F session grades, and shows you exactly where your agents spend time, burn tokens, and hit errors. Observatory tells you if a server is safe. agent-obs tells you what your agent did with it. Free, local-first, npm install -g agent-obs.

Contributors ✨

Thanks to these amazing people who have contributed:

  • leemeo3 — 3 Safety Index targets (Git, Chrome DevTools, Filesystem MCP)

  • albatrossflyon-coder — GitHub MCP Safety Index (#201)

  • tanishxdev — Legacy CLI deprecation warnings (#187)

  • sansynx — CLI format validation (#182)

See all contributors →

Contributing

We welcome contributors! This project follows a Contributor Covenant Code of Conduct. The fastest way to get involved:

good first issue

git clone https://github.com/KryptosAI/mcp-observatory.git && cd mcp-observatory && npm install && npm test

The most common first contribution is adding an MCP server to the Safety Index (10-15 minutes). See CONTRIBUTING.md for full guidelines, code standards, and the contributor recognition ladder.


If Observatory saved you a broken deploy, consider giving it a star. It helps others find the project.

Available Tools

13 tools
check_serverA

Use this to test a specific MCP server before installing or after updating it. Launches the server by command, checks all capabilities, and saves a run artifact for future comparison. Example: check_server({ command: 'npx -y @modelcontextprotocol/server-everything' }). Use deep=true to invoke tools, security=true to analyze schemas for vulnerabilities.

ParametersJSON Schema
NameRequiredDescriptionDefault
argsNoAdditional arguments for the command.
deepNoAlso invoke safe tools to verify they execute.
commandYesThe command to launch the MCP server (e.g. 'npx -y @modelcontextprotocol/server-everything').
securityNoRun security analysis on tool schemas.

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Describes key behaviors: launches server, checks capabilities, saves artifact. Explains deep and security parameters' effects. No annotations, so description carries burden; it is thorough but could mention if there are any side effects.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences plus an example; no fluff. Purpose front-loaded, each sentence adds information. Highly efficient.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a testing tool with 4 parameters and no output schema, description explains all necessary usage, example, and parameter purposes. Complete for its complexity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, providing baseline. Description adds value by explaining deep and security flags beyond schema descriptions, and gives a concrete example.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool tests a specific MCP server before installation or after update, with actions: launch, check capabilities, save artifact. It distinguishes from siblings by focusing on testing rather than scanning or scoring.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly says when to use (pre-install, post-update), providing clear context. Lacks explicit when-not-to-use or alternatives, but the purpose is well-defined.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

ci_reportB

Generate a CI regression report from run artifacts.

ParametersJSON Schema
NameRequiredDescriptionDefault
artifactsDirNoDirectory containing run artifacts. Defaults to .mcp-observatory/runs/

TDQS

B3.2/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden but only states it generates a report, implying a read operation. It does not disclose side effects, required permissions, or output format, leaving significant ambiguity.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, focused sentence of 9 words, perfectly front-loaded and efficient with no extraneous content.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the simple schema (1 optional parameter) and no output schema, the description lacks specifics about the report's nature, contents, or return value, leaving the agent insufficiently informed for correct invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so baseline is 3. The description adds no new meaning beyond the schema's parameter details, merely echoing 'from run artifacts', which is already apparent.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description 'Generate a CI regression report from run artifacts' uses a specific verb (Generate) and resource (CI regression report from run artifacts), clearly distinguishing it from sibling tools like get_last_run or scan.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description offers no guidance on when to use this tool versus alternatives, such as diff_runs or check_server, nor does it specify prerequisites or context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

diff_runsA

Use this to find what changed between two server checks. Compares two run artifacts and surfaces regressions (things that broke), recoveries (things that got fixed), schema drift (added/removed/changed tool parameters), and gate status changes. Essential after updating a server.

ParametersJSON Schema
NameRequiredDescriptionDefault
baseYesPath to the base run artifact JSON file.
headYesPath to the head run artifact JSON file.
formatNoOutput format: 'markdown' (default) or 'json'.

TDQS

A3.6/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Describes what the tool surfaces (changes, regressions, etc.), but with no annotations, the description lacks information on side effects, permissions, or whether it reads/writes. The behavior is implied but not fully transparent.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with purpose, no fluff. Every sentence adds value.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Covers the main use case and output types, but lacks details on output format (though format parameter is in schema) and prerequisites like file existence. Given no output schema, it is fairly complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema covers all parameters with descriptions; the description adds context about the comparison but does not provide additional semantics beyond the schema for each parameter.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly states it finds changes between two run artifacts, listing specific types (regressions, recoveries, schema drift, gate status changes). However, it does not differentiate from sibling tools that may also compare runs or history.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides a specific use case 'Essential after updating a server', but does not include when to avoid using it or alternative tools for similar tasks.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_historyC

Get health score trends for MCP servers from run history.

ParametersJSON Schema
NameRequiredDescriptionDefault
targetNoFilter to a specific target ID.

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description should disclose behavioral traits. It states it 'gets' data (read operation) but does not mention potential impacts, auth requirements, or rate limits. The behavior is minimally transparent.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence of 10 words, highly concise and front-loaded with the core purpose. No unnecessary words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no output schema, the description should explain the return format (e.g., list of scores, timeline). It does not. Also missing details about required run history context. Incomplete for a tool with no annotations.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with a single parameter described. The description adds no additional meaning beyond the schema, so baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool retrieves health score trends from run history, using a specific verb and resource. However, it does not explicitly distinguish from sibling tools like get_last_run or diff_runs.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided on when to use this tool versus alternatives such as get_last_run (single result) or diff_runs (comparison). The description lacks context about appropriate use cases.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_last_runA

Return the last Observatory run for a server, including handshake allow/deny from the local receipt. No hosted account required.

ParametersJSON Schema
NameRequiredDescriptionDefault
targetIdYesThe target ID to find the last run for (e.g. server name or command).

TDQS

A3.6/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It states the operation returns data and adds context about handshake details and that no account is needed. However, it does not explicitly mention that this is a read-only operation, what happens if no run exists (e.g., returns null or errors), or any other behavioral nuances. It adds some value but lacks full transparency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences, front-loaded with the primary purpose, and includes only essential additional context (handshake detail and no-account requirement). There is no wasted wording; every clause earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool is simple (one parameter, no output schema), the description covers the main operation and a key prerequisite. It would be more complete with a note on error handling (e.g., what happens if no run is found) or a mention that it's read-only. However, for its complexity, it is nearly complete and clear enough for an agent.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% for the single parameter `targetId`, which already includes an explanation. The tool description adds minimal semantic value beyond the schema—it repeats that it's for a server, but does not elaborate on the format, required patterns, or how it is used in the operation. The description is adequate but not enriching.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Return the last Observatory run for a server'. It specifies the resource (Observatory run), the scope (last run for a server), and adds a distinctive detail (including handshake allow/deny). This differentiates it from siblings like get_history (which likely returns all runs) and check_server (which likely checks status).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no explicit guidance on when to use this tool versus alternatives. It mentions 'No hosted account required', which is a prerequisite/condition, but does not say 'use this to get the most recent run' or contrast it with siblings like get_history or scan. The usage context is only implicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

lock_verifyA

Verify that live MCP servers still match a previously saved lock file. Detects schema drift, added/removed tools, and breaking changes.

ParametersJSON Schema
NameRequiredDescriptionDefault
configNoPath to MCP config file.

TDQS

A3.5/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries full burden. It states what is detected but omits key behavioral traits: whether the tool is read-only, whether it modifies state, what triggers an error, or how the result is presented. For a verification tool, these gaps reduce transparency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

One concise sentence that conveys the core purpose without wasted words. Information density is high.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (verification against a lock file) and no output schema, the description should hint at the return format (e.g., boolean, diff report). It doesn't explain what a lock file is or how it is created, assuming context from sibling tools. Adequate but with gaps.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The single parameter 'config' has a description in the schema ('Path to MCP config file.'). The tool description does not add any additional information about the parameter, such as format, defaults, or how it relates to the lock file. With 100% schema coverage, baseline is 3.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly states the verb 'verify' and the resource 'live MCP servers match a previously saved lock file'. It specifies the exact purpose: detecting schema drift, added/removed tools, and breaking changes. This distinguishes it from sibling tools like 'verify' (generic) and 'check_server' (single server check).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Description implies usage context (when a lock file exists and drift detection is needed), but does not explicitly say when to use this tool vs alternatives. No exclusions or prerequisites are provided. With nine siblings, more guidance would be helpful.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

recordA

Use this to capture a baseline of a working MCP server. Records all JSON-RPC traffic to a cassette file that can be replayed offline (no server needed) or used to verify future versions haven't broken anything. Like VCR for MCP.

ParametersJSON Schema
NameRequiredDescriptionDefault
argsNoAdditional arguments for the command.
commandYesThe command to launch the MCP server.

TDQS

A4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries full burden. It explains the core behavior (records all JSON-RPC traffic to a cassette file) and mentions offline replay capability. However, it lacks details about side effects, required permissions, or the format of the cassette file.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise (three sentences), front-loaded with the purpose, and uses an analogy ('Like VCR for MCP') for clarity. Every sentence adds value.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity (2 parameters, no nested objects), the description is largely complete. However, it does not explicitly state the return value (e.g., the path to the cassette file), which would be helpful for an AI agent using the tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

All parameters are described in the input schema (100% coverage). The description does not add any additional semantic meaning beyond what the schema already provides, so a baseline score of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: capture a baseline of a working MCP server by recording JSON-RPC traffic to a cassette file. It uses a specific verb ('capture', 'records') and resource, and distinguishes from siblings like 'replay' and 'verify'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context for when to use the tool (to capture a baseline for offline replay or verification), but it does not explicitly state when not to use it or mention alternatives among siblings. The context is sufficient for an AI agent to decide.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

replayA

Use this to test a server without running it. Replays a previously recorded cassette offline and runs all checks against the recorded responses. Useful in CI or when the live server is unavailable.

ParametersJSON Schema
NameRequiredDescriptionDefault
cassetteYesPath to a cassette JSON file.

TDQS

A3.9/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description discloses that the tool runs offline and replays recorded responses, but does not specify side effects (e.g., no modifications) or output format. Since no annotations are provided, the description carries the full burden and could be more detailed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is three sentences, front-loaded with purpose, and every sentence adds value. No unnecessary words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description explains purpose, behavior, and use cases but does not mention the output format or return value, which is important since no output schema is provided. For a simple tool, it's adequate but incomplete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with a description for the 'cassette' parameter. The description adds no additional semantic value beyond the schema, so baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool replays a recorded cassette to test a server offline, which is a specific verb-resource combination. It distinguishes from sibling tools like 'check_server' which likely test live servers.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly mentions usefulness in CI or when the live server is unavailable, giving clear context for when to use. It lacks explicit when-not-to-use or alternative mentions, but the context is strong enough.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

scanA

Use this to check if all your MCP servers are healthy. Auto-discovers servers from Claude config files, connects to each one, and verifies tools/prompts/resources respond correctly. Use with deep=true to also invoke tools and confirm they actually execute. Returns pass/fail status for every server.

ParametersJSON Schema
NameRequiredDescriptionDefault
deepNoAlso invoke safe tools to verify they execute.
configNoPath to a specific MCP config file. If omitted, scans default locations.
securityNoRun security analysis on tool schemas.

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations present, so description fully covers behavior: auto-discovers servers from config files, connects, verifies responses, and in deep mode invokes tools. Discloses return is pass/fail status.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Concise paragraph with front-loaded purpose, followed by parameter guidance. Every sentence adds value; no wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Complete description for a health-check tool with 3 optional parameters and no output schema. Explains return format, auto-discovery, and verification steps.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema covers 100% of parameters with descriptions. Description adds usage context (e.g., deep=true invokes tools, config path specific, security analysis) beyond schema definitions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states it checks MCP server health with auto-discovery and verification of tools/prompts/resources. Distinguishes from siblings like check_server (single server) and verify (different scope) by describing the comprehensive health check.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly says 'Use this to check if all your MCP servers are healthy' and suggests when to use deep=true. Does not mention when not to use or alternatives, but provides sufficient context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

score_serverB

Use this to get a quick health grade for an MCP server. Runs all checks (capabilities, tool invocation, security) and returns a 0-100 score with A-F grade and detailed breakdown across protocol compliance, schema quality, security, reliability, and performance.

ParametersJSON Schema
NameRequiredDescriptionDefault
argsNoAdditional arguments for the command.
commandYesThe command to launch the MCP server.

TDQS

B3.4/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are present, so the description carries full burden. It states 'runs all checks' but does not disclose potential side effects (e.g., launching a server or modifying state), whether it requires permissions, or what happens on failure. This is insufficient transparency for a tool that executes commands.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences long, front-loaded with the primary use case, and efficiently enumerates the checks and output. No unnecessary words; each sentence adds value.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool has no output schema, the description adequately explains the return value (0-100 score, A-F grade, breakdown across multiple dimensions). It lists the areas checked but could mention prerequisites (e.g., server must be running) or result interpretation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3. The description adds no additional meaning beyond the schema's parameter descriptions; it merely repeats the concept of 'command' and 'args' without deepening understanding.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description explicitly states the action ('get a quick health grade') and the target ('an MCP server'), listing the checks performed and output format. It is distinct from sibling tools like 'scan' or 'check_server' but does not explicitly differentiate itself.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description mentions 'Use this to get a quick health grade', implying usage context. However, it does not provide when-not-to-use guidance or compare with alternatives such as 'verify' or 'check_server' for more detailed checks.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

suggest_serversA

Use this when setting up a project or wondering what MCP servers to add. Scans the working directory for languages, frameworks, databases, and cloud providers, lists currently configured servers, and cross-references the MCP registry to recommend servers you're missing.

ParametersJSON Schema
NameRequiredDescriptionDefault
cwdNoWorking directory to scan for environment signals. Defaults to process.cwd().

TDQS

A4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so description carries full burden. It describes scanning, listing, and cross-referencing behavior but does not disclose potential side effects, network usage, error handling, or whether it modifies any files. Adequate but not detailed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, no wasted words. First sentence gives usage context, second explains the process. Front-loaded and efficient.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (scanning, listing, cross-referencing) and no output schema, the description explains input and general behavior but does not specify the output format or what exactly 'recommend servers you're missing' means. Slightly incomplete but still good for a suggestion tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Only parameter 'cwd' has schema description 'Working directory to scan for environment signals. Defaults to process.cwd().' Description adds context about the scanning purpose, but the schema already covers the meaning. With 100% schema coverage, baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool scans working directory for languages, frameworks, databases, and cloud providers, lists configured servers, and cross-references the MCP registry to recommend missing servers. It sets it apart from siblings like 'scan' or 'check_server' by focusing on recommendations for project setup.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly says 'Use this when setting up a project or wondering what MCP servers to add.' Provides clear context for when to use, but does not mention when to use alternative sibling tools like 'scan' or 'check_server' instead.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

verifyA

Use this after updating a server to confirm nothing broke. Connects to the live server, sends the same requests from a recorded cassette, and compares responses. Reports exactly what changed — added tools, removed parameters, different response shapes.

ParametersJSON Schema
NameRequiredDescriptionDefault
argsNoAdditional arguments for the command.
commandYesThe command to launch the MCP server.
cassetteYesPath to a cassette JSON file.

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries full burden. It explains key behaviors: connects to live server, sends requests from a recorded cassette, compares responses, and reports changes. It lacks details on potential side effects (e.g., read-only nature) or prerequisites, but covers core actions well.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise at two sentences, with no redundant information. Every sentence adds value, clearly stating purpose, method, and output.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no output schema, the description partially explains the output ('reports exactly what changed'). However, it doesn't specify the format or structure of the report, which could be critical for programmatic use. Otherwise, context is sufficient for a verification tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so baseline is 3. The description adds no new information beyond the schema parameter descriptions (e.g., 'path to cassette', 'command', 'args'). It does not enhance semantic understanding.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: verifying a server after an update by comparing responses against a recorded cassette. It specifies the verb ('confirm nothing broke') and resource ('server'), and effectively distinguishes from sibling tools like 'record' and 'replay'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit context for when to use ('after updating a server'), which is helpful. However, it does not mention when not to use it or name alternative tools (e.g., 'replay' vs 'verify'), leaving some ambiguity.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

watchA

Use this to check a server and see what changed since the last check. Runs all checks, saves the result, and diffs against the previous run for the same target. Shows regressions, recoveries, and schema drift in one call.

ParametersJSON Schema
NameRequiredDescriptionDefault
argsNoAdditional arguments for the command.
commandYesThe command to launch the MCP server.

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden. It discloses that the tool runs all checks, saves results, diffs against previous runs, and shows regressions, recoveries, and schema drift. It does not mention side effects or auth needs, but the behavioral description is comprehensive for the tool's purpose.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three sentences with no wasted words. The purpose is front-loaded, and every sentence adds value. Excellent conciseness.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite no output schema, the description explains what the tool shows (regressions, recoveries, schema drift). It clarifies that a previous run is needed by mentioning 'since the last check' and 'previous run for the same target.' Complete for its complexity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the schema already describes both parameters (command and args). The description adds no additional parameter meaning, so baseline score of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states it checks a server and shows changes since last check, specifying verb, resource, and scope. It distinguishes from siblings like 'scan' and 'diff_runs' by noting it runs all checks, saves results, and diffs against previous runs in one call.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description says 'Use this to check a server and see what changed since the last check,' providing clear context. It implies a usage scenario but does not explicitly mention when not to use or provide alternatives. Still, the guidance is strong.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. 3 tool updatesv1.35.10
    • Addedci_report
    • Addedget_history
    • Addedrecord
  2. 8 tool updatesv1.35.4
    • Addedcheck_server
    • Addedget_last_run
    • Addedlock_verify
    • Addedreplay
    • Addedscan
    • Addedscore_server
    • Addedsuggest_servers
    • Addedwatch
  3. 11 tool updatesv1.33.0
    • Removedcheck_server
    • Removedci_report
    • Removedget_history
    • Removedget_last_run
    • Removedlock_verify
    • Removedrecord
    • Removedreplay
    • Removedscan
    • Removedscore_server
    • Removedsuggest_servers
    • Removedwatch

TDQS

A3.7/5.0
Disambiguation4/5

Most tools have distinct purposes (e.g., scan auto-discovers, check_server tests a specific server, score_server returns a grade), but some comparison tools like diff_runs, verify, watch, and lock_verify could be confused due to overlapping functions.

Naming Consistency3/5

Most tools follow a verb_noun pattern (get_last_run, check_server, diff_runs), but some are single verbs (scan, verify, watch) and ci_report is noun_noun, causing minor inconsistency.

Tool Count5/5

13 tools cover the domain of MCP server monitoring, checking, recording, and verification without being excessive. Each tool earns its place.

Completeness5/5

The tool surface is complete: health checks, history, comparison, recording/replaying, suggestions, and CI reporting. No obvious missing functionality for the stated purpose.

Maintenance

ActivityActive
ResponsivenessWithin a week

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/KryptosAI/mcp-observatory'

If you have feedback or need assistance with the MCP directory API, please join our Discord server