Skip to main content
Glama

Game Agent Harness

Game Agent Harness is a deterministic test runner for Godot games. It launches the demo in a separate process, calls a narrow domain API, validates JSON responses, simulates input, captures screenshots, and writes diagnostic artifacts.

Licensed under the Apache License 2.0.

The repository contains an original 8x6 tactics micro-game. It uses primitive Godot drawing only; no reference-project code or artwork is included.

Requirements

  • Godot 4.6.3

  • Python 3.11 or newer

  • macOS, Linux, or Windows with a display available for screenshot and UI tests

The Python client and vendored Godot addon are both pinned to godot-e2e==1.2.0. The stdio MCP facade uses the official mcp==2.0.0 Python SDK.

Related MCP server: godot-mcp-rts

Install

python3 -m venv .venv
.venv/bin/pip install -e '.[dev]'

Open examples/tactics_demo/project.godot in Godot to play the demo manually. The automation server remains dormant in a normal game session.

Run scenarios

.venv/bin/python -m game_agent_harness run scenarios/tactics_smoke.yaml
.venv/bin/python -m game_agent_harness run scenarios/tactics_full.yaml
.venv/bin/python -m game_agent_harness run scenarios/tactics_ui.yaml
.venv/bin/python -m game_agent_harness soak scenarios/tactics_smoke.yaml --runs 100

Exit code 0 means pass, 1 means a game assertion failed, and 2 means a configuration, process, transport, or protocol error.

Every run creates artifacts/<run-id>/ with the resolved scenario, commands, events, states, screenshots, captured Godot logs, and reports in JSON and Markdown. Give report.md and its adjacent JSONL files to Codex when diagnosing a failed run.

MCP server

Start the local stdio server with an absolute workspace path:

.venv/bin/game-agent-harness-mcp \
  --workspace-root /absolute/path/to/game-agent-codex

An MCP host can launch it with this configuration:

{
  "mcpServers": {
    "game-agent-harness": {
      "command": "/absolute/path/to/game-agent-codex/.venv/bin/game-agent-harness-mcp",
      "args": [
        "--workspace-root",
        "/absolute/path/to/game-agent-codex"
      ]
    }
  }
}

The server publishes four tools:

  • list_scenarios lists YAML files under scenarios/.

  • run_scenario runs one listed scenario.

  • soak_scenario repeats a listed scenario, with a hard limit of 100 runs.

  • read_report reads report.json from one direct child of artifacts/.

The MCP layer uses the same ScenarioRunner as the CLI. It does not expose arbitrary filesystem paths, Godot node calls, property writes, or a code-editing tool. Scenario projects and generated artifacts must stay inside the configured workspace.

Game Agent API

/root/GameAgent exposes six methods during an --e2e session:

agent_get_manifest()
agent_reset(options)
agent_get_state()
agent_list_actions()
agent_perform_action(request)
agent_get_events(after_sequence)

The 0.1.0 contract is defined by the schemas under protocol/. An action request carries a unique request_id, the caller's expected_revision, an action name, and its arguments. A stale revision or invalid action is rejected without changing game state.

To add an action:

  1. Implement validation and behavior in TacticsGameController.perform_action.

  2. Add its argument schema to GameAgent.agent_get_manifest.

  3. Return a stable error code without incrementing revision on failure.

  4. Add a domain test and a scenario step.

The CLI deliberately does not expose arbitrary Godot node calls.

Verify

scripts/verify.sh

The script runs headless Godot domain tests, Python unit tests, Ruff, mypy, and the smoke scenario. It uses .venv/bin/python when available, otherwise python3. Set GAME_AGENT_PYTHON=/path/to/python to select another installed environment. The package is loaded directly from orchestrator/, but its third-party dependencies must be installed first.

VisualProbe

VisualProbe V1 is an optional, versioned contract for deterministic 3D inspection. It is isolated in examples/visual_probe_fixture, a Forward+ fixture that does not affect the Compatibility renderer used by tactics_demo. gameplay is deliberately outside V1: supported views are only front, back, left, right, and top; supported passes are beauty, silhouette, depth, normal, object_id, and wireframe.

The manifest is the only public capability registry. capabilities, when present, remains a unique list of strings; it must not contain a VisualProbe object. A V1 game publishes this shape (targets and variants are game-owned):

{
  "protocol_version": "0.1.0",
  "capabilities": ["domain_action"],
  "visual_probe": {
    "version": 1,
    "targets": {
      "asset_name": {
        "views": ["front", "back", "left", "right", "top"],
        "render_passes": ["beauty", "silhouette"],
        "max_resolution": {"width": 1280, "height": 720},
        "variants": {
          "height_scale": {"type": "number", "default": 1.0, "minimum": 0.5, "maximum": 2.0}
        }
      }
    }
  }
}

/root/GameAgent exposes these V1 methods. The JSON Schemas in protocol/visual-probe.schema.json are normative.

  • agent_get_manifest() returns the normal manifest above.

  • agent_prepare_visual_capture(request) accepts {target, views, render_passes, resolution: {width, height}, projection: "orthographic"}. Views and passes are unique, non-empty manifest values, at most 5×6 pairs; dimensions are positive and at most 1280×720. It returns {ok: true, count}.

  • agent_read_visual_capture() returns exactly one pending image per call: {ok: true, images: [{key, view, render_pass, width, height, png_base64}], renderer}. renderer contains projection, source, godot_version, and platform. The harness waits frames between reads. This two-phase API is the V1 capture API; agent_capture_visual is not part of V1 and is rejected by the VisualProbe transport.

  • agent_inspect_geometry({target}) returns {ok: true, geometry_snapshot}.

  • agent_apply_visual_variant({target, values}) applies one registered, type- and range-checked variant and returns {ok: true, token}. Only one variant can be active.

  • agent_restore_visual_variant({token}) restores it and returns {ok: true}.

Stable VisualProbe errors are visual_probe_unavailable, unknown_visual_target, unsupported_view, unsupported_render_pass, invalid_visual_variant, invalid_visual_request, visual_capture_failed, and visual_capture_not_ready. Errors have shape {ok: false, error: {code, message}}; no method accepts Godot node paths, method names, or host save paths.

Install the optional image stack and use the dedicated verifier on a GPU runner:

.venv/bin/pip install -e '.[dev,visual]'
scripts/verify-visual.sh

Visual runs store PNGs beneath visual/targets/<target>/; variant captures use visual/targets/<target>/variants/<id>/. A baseline package is exactly scenarios/baselines/<id>/metadata.json plus images/<target>/<view>-<pass>.png. metadata.json has visual_probe_version, godot_version, renderer, platform, width, height, and an images map keyed by <target>:<view>:<pass>. Renderer and dimensions must match before comparison; the V1 harness compares silhouette IoU, occupancy, centre/size deltas, and contour distance. PNGs and baseline files are bounded regular files read without following symlinks.

Each visual scenario has a fresh Godot process. A capture failure closes that process; on client disconnect or wait timeout the loopback server clears pending capture state and restores the active variant. The fixture transport listens only on 127.0.0.1, requires the non-empty random token generated by godot-e2e, fails closed after a 10-second handshake timeout, accepts frames up to 64 MiB, and permits only the methods listed above on /root/GameAgent. It rejects generic set_property, call_method, change_scene, batch, and all screenshots (including client-supplied save paths). A missing Godot 4.6 installation, GPU environment, or visual dependency is a configuration error (exit code 2), never a passing substitute.

Reference projects

The design was informed by:

No code or art was copied from these projects. See THIRD_PARTY_NOTICES.md for the vendored dependency.

Available Tools

9 tools
compare_visual_runsC

Return the VisualProbe reports for two confined artifact directories.

ParametersJSON Schema
NameRequiredDescriptionDefault
left_artifact_dirYes
right_artifact_dirYes

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

C2.7/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It only says 'Return', which implies a read operation, but it does not disclose whether the tool actually computes a comparison, whether it requires pre-existing artifacts, how it handles missing directories, or what 'confined' means operationally.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single front-loaded sentence with no filler or redundant phrasing. It is appropriately short, though its brevity contributes to the missing detail penalized in other dimensions.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no annotations and two required path-like parameters, the description does not explain how to choose the left and right directories, when to prefer this tool over sibling visual-report tools, or what a VisualProbe report represents. The output schema may cover return shape, but the invocation and selection context is incomplete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema provides no descriptions for left_artifact_dir or right_artifact_dir (0% coverage), and the description merely restates that they are two confined artifact directories. It does not explain path format, how to obtain such directories, or what makes them 'confined'. The parameter names are somewhat self-explanatory, but the description adds very little beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ('Return') and resource ('VisualProbe reports') and scopes it to 'two confined artifact directories'. The name 'compare_visual_runs' and the left/right parameter pair reinforce its purpose, but the description itself doesn't explicitly say it performs a comparison or how that differs from simply reading two reports separately.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no guidance on when to use this tool versus siblings like read_visual_report or read_visual_artifact. The description only implies it is for retrieving reports from two directories, but states no prerequisites, exclusions, or alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_scenariosA

List scenario files allowed within this workspace.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A3.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations present, the description carries the full behavioral disclosure burden. It does add one meaningful trait beyond the name — that only files 'allowed within this workspace' are returned, implying permission/workspace scoping. However, it says nothing about ordering, hidden files, or the read-only nature beyond what 'List' implies.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single seven-word sentence that front-loads the verb and communicates resource and scope with zero filler. Every word earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a zero-parameter list tool with an output schema already covering return values, the description is nearly sufficient: it identifies what is listed and the workspace constraint. It falls slightly short by not situating the tool in the workflow (e.g., that listing scenarios is a precursor to running or reading them).

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has zero parameters and the input schema is empty, so there is nothing for the description to explain about arguments. Per the zero-parameter baseline, a 4 is appropriate; the description correctly does not waste space on parameter details.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb ('List') and a specific resource ('scenario files') with a scope qualifier ('allowed within this workspace'). It is not a tautology and implicitly differentiates from the sibling list_visual_artifacts by resource type, though it never names that sibling explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage context is only implied: listing scenario files is naturally a discovery step before run_scenario or soak_scenario. The description offers no explicit when-to-use guidance, no exclusions, and names no alternatives among the eight sibling tools.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_visual_artifactsB

List safe PNG files directly inside an artifact directory's visual folder.

ParametersJSON Schema
NameRequiredDescriptionDefault
artifact_dirYes

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

B3.3/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the burden of behavioral disclosure. It states two meaningful constraints—only 'safe' PNGs and only files 'directly inside' the visual folder—but leaves 'safe' undefined and does not mention behavior for missing directories, symlinks, or permission failures.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, front-loaded sentence with no filler. Every word contributes: verb, resource, safety qualifier, and location are all present.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool is simple and has an output schema, so return values need no explanation. However, the description leaves 'safe' ambiguous and does not address edge cases, making it only moderately complete for an agent invoking it autonomously.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has 0% description coverage, so the description must compensate. It does relate artifact_dir to the 'artifact directory's visual folder', giving the parameter some role, but it does not clarify path format, expected directory layout, or whether artifact_dir is a path or identifier.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly identifies the action ('List') and the specific resource ('safe PNG files' in the artifact directory's visual folder). It distinguishes itself from read/run/compare siblings by focusing on listing, though it does not explicitly name or contrast any sibling.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives like list_scenarios or read_visual_artifact. It implies a simple listing use case but does not state prerequisites, exclusions, or when not to use it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

read_reportA

Read report.json from one direct child of the workspace artifacts directory.

ParametersJSON Schema
NameRequiredDescriptionDefault
artifact_dirYes

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the behavioral disclosure burden. It correctly indicates a read operation and restricts the scope to a single direct child, which is useful. However, it does not mention what happens when report.json is missing, whether the artifact_dir must be a directory, or how errors are surfaced.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence that fully conveys the tool's purpose and scope without any waste. It is front-loaded with the key action and resource, making it easy for an agent to parse quickly.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple one-parameter read tool with an output schema available, the description covers the essential scope and location. The main gap is lack of guidance about error cases or path conventions, but overall it is sufficiently complete for an agent to invoke the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema gives no description for the artifact_dir parameter and coverage is 0%, so the description must compensate. It does so by clarifying that artifact_dir refers to a direct child of the workspace artifacts directory and that the tool looks for report.json inside it. It stops short of specifying path format details, but for a single parameter this is sufficient.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb ('read'), a specific resource ('report.json'), and a precise location ('one direct child of the workspace artifacts directory'). This clearly distinguishes it from sibling tools like read_visual_report, which target a different artifact type.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies when to use this tool—when you need report.json from a direct child of the artifacts directory—but it does not explicitly mention alternatives or say when not to use it. The usage context is reasonably clear from the description alone, though no exclusions are stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

read_visual_artifactB

Read one safe PNG as an MCP image content block.

ParametersJSON Schema
NameRequiredDescriptionDefault
pathYes
artifact_dirYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
dataYes
typeNo
_metaNo
mimeTypeYes
annotationsNo

TDQS

B3.3/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full behavioral burden. It usefully discloses that the tool returns an MCP image content block and constrains input to 'safe PNG'. However, it does not explain what 'safe' means, how unsafe files are handled, or what errors or limitations may occur.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single concise, front-loaded sentence that states the action, the input type, and the output format. There is no wasted text or irrelevant detail.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Although an output schema exists, the description is incomplete for a tool with two required parameters at 0% schema coverage. Key contextual information is missing: how to obtain valid artifact paths, what 'artifact_dir' means, what 'safe' implies, and when to prefer this tool over sibling tools.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, and the description does not explain the roles of 'path' and 'artifact_dir'. The word 'PNG' hints that 'path' should point to a PNG file, but there is no clarification of how the two required parameters relate or what values are expected.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ('Read') and a specific resource ('one safe PNG'), and clearly states the output form ('MCP image content block'). It is easy to distinguish from siblings like list_visual_artifacts and read_visual_report because it targets a single image artifact rather than a listing or report.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no guidance about when to use this tool versus alternatives. It does not mention that list_visual_artifacts should probably be used to discover valid paths, nor does it differentiate itself from read_visual_report or compare_visual_runs. The usage context is only implied by the tool name and description.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

read_visual_reportB

Read VisualProbe JSON reports from one direct artifact directory.

ParametersJSON Schema
NameRequiredDescriptionDefault
artifact_dirYes

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

B3.3/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. The term 'direct artifact directory' hints at a non-recursive or single-level scope, which is useful, but the description does not clarify expected path format, errors, or whether multiple reports are read. As a read operation it is inherently non-destructive, but additional behavioral context would help.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single focused sentence with no filler. The verb and object are placed first, making the tool's purpose immediately understandable.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple one-parameter tool with an output schema, the description is mostly adequate. However, in the context of siblings like read_visual_artifact and compare_visual_runs, the description does not clarify the boundaries between these tools or what 'direct' means in practice, leaving some ambiguity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It does clarify that artifact_dir refers to a 'direct artifact directory' for VisualProbe JSON reports, which adds meaning beyond the bare parameter name. However, it does not specify whether the value should be a path, identifier, or relative/absolute reference.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly identifies the action (read), the resource (VisualProbe JSON reports), and the scope (one direct artifact directory). It does not explicitly differentiate from siblings like read_report or read_visual_artifact, but the specific 'VisualProbe JSON reports' phrasing provides reasonable distinction.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives no guidance on when to use this tool versus alternatives such as read_visual_artifact, read_report, or list_visual_artifacts. An agent would have to infer the intended use case from the name alone.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

run_scenarioA

Run one allowlisted scenario path relative to the scenarios directory.

ParametersJSON Schema
NameRequiredDescriptionDefault
scenarioYes

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A3.8/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden of behavioral disclosure. It only says 'Run' and constrains the path to allowlisted scenarios; it does not mention side effects, execution duration, whether a report is produced, or any permission requirements. For an execution tool, this is a meaningful gap.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single sentence with no filler. The core action and key constraint are front-loaded, and every word contributes useful information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple one-parameter tool with an output schema, the description is largely self-sufficient: it defines the purpose and parameter semantics. It is only incomplete regarding behavioral effects and explicit routing to alternatives, but those are secondary given the tool's simplicity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It does this well by explaining that the scenario parameter is a single allowlisted path relative to the scenarios directory. It could be stronger with examples or allowed-value hints, but it provides the essential meaning beyond the bare schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names the exact action ('Run'), the target resource ('one allowlisted scenario path'), and the location convention ('relative to the scenarios directory'). This clearly differentiates it from siblings like list_scenarios and soak_scenario.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for executing a single scenario, but it never explicitly states when to choose this over soak_scenario, run_visual_scenario, or other siblings. There are no alternative names or exclusion conditions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

run_visual_scenarioB

Run one allowlisted scenario containing VisualProbe steps.

ParametersJSON Schema
NameRequiredDescriptionDefault
scenarioYes

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

B3.4/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

There are no annotations, so the description carries the full burden of behavioral disclosure. It only states that the scenario is allowlisted and contains VisualProbe steps; it does not disclose side effects, whether test artifacts are produced, auth needs, failure behavior, or any consequences of running the scenario.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single front-loaded sentence with no filler. Every word adds information about scope or constraints.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no annotations and no schema description coverage, the tool needs more contextual support than this one-liner provides. The agent is left to infer valid scenario sources, execution effects, and how this relates to sibling tools like list_scenarios or read_visual_report.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, and the 'scenario' parameter has no schema-level documentation. The description adds some meaning by saying the value must be an allowlisted scenario containing VisualProbe steps, but it does not specify the expected format, identifier type, or how to discover valid scenario names.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb ('Run'), a resource ('scenario'), and two distinguishing constraints ('allowlisted' and 'containing VisualProbe steps'). This clearly differentiates it from siblings like run_scenario, soak_scenario, and the read/report tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies use for VisualProbe-based scenarios and hints at an allowlist prerequisite, but it never explicitly says when to choose this over run_scenario or soak_scenario, nor does it direct the agent to list_scenarios for valid inputs. Usage context is present but no exclusions or alternatives are stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

soak_scenarioB

Repeat an allowlisted scenario within the server's hard run limit.

ParametersJSON Schema
NameRequiredDescriptionDefault
runsYes
scenarioYes
minimum_pass_rateNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

B3.3/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the burden, and it does disclose two useful constraints: the scenario must be allowlisted and execution is bounded by the server's hard run limit. However, it does not explain what happens when runs exceed the limit, whether the operation has side effects on server state, or how results are returned.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, front-loaded sentence that wastes no words. Every phrase ('Repeat', 'allowlisted', 'hard run limit') contributes meaning.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with no annotations, low schema coverage, and no parameter descriptions, the description leaves important gaps: the semantics of minimum_pass_rate, the meaning of the server's run limit, and how to identify an allowlisted scenario are not covered. The output schema exists, so return format is not a missing piece, but the operational context is incomplete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% and the description does not compensate: it never mentions runs or minimum_pass_rate, and only indirectly relates to scenario. An agent cannot tell what values are valid for runs, what the default 0.99 threshold means, or how the parameters interact with the hard run limit.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb, 'Repeat', and identifies a resource, 'an allowlisted scenario', which makes the soak-testing intent clear. It is distinguishable from run_scenario by implying repeated execution, though it does not explicitly name an alternative.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The phrase 'Repeat an allowlisted scenario' implies this tool is for running a scenario multiple times rather than once, but it does not explicitly say when to choose this over run_scenario or list_scenarios. No exclusions or alternative conditions are provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. 9 tool updatesv0.2.0
    • First observedcompare_visual_runs
    • First observedlist_scenarios
    • First observedlist_visual_artifacts
    • First observedread_report
    • First observedread_visual_artifact
    • First observedread_visual_report
    • First observedrun_scenario
    • First observedrun_visual_scenario
    • First observedsoak_scenario

TDQS

A3.5/5.0
Disambiguation4/5

Most tools are clearly distinct, but run_scenario/run_visual_scenario and read_report/read_visual_report could be confused without knowing scenario types or report nesting. Descriptions clarify the visual/VisualProbe specializations enough for an agent to choose.

Naming Consistency5/5

All tools follow a consistent verb_noun snake_case pattern with an optional visual modifier, such as list_scenarios, run_visual_scenario, read_visual_artifact, and compare_visual_runs. There are no mixed conventions or inconsistent verb styles.

Tool Count5/5

9 tools is well within the ideal range for a focused harness: scenario discovery/execution, report access, and visual artifact handling are each represented without obvious bloat. The set is appropriately scoped for its purpose.

Completeness4/5

Core workflows are covered: list/run/soak scenarios, read standard reports, and run/read/list/compare visual scenarios and artifacts. Minor gaps include no direct way to list artifact/report directories or inspect a scenario for VisualProbe steps before running, but these can often be worked around by convention.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/TheBulaDev/game-agent-harness'

If you have feedback or need assistance with the MCP directory API, please join our Discord server