Skip to main content
Glama

AI Workbench

License

AI Workbench supervises AI coding agents, captures evidence, validates work, applies acceptance policy, and produces auditable PR-ready reports.

The PyPI package remains ai-workbench-mcp for this public alpha because the ai-workbench package name is already occupied. The product and CLI are AI Workbench:

pip install ai-workbench-mcp
ai-workbench --help

Current source metadata targets unpublished ai-workbench-mcp==0.8.0a0. This public alpha consolidates local supervision, evidence capture, validation, acceptance policy, and PR reporting into one product surface.

Public Alpha Warning

The supervisor is the preferred automated evidence path, but daemon, Codex hook, and OpenCode adapter coverage are alpha mechanisms. AI Workbench checks evidence quality and acceptance readiness; it does not prove the work is absolutely correct. High-risk work still requires human review.

Related MCP server: corbat

Architecture

  • AI Workbench supervisor captures local evidence.

  • AI Workbench validation writes validation_report.json.

  • AI Workbench quality gate writes revision_decision.json.

  • AI Workbench PR/report surfaces render accept, needs_review, or block.

Agent output is a proposal. Workbench accepts evidence.

MCP is the connection protocol. AI Workbench MCP is the tool server. Acceptance is decided by the selected validation profile and quality gate. The agent performs. Workbench accepts. MCP connects them.

Quick Start

Register a project once and start the local supervisor:

pip install ai-workbench-mcp
ai-workbench supervisor setup --project-dir . --task-type code_change
ai-workbench supervisor start

Run Codex, OpenCode, Goose, or another supported local workflow in the project. Then inspect the latest report:

ai-workbench supervisor status
ai-workbench reports show latest --project-dir .

Render PR-ready artifacts from a finalized run:

ai-workbench pr-gate --run-dir runs/<run_id>

The canonical local run ledger is:

runs/<run_id>/
  task_metadata.json
  final_prompt.md
  model_selection.json
  model_output.md
  validation_report.json
  revision_decision.json
  run_log.jsonl
  metadata.json
  transcript.jsonl
  commands.jsonl
  workspace/
  validation/
  artifacts/

validation_report.json and revision_decision.json are the final acceptance authority. Supporting supervisor reports are local evidence, not a substitute for those Workbench artifacts.

Codex Hooks

Install project-local Codex hooks:

ai-workbench setup codex --project-dir . --task-type code_change

Restart Codex or start a new session, open /hooks, review the project hook, and trust it once. Until a hook event is observed, supervisor status reports Codex coverage as configured but unverified.

Goose MCP

AI Workbench still exposes the same MCP tool lifecycle. Register the server with Goose or another MCP host using:

ai-workbench mcp serve

The seven MCP tools remain:

workbench_open_run
workbench_select_policy_pack
workbench_select_model
workbench_record_execution
workbench_validate_run
workbench_quality_gate
workbench_analyze_runs

PR Gate

Workbench PR acceptance consumes real Workbench run evidence:

ai-workbench pr-gate \
  --run-dir runs/<run_id> \
  --out runs/pr_gate/pr_comment.md \
  --json-out runs/pr_gate/pr_decision.json

Outcomes are exactly:

  • accept

  • needs_review

  • block

Missing, unreadable, or scaffold-only evidence blocks. A green CI run, uploaded artifact, sticky PR comment, or model self-claim is not acceptance evidence.

Bootstrap Assets

To add starter configs, prompts, recipes, docs, and the GitHub PR-gate workflow to a repository:

ai-workbench bootstrap --target .

The bootstrap keeps runs/ ignored.

Package Demo

For a package-only synthetic demo:

ai-workbench demo --target ./workbench-first-run

This shows accept, needs_review, and block PR-gate outcomes with fixture evidence. It is not a real target-repository acceptance run.

Development

python -m pip install -e ".[dev,publish]"
python -m pytest -q -p no:cacheprovider
python -m ruff check . --no-cache
python -m mypy --no-sqlite-cache --no-incremental
ai-workbench demo --target runs/package_demo_smoke
ai-workbench validate --project ai_workbench_mcp --profile scaffold --run-dir runs/scaffold-smoke

Do not commit runs/. Committed sample evidence must be sanitized and live under examples/.

Docs

Recipes:

Sample evidence:

License

Apache-2.0. See LICENSE. MIT-origin attribution for the consolidated Prove It code is retained in NOTICE.

Available Tools

7 tools
workbench_analyze_runsB

Analyze local Workbench run ledgers and write report artifacts.

ParametersJSON Schema
NameRequiredDescriptionDefault
runs_dirNoruns
task_typeNo
sinceNo
out_dirNo
evals_dirNoevals/golden_cases
evidence_scopeNoall

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

B3.2/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries full burden. It states 'analyze' and 'write report artifacts' but does not disclose potential side effects (e.g., file creation), required permissions, or if it is read-only. The lack of detail leaves the agent uncertain about behavioral traits.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence that states the core action efficiently. It is not verbose, but brevity comes at the cost of missing critical details. It could be expanded slightly without losing conciseness.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has 6 parameters, no schema descriptions, and no annotations. The description does not explain parameters, output, or prerequisites. Although an output schema exists, the description does not leverage it to clarify return values. This is insufficient for an agent to use the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0% (no parameter descriptions in schema). The description provides no parameter-specific information, failing to clarify the roles of 'runs_dir', 'task_type', 'since', etc. Baseline would be 1-2, and the description does not compensate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description specifies the verb 'analyze' and 'write' with clear resources: 'local Workbench run ledgers' and 'report artifacts'. It distinguishes the tool from siblings like workbench_open_run and workbench_validate_run by focusing on analysis and artifact generation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No explicit guidance on when to use this tool versus alternatives. The purpose is inferred from the description and sibling names, but there is no direct comparison or context for selection.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

workbench_open_runC

Create a Workbench run folder and initial evidence artifacts.

ParametersJSON Schema
NameRequiredDescriptionDefault
projectYes
taskYes
run_dirNo
promptNoimplement_request_change_request
riskNomedium
context_profileNo
recipeNo
changed_filesNo
docsNo
include_diffNo
execution_hostNogoose
auto_select_policy_packNo
policy_packNo
validation_profileNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

C2.3/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries full burden. It only states 'Create,' which implies mutation but does not disclose side effects, permissions, or safety implications (e.g., idempotency, data overwrite).

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness2/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise (one sentence) but lacks structure. It does not front-load critical information, and the brevity sacrifices clarity for a tool with 14 parameters.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness1/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the complexity (14 parameters, no annotations, output schema exists but unmentioned), the description is grossly incomplete. It fails to explain return values, parameter usage, or integration with sibling tools.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% across 14 parameters, and the description adds no parameter-specific guidance. Baseline expectation is high, but the tool's one-line description does not explain any parameter purpose, defaults, or relationships.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's action ('Create') and resource ('Workbench run folder and initial evidence artifacts'). It distinguishes from sibling tools like workbench_analyze_runs or workbench_validate_run, which focus on analysis/validation rather than creation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool vs. alternatives. It does not specify prerequisites, scenarios, or exclusion cases, leaving the agent to infer context from the name alone.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

workbench_quality_gateD

Run the Workbench quality gate for a run directory.

ParametersJSON Schema
NameRequiredDescriptionDefault
projectYes
run_dirYes
modeNoauto
riskNo
validation_reportNo
review_promptNo
review_outputNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

D1.9/5.0
Behavior1/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations exist, and the description fails to disclose any behavioral traits such as side effects, permissions needed, or potential impacts. For a tool that likely performs a mutation (running a quality gate), this is a critical gap.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness2/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence with no waste, but it is overly terse, lacking essential information. Conciseness is achieved at the expense of utility.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness1/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given 7 parameters, no annotations, and no parameter documentation, the description is vastly incomplete. Even with an output schema, the description fails to explain purpose of parameters or expected behavior.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, and the description adds no meaning to the 7 parameters beyond their names. Parameters like 'mode', 'risk', 'validation_report' are left entirely unexplained.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly states the verb 'Run' and the resource 'Workbench quality gate for a run directory,' making the function specific. It distinguishes from siblings like workbench_analyze_runs and workbench_validate_run, though additional context on what the quality gate does would differentiate it further.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines1/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool versus alternatives like workbench_validate_run or workbench_analyze_runs. No exclusions or context provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

workbench_record_executionC

Capture Goose/model response text into Workbench evidence artifacts.

ParametersJSON Schema
NameRequiredDescriptionDefault
projectYes
run_dirYes
response_textYes
files_touchedNo
model_output_statusNoresponse_captured
run_statusNoin_progress
response_sourceNogoose
validationNo
follow_upNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

C2.4/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description must detail behavioral traits. It only mentions 'capture,' implying a write operation, but fails to disclose if data is appended/overwritten, required permissions, or side effects.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence, which is concise, but it lacks structure and key details. It is front-loaded but too sparse to be effective.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool has 9 parameters and no annotations, the description is far too brief. It provides only a high-level purpose without covering critical context for correct invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, and the description adds no information about any of the 9 parameters. It does not explain what 'project,' 'run_dir,' or 'response_text' mean, leaving the agent without guidance.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool captures 'Goose/model response text into Workbench evidence artifacts,' using a specific verb and resource. It distinguishes from sibling tools like workbench_analyze_runs, though it could be more explicit about its unique role.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use or avoid this tool, or how it differs from siblings like workbench_validate_run. A single sentence does not provide usage context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

workbench_select_modelC

Select a Workbench model tier and write model_selection.json.

ParametersJSON Schema
NameRequiredDescriptionDefault
projectYes
task_typeYes
riskYes
outYes
validation_strengthNomedium
promptNo
complexity_scoreNo
test_complexity_levelNo
instruction_followingNonormal
task_textNo
code_filesNo
recipeNo
validation_profileNo
routing_feedback_pathNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

C2.4/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided; description only states it selects a model tier and writes a file. No disclosure of side effects (e.g., overwriting files), permissions, or read/write characteristics. Minimal behavioral insight.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is very concise (one sentence), but this conciseness comes at the expense of necessary detail. It is not overly long, but it is incomplete.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness1/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 14 parameters and no schema descriptions, the description is severely incomplete. It does not explain how to use the tool, what the output file contains, or the role of any parameter. Output schema exists but is not leveraged in the description.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema has 14 parameters with 0% description coverage. Description does not mention any parameters or explain how project, task_type, risk, etc., influence model selection. No semantic meaning added beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states the verb 'Select' and the resource 'Workbench model tier', and specifies output 'write model_selection.json'. It distinguishes from siblings like workbench_select_policy_pack, but lacks detail on inputs and selection criteria.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool versus siblings or alternatives. No mention of prerequisites, scenarios, or exclusions. The description is silent on usage context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

workbench_select_policy_packB

Recommend an advisory Workbench policy pack from task metadata.

ParametersJSON Schema
NameRequiredDescriptionDefault
task_textNo
task_typeNo
changed_filesNo
promptNo
riskNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

B3.4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations present, so description must disclose behavior. It indicates an advisory (likely read-only) function, but does not explicitly state side effects, prerequisites, or whether it modifies state. Adequate but not detailed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence, concise and front-loaded. However, it is overly terse, sacrificing necessary detail for brevity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

While an output schema exists (not shown), the description does not explain what the tool returns or how it utilizes the input parameters. Five optional parameters with no guidance make it incomplete for proper use.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 5 parameters and 0% schema description coverage, the description must add meaning. However, it only mentions 'task metadata' without explaining any of the five parameters (task_text, task_type, etc.). The agent has to infer purpose from parameter names alone.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's action ('Recommend'), resource ('advisory Workbench policy pack'), and source ('from task metadata'). This distinguishes it from siblings like workbench_select_model, which deals with model selection.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No explicit guidance on when to use this tool versus alternatives. The description implies use for policy pack recommendation, but lacks context on when not to use it or how it differs from siblings like workbench_analyze_runs.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

workbench_validate_runC

Run deterministic Workbench validation over a run directory.

ParametersJSON Schema
NameRequiredDescriptionDefault
projectYes
out_dirYes
profileNo
changed_filesNo
task_test_commandNo
report_nameNovalidation_report.json

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

C2.2/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided; description only mentions 'deterministic' but does not clarify side effects, permissions, or whether the tool is read-only. Inadequate behavioral context.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Single sentence, no redundancy, but excessive brevity sacrifices necessary detail. Efficiency is not a virtue when critical information is omitted.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With 6 parameters, zero schema descriptions, no annotations, and an output schema not detailed, the description fails to provide sufficient context for correct tool invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema has 6 parameters with 0% description coverage. The tool description provides no explanation of any parameter, leaving the agent uninformed about how to fill them.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose3/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states the action (Run) and resource (Workbench validation over a run directory) but does not differentiate from sibling tools like workbench_analyze_runs or workbench_quality_gate.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool versus alternatives. Lacks context for appropriate usage.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. 7 tool updatesv0.1.0
    • First observedworkbench_analyze_runs
    • First observedworkbench_open_run
    • First observedworkbench_quality_gate
    • First observedworkbench_record_execution
    • First observedworkbench_select_model
    • First observedworkbench_select_policy_pack
    • First observedworkbench_validate_run

TDQS

C2.9/5.0
Disambiguation5/5

Each tool has a clearly distinct purpose: analyzing runs, opening runs, quality gating, recording execution, selecting model, selecting policy pack, and validating runs. No overlap in functionality.

Naming Consistency4/5

All tools share the 'workbench_' prefix and mostly follow a verb_noun pattern (e.g., analyze_runs, open_run). Minor deviation with 'quality_gate' which is noun_verb, but overall consistent.

Tool Count5/5

With 7 tools, the set is well-scoped for an AI workbench run management domain. Not too few to lack utility, not too many to be unwieldy.

Completeness3/5

The tools cover core actions like opening, analyzing, recording, and validating runs, but lack lifecycle management tools such as listing, closing, or deleting runs. Some gaps exist for a complete workflow.

Maintenance

ActivityStale
ResponsivenessWithin a week

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/hrishikesh-thakre/ai-workbench-mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server