Integrates contract-driven test enforcement into GitHub Actions CI/CD pipelines.
Enforces test contracts on code generated by GitHub Copilot, providing validation, traceability, and reporting.
Integrates contract-driven test enforcement into GitLab CI/CD pipelines.
Provides hooks for contract-driven test enforcement as a pre-commit step.
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@io.github.phoenice-labs/universal-test-framework run all tests and generate a compliance report"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Contract-driven test enforcement and reporting for LLM-generated code — via MCP, VS Code, Copilot CLI, Claude, Cursor, or Python SDK.
UTF is an MCP server and contract enforcement engine. It does not replace your AI coding assistant — it gives every test your AI writes a mandatory quality gate, a structured audit trail, and a comprehensive compliance report.
AI tools — GitHub Copilot, Claude, Cursor, Gemini — generate tests fast. The problem: fast ≠ trustworthy.
Without UTF | With UTF |
Tests exist but nobody knows why | Every test is linked to a requirement |
"Covers everything" — nobody can prove it | Traceability matrix maps REQ → test |
CI passes; real behavior untested | Gap analysis flags what is NOT covered |
LLM wrote a test that asserts | Meaningfulness check rejects trivial assertions |
No history of what was tested | Persistent registry survives session restarts |
Report shows pass/fail counts only | 8-section per-test detail cards in HTML report |
You may already use markdown files (AGENTS.md, copilot-instructions.md, .cursorrules) to guide your AI. UTF is complementary — not competing:
Markdown instructions | UTF |
Describe how to write tests | Enforce a contract on every test produced |
Rely on LLM to follow instructions | Block tests that fail the contract at generation time |
No verification after generation | Score each test 0–1 against 8 measurable criteria |
No persistent state between sessions | SQLite registry persists all tests and results |
No compliance report | HTML report with per-test 8-section detail cards |
Use markdown instructions to shape how your AI thinks. Use UTF to verify and report on what it produced.
Related MCP server: TNL
13 MCP tools — generate, validate, register, analyze, trace, execute, mutate, report, and query tests
6 languages — Python, TypeScript, JavaScript, Java, Go, C++
7 test types — unit, integration, API, E2E, contract, performance, security
8-section contract — every test must pass all 8 sections or it is blocked (minimum score: 0.85)
3-segment test IDs — TC-{PRJ}-{MODULE}-{NNN} and classic TC-{MODULE}-{NNN} both accepted
Five install modes — uvx (zero-clone), local clone, Docker, GitHub MCP Registry, pip SDK
VS Code @utf — chat participant with slash commands
CI/CD ready — GitHub Actions, GitLab CI, pre-commit hooks
Per-project overrides — YAML rules in .utf/rules/ override global defaults
# | Method | How Started | Registry Location |
| Best For |
1 | uvx (zero-install) |
|
| ❌ use natural language | Any developer with uv |
2 | Local Clone |
|
| ✅ with extension | Contributing / customizing UTF |
3 | Docker / HTTP |
| Named volume or bind mount | ❌ use natural language | Team / remote / CI |
4 | GitHub MCP Registry | Auto via client |
| ❌ use natural language | Discoverable via MCP marketplace |
5 | pip + SDK | Python import | Caller controls cwd | N/A | CI scripts / programmatic |
All methods (1, 2, 4) using stdio transport write the registry to ${workspaceFolder}/.utf/utf.db — isolated per project and persistent across sessions.
Requires uv. No cloning, no venv.
Add to %APPDATA%\Code\User\mcp.json (Windows) or ~/.config/Code/User/mcp.json (macOS/Linux):
{
"servers": {
"utf": {
"type": "stdio",
"command": "uvx",
"args": ["--from", "universal-test-framework", "utf-server", "--transport", "stdio"],
"cwd": "${workspaceFolder}",
"env": {
"UTF_PROJECT_DIR": "${workspaceFolder}"
}
}
}
}Registry:
.utf/utf.dbinside${workspaceFolder}— isolated per project, persistent across sessions.
Reload VS Code after editingmcp.json.
git clone https://github.com/phoenice-labs/Universal-Test-Framework
cd Universal-Test-Framework
# Register MCP server, install @utf extension, copy global prompts
.\scripts\install-vscode-mcp.ps1
# Optionally scaffold a specific project
.\scripts\install-vscode-mcp.ps1 -InitProject -ProjectDir C:\my-projectReload VS Code → open Copilot Chat → ask naturally: generate e2e tests for my backend.
The install script writes this entry to mcp.json:
{
"utf": {
"type": "stdio",
"command": "<python>",
"args": ["-m", "mcp_server.server", "--transport", "stdio"],
"cwd": "${workspaceFolder}",
"env": { "PYTHONPATH": "<UTF_install_dir>" }
}
}Registry:
${workspaceFolder}/.utf/utf.db— isolated per project.@utfslash commands are available after the VSIX extension is installed.
# Start the UTF server
cd Universal-Test-Framework
docker compose -f docker/docker-compose.yml up -d
# MCP server available at http://localhost:8765/sseConnect from VS Code by adding to mcp.json:
{
"servers": {
"utf-remote": {
"type": "sse",
"url": "http://localhost:8765/sse"
}
}
}Registry: persisted in a named Docker volume (
utf_registry → /app/.utf/utf.db).
For per-project isolation with Docker, use a bind-mount indocker-compose.yml:volumes: - /path/to/your/project/.utf:/app/.utfBecause Docker uses HTTP/SSE transport (no
${workspaceFolder}templating), passproject_direxplicitly in tool calls, or setUTF_PROJECT_DIRin the container environment.
Per-project Docker workflow:
# docker-compose.override.yml
services:
utf-server:
volumes:
- ./my-project/.utf:/app/.utf
environment:
UTF_PROJECT_DIR: /appOnce published, UTF is discoverable via the MCP marketplace. Clients that support server.json install it automatically. The generated mcp.json entry is equivalent to Method 1 (uvx).
Registry isolation: The MCP registry schema does not support a
cwdfield at the registry level.
UTF resolves project isolation via (in priority order):
project_dirargument passed to each tool call
UTF_PROJECT_DIRenvironment variable
Path.cwd()fallback (server's working directory)For correct isolation, ensure the MCP client writes
"cwd": "${workspaceFolder}"and"UTF_PROJECT_DIR": "${workspaceFolder}"in the generated entry (UTF'smcp-gallery.jsondoes this).
pip install universal-test-frameworkfrom mcp_server.tools.generate_tests import generate_tests
from pathlib import Path
result = generate_tests(
test_type="unit",
source_code=open("src/auth.py").read(),
project_dir=str(Path.cwd()), # ← pass explicitly for correct registry isolation
)
print(result["suite_code"])Registry:
<project_dir>/.utf/utf.dbwhenproject_diris passed; falls back toPath.cwd().
Tool | Description |
| Generate a complete test suite satisfying the 8-section contract |
| Validate any test (generated or hand-written) against the contract |
| Identify coverage gaps in an existing test suite |
| Build a requirements → tests traceability matrix |
| Recommend test types with rationale from source code |
| Auto-detect programming language and test framework |
| Import JUnit XML from any test run into the UTF registry |
| Query the persistent per-project test registry |
| Execute test files and return structured CI results |
| Run mutation testing and return mutation score |
| Get gap analysis, coverage health, and trend report |
| Generate HTML / JUnit / JSON contract compliance report |
| Server health check: version, uptime, tool count |
Every test generated by UTF must satisfy all 8 sections. Tests that fail any hard section are blocked — returned in blocked_tests, never silently included.
# | Section | What It Must Contain | Weight |
1 |
| Unique ID: | 10% |
2 |
| Rationale tied to a requirement (≥50 chars) | 10% |
3 |
| At least one | 15% |
4 |
| GIVEN / WHEN / THEN with inputs, mocks, assertions (≥100 chars) | 20% |
5 |
| Coverage type + module + estimated % | 15% |
6 |
| Precise return values, status codes, state changes (≥50 chars) | 15% |
7 |
| Honest list of what this test does NOT cover (≥40 chars) | 10% |
8 |
| Self-assessment: meaningful / redundant / hallucinated (≥50 chars) | 5% |
Minimum passing score: 0.85. Tests below this threshold are blocked regardless of individual section presence. "none" in gaps or vague rationale like "to test the function" are rejected.
Language | Frameworks | Test Types |
Python | pytest | unit, integration, api, e2e, security, performance |
TypeScript | Jest, Vitest, Playwright | unit, integration, e2e, api |
JavaScript | Jest, Vitest | unit, integration |
Java | JUnit 5 + AssertJ, Maven | unit, integration, api |
Go | go-test + testify | unit, integration, api |
C++ | Google Test | unit |
This is the canonical flow for using UTF with any AI CLI (Copilot, Claude, Cursor) or VS Code. Follow the phases in order — skipping Phase 1 registration means the report has no Per-Test Contract Detail cards.
┌─────────────────────────────────────────────────────────────────────────┐
│ PHASE 1 — CONTRACT GENERATION (LLM writes, UTF validates + registers) │
│ │
│ ① generate_tests (scaffold) │
│ ② LLM writes real test methods — each with its own TC-ID and │
│ 8-section comment block (WHY / REQ / HOW / COV / OUT / GAP / MEAN)│
│ ③ validate_test_contract — score must be ≥ 0.85 per test │
│ ④ register_contracts — parse test file, upsert status=generated rows │
│ ⑤ generate_report — verify Per-Test Contract Detail cards appear │
│ │
│ PHASE 2 — EXECUTION (pytest/jest/maven runs, results captured) │
│ │
│ ⑥ pytest --junit-xml=utf-tests/reports/results.xml │
│ ⑦ import_test_results — upsert executed/failed rows │
│ ⑧ generate_report — now shows contract cards AND pass/fail status │
│ │
│ PHASE 3 — HEALTH (ongoing coverage quality) │
│ │
│ ⑨ feedback_status — gap analysis, drift alerts, trend over 30 days │
│ ⑩ Address gaps → add tests → back to Phase 1 │
└─────────────────────────────────────────────────────────────────────────┘Why register before running? The UTF registry has two record types.
status=generatedrecords (created byregister_contracts) drive the Per-Test Contract Detail cards in the HTML report.status=executed/failedrecords (created byimport_test_results) drive the Execution Results table. Both must exist for a test to appear in both sections. Running pytest before registering means you get execution rows but no 8-section detail cards.
Every test METHOD must have its own inline comment block — not a class docstring:
def test_health_returns_200(self, live_backend):
# ─── TC-FIQ-HLT-001 ──────────────────────────────────────────────────────
# WHY_GENERATED: The /health endpoint is the primary liveness signal for
# load balancers and K8s probes. Non-200 = platform unavailable.
# REQUIREMENT_MAPPING: REQ-E2E-001
# HOW_IT_EXERCISES: GIVEN backend is running at http://localhost:8001
# WHEN GET /health is called THEN HTTP 200 is returned.
# COVERAGE_CONTRIBUTION: Line coverage of health route; ~15% of health module
# EXPECTED_OUTCOME: HTTP 200; elapsed < 500ms
# GAPS_MISSING: Does not test health under load; no auth header tested
# MEANINGFULNESS_CHECK: Meaningful — gateway test for all other tests
# ─────────────────────────────────────────────────────────────────────────
r = requests.get(f"{live_backend}/health", timeout=5)
assert r.status_code == 200Test ID formats accepted: TC-HLT-001 (2-segment) or TC-FIQ-HLT-001 (3-segment project-prefixed).
UTF tools are called via natural language — no special syntax required:
# Phase 1 — Generate and register
generate e2e tests for backend/app/api/routes/
register contracts for utf-tests/test_myapp_e2e.py
utf report
# Phase 2 — After running pytest
import junit xml utf-tests/reports/results.xml
utf report
# Phase 3 — Health check
utf status
what are my coverage gaps?
build a traceability matrixClaude supports MCP servers natively. With UTF added to your MCP config:
# Natural language triggers UTF MCP tools automatically
"Generate e2e tests for my FastAPI backend at backend/app/"
"Register contracts for utf-tests/test_myapp_e2e.py"
"Generate the UTF report"
"Show UTF status and gaps"
"Validate this test against the 8-section contract: [paste test]"To add UTF to Claude's MCP config (~/.config/claude/mcp.json or equivalent):
{
"mcpServers": {
"utf": {
"command": "uvx",
"args": ["--from", "universal-test-framework", "utf-server", "--transport", "stdio"],
"env": { "UTF_PROJECT_DIR": "/path/to/your/project" }
}
}
}In Cursor, add UTF as an MCP server in .cursor/mcp.json (project-level) or ~/.cursor/mcp.json (global):
{
"mcpServers": {
"utf": {
"command": "uvx",
"args": ["--from", "universal-test-framework", "utf-server", "--transport", "stdio"],
"cwd": "${workspaceFolder}",
"env": { "UTF_PROJECT_DIR": "${workspaceFolder}" }
}
}
}Then ask Cursor naturally:
Generate unit tests for src/auth.py using UTF
UTF report
Register contracts for tests/test_api.pyThe MCP entry is identical regardless of client:
{
"utf": {
"type": "stdio",
"command": "uvx",
"args": ["--from", "universal-test-framework", "utf-server", "--transport", "stdio"],
"cwd": "${workspaceFolder}",
"env": { "UTF_PROJECT_DIR": "${workspaceFolder}" }
}
}UTF uses natural language detection — the same prompts work across all MCP-compatible AI clients.
UTF is invoked via natural language in the GitHub Copilot CLI — there are no special slash commands or @utf syntax at the CLI prompt. Simply describe what you want and the MCP tools are called automatically.
# In the GitHub Copilot CLI terminal (gh copilot / copilot-cli)
generate unit tests for backend/app/routes/auth.py
generate e2e tests covering REQ-001 through REQ-024
register contracts for utf-tests/test_myapp_e2e.py
utf report
show utf status
validate this test against the 8-section contract
what are the coverage gaps?
suggest test types for my project
build a traceability matrixWhat you say | UTF MCP tool invoked | What happens |
|
| Scans source, infers requirements, produces 8-section test suite |
|
| E2E suite with happy path + negatives + edge cases |
|
| API contract tests with HTTP assertions |
|
| Auth, injection, and boundary security tests |
|
| Parses test file, extracts per-method 8-section blocks, upserts generated rows |
|
| Scores test 0–1 against all 8 contract sections |
|
| Gap analysis, coverage health, trend report |
|
| HTML + JUnit + JSON contract compliance report |
|
| Register results from an existing pytest/Maven run |
|
| Identifies uncovered symbols and missing test paths |
|
| Requirements → tests coverage mapping |
|
| Recommends test types from source or requirements |
|
| Lists registered tests for the current project |
|
| Executes test files and records results in registry |
|
| Mutation score with killed/survived breakdown |
|
| Server uptime, version, tool count |
The UTF SQLite registry persists per-project across all sessions:
your-project/
└── .utf/
├── utf.db ← SQLite registry (persists between sessions)
├── utf-config.yaml ← Optional project overrides
├── rules/ ← Optional YAML rule overrides
└── reports/
├── contract_YYYYMMDD_HHMMSS.html
├── contract_YYYYMMDD_HHMMSS.xml
└── contract_YYYYMMDD_HHMMSS.jsonOnce tests are generated (generate_tests), they are registered in .utf/utf.db. Subsequent calls to utf report, utf status, and query registry read from this persistent store — no re-generation required between sessions.
When the MCP server cannot infer your project root automatically, pass project_dir explicitly:
generate unit tests for src/auth.py in project C:/my-projectOr configure "cwd": "${workspaceFolder}" in your mcp.json (already set in the quickstart above) so the server always starts in the correct workspace.
After running .\scripts\install-vscode-mcp.ps1 (Option 2) or adding the uvx MCP entry (Option 1):
Note: The
@utfprefix and/slash-commandsyntax only work if you have the UTF VS Code Chat Participant extension installed (included via Option 2). In GitHub Copilot CLI and standard VS Code Copilot Chat without the extension, use natural language — the MCP tools are invoked automatically. See GitHub Copilot CLI Usage above.
Command | Effect |
| Unit tests for selected/active code |
| API tests |
| Integration tests |
| End-to-end tests |
| Security / auth tests |
| Performance / load tests |
| Validate a test against the 8-section contract |
| Identify coverage gaps in the current suite |
| Build requirements → tests traceability matrix |
| Generate HTML contract compliance report |
| Server health and installation status |
In standard VS Code Copilot Chat or GitHub Copilot CLI, just ask:
generate unit tests for this file
show utf status
utf report
what are my coverage gaps?
validate this test against the contractUTF reads the open file, detects language and framework, infers requirements from function signatures, generates happy-path + negative + edge-case tests — all validated against the 8-section contract.
Use UTF directly in scripts or CI pipelines without the MCP server:
from mcp_server.tools.generate_tests import generate_tests
from mcp_server.tools.validate_contract import validate_test_contract
# Generate tests — UTF infers language, framework, and requirements
result = generate_tests(
test_type="unit",
source_code=open("src/auth.py").read(),
requirements_text="US-101: Users must be authenticated before accessing dashboard",
)
print(result["suite_code"]) # Executable test file
print(result["traceability_matrix"]) # Requirements → tests mapping
print(result["gaps"]) # Identified coverage gaps
# Validate any existing test
validation = validate_test_contract(test_content=my_test_markdown)
print(f"Score: {validation['score']:.0%} Valid: {validation['is_valid']}")# .github/workflows/test-quality-gate.yml
name: UTF Contract Gate
on: [push, pull_request]
jobs:
contract-gate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: { python-version: "3.11" }
- run: pip install universal-test-framework
- name: Validate test contract
run: |
python - <<'EOF'
from mcp_server.tools.validate_contract import validate_test_contract
import glob, sys, pathlib
failures = []
for f in glob.glob("tests/**/*.py", recursive=True):
content = pathlib.Path(f).read_text()
result = validate_test_contract(test_content=content)
if not result["is_valid"]:
failures.append(f"{f}: score {result['score']:.0%}")
if failures:
print("Contract failures:\n" + "\n".join(failures))
sys.exit(1)
print("All tests passed contract validation")
EOFCreate .utf/rules/project-overrides.yaml in your project root:
# Raise minimum score for safety-critical code
contract:
min_score: 0.90 # default: 0.85
hard_block_below: 0.75
# Match your Jira project key
traceability:
requirement_id_pattern: "^(PROJ-\\d+|AC-\\d+(\\.\\d+)?|NFR-\\d+)$"
# Tests generated per requirement per type
scenario_counts:
happy_path: 1
negative: 2
boundary: 1
edge_case: 1
# Coverage advisory thresholds (appear in report, do not block)
coverage:
line_target: 85
branch_target: 75
mutation_score_target: 70Rules are deep-merged: project overrides layer on top of UTF's global defaults. The 8-section contract structure itself cannot be overridden.
UTF MCP Server (stdio or HTTP/SSE)
│
├── mcp_server/server.py — FastMCP entrypoint, 13 registered tools
│
├── mcp_server/engine/ — Core processing
│ ├── language_detector.py — Detects language + framework from code/path
│ ├── rule_engine.py — Loads and merges YAML rules
│ ├── contract_validator.py — Enforces 8-section contract, scores tests
│ ├── template_renderer.py — Jinja2 test file generation
│ ├── context_resolver.py — Resolves project context for generation
│ └── framework_mapper.py — Maps language → framework → test runner
│
├── mcp_server/tools/ — One module per MCP tool
├── mcp_server/registry/ — SQLite per-project test registry
├── mcp_server/execution/ — pytest, Vitest, Maven, go-test adapters
├── mcp_server/mutation/ — mutmut, Stryker, PIT, Gremlins adapters
├── mcp_server/reporting/ — HTML, JUnit XML, JSON report generation
├── mcp_server/feedback/ — CI listener, trend analysis, gap reopener
│
├── rules/ — Global YAML rules (language, framework, type)
├── templates/ — Jinja2 test templates per language/framework
├── agent-customization/ — VS Code copilot-instructions + prompt palette
└── vscode-extension/ — @utf VS Code Chat Participant (VSIX)Transport modes:
stdio — default, used by VS Code MCP client and uvx
HTTP/SSE — for remote/team deployment (--transport http --port 8765)
SQLite registry resolves to .utf/utf.db relative to the caller's project root (the cwd in mcp.json, or the project_dir parameter passed to any tool). Each project has its own isolated registry — no shared state. The registry persists across all sessions until explicitly cleared.
Method | Command |
| Requirements |
uvx (zero-clone) |
|
| |
Local clone |
|
| Git, Python 3.11+ |
Docker |
| Named volume; bind-mount for per-project | Docker |
GitHub MCP Registry | Auto via MCP client |
| uv (auto-installed) |
pip + SDK |
| Pass | Python 3.11+ |
All setups resolve the SQLite registry path using the same priority chain:
1. project_dir argument (explicit per-tool call)
2. UTF_PROJECT_DIR environment variable
3. Path.cwd() at server start (fallback — avoid for multi-project use)The recommended approach for all setups: set both "cwd": "${workspaceFolder}" and
"env": { "UTF_PROJECT_DIR": "${workspaceFolder}" } in your mcp.json entry.
This ensures registry isolation works even if a tool call omits project_dir.
All MCP tool inputs are validated via Pydantic before processing
No secrets, credentials, or PII are logged or stored
The registry (utf.db) is local to each project and never transmitted
Docker image runs as non-root user
Rate limiting is enforced on the HTTP/SSE transport
MIT — see LICENSE
Analyze the coverage of a test suite and identify gaps.
| Name | Required | Description | Default |
|---|---|---|---|
| tests | Yes | List of test dicts (output from generate_tests) | |
| language | No | Language for tool recommendations | |
| test_type | No | Test type for threshold lookup | unit |
| source_code | No | Source code to identify uncovered symbols |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must disclose behavioral traits. It only states what the tool does without indicating side effects, permissions, or output characteristics. It does not say whether it modifies state or what 'gaps' means in terms of behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, front-loaded sentence that efficiently conveys the tool's purpose. There is no unnecessary repetition, and it is easy to read. However, it could be expanded slightly to improve completeness without sacrificing brevity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite having an output schema and 4 parameters, the description is very brief. It does not explain what constitutes a 'gap', how coverage is measured, or the role of optional parameters like 'language' and 'source_code'. This under-specification leaves the agent unclear on how to invoke the tool effectively.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 100% description coverage, so parameters are documented in the schema. The description adds no additional semantics beyond the schema. Baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: analyzing test coverage and identifying gaps. The verb 'analyze' with the resource 'coverage' is specific. However, it does not explicitly distinguish itself from sibling tools like 'validate_test_contract' or 'build_traceability_matrix', which might also involve analysis.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool, prerequisites, or when alternatives are better suited. There is no mention of required input state or context. For a tool with multiple siblings, this omission reduces usability.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Build a requirements-to-tests traceability matrix.
| Name | Required | Description | Default |
|---|---|---|---|
| tests | Yes | List of test dicts (must include test_id and requirement_mapping) | |
| requirements | No | Explicit list of requirement IDs to check coverage against |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden but only says 'Build a ... matrix', omitting behavioral details such as whether the matrix is returned, saved, or has side effects. No mention of required permissions, rate limits, or idempotency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence, concise and front-loaded. However, it could be slightly more informative without significant expansion.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given that an output schema exists, return value explanation is less critical. However, the description lacks context about prerequisites (e.g., format of test objects), error handling, or performance implications for large arrays.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so baseline is 3. The description does not add extra meaning beyond what the schema already provides for each parameter.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Build' and the specific resource 'requirements-to-tests traceability matrix', distinguishing it from sibling tools like analyze_coverage and generate_tests.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives such as analyze_coverage or generate_tests. The description does not mention prerequisites, typical use cases, or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Detect programming language and test framework from source code.
| Name | Required | Description | Default |
|---|---|---|---|
| file_path | No | Optional file path for extension-based detection | |
| source_code | Yes | Source code to analyze |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, so the description must compensate. However, it only states the detection function and reveals no behavioral traits such as execution context, latency, side effects, or security implications. The agent cannot assess safety or performance expectations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence with no wasted words. It is front-loaded with the primary action. While terse, it efficiently conveys the core purpose, earning a high score for conciseness despite lacking depth.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the existence of an output schema, return values need not be explained. However, the description omits details like supported languages/frameworks, error handling, and the role of the file_path parameter. This incompleteness could lead to incorrect invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents both parameters. The description does not add extra meaning beyond what is in the schema. Baseline of 3 is appropriate as no additional clarification is needed.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool detects programming language and test framework from source code, using a specific verb and resource. It distinguishes from sibling tools like generate_tests or validate_test_contract which have different purposes.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No usage guidance is provided. The description does not indicate when to use this tool, when not to, or mention any alternative tools among the siblings. The agent receives no hints about prerequisites or scenarios for using this tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Get UTF feedback loop status: gap analysis, coverage health, trend, and delta requirements.
| Name | Required | Description | Default |
|---|---|---|---|
| check_gaps | No | Run gap analysis and coverage health check (default: True) | |
| project_id | No | Project to report on (defaults to current directory name) | |
| trend_days | No | Number of days of history to include in trend (default: 30) | |
| project_dir | No | Absolute path to the caller's project root. Points the registry at <project_dir>/.utf/utf.db. Defaults to the current working directory of the MCP client process. | |
| requirements | No | Requirements text to compare against registry for delta detection | |
| compute_trend | No | Compute coverage trend over time (default: True) |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden. It indicates a read operation ('Get') but does not explicitly state that it is non-destructive, has no side effects, or any authentication/rate-limit constraints. The description is minimal but not misleading.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single concise sentence that front-loads the main purpose and key outputs. Every part is meaningful and no redundant information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has 6 optional parameters and an output schema (not shown), the description covers the essential facets. It does not detail return format, but the output schema presumably handles that. Slight gap: no mention of defaults or how project context is resolved, but overall adequate for a status-query tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
All 6 parameters have schema descriptions (100% coverage), so the baseline is 3. The tool description adds context by mapping components like 'gap analysis' to check_gaps and 'trend' to compute_trend, but this is already implied by parameter names and schema descriptions. No significant extra meaning beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Get', the resource 'UTF feedback loop status', and enumerates the components: gap analysis, coverage health, trend, and delta requirements. This is specific and distinguishes it from sibling tools like analyze_coverage or generate_report.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies its use for obtaining a summary status of the feedback loop, but it lacks explicit guidance on when to use this tool versus alternatives (e.g., query_registry for raw data, analyze_coverage for deep dive). No when-not-to-use or alternative references are provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Generate the 8-section contract compliance report (Tool #12).
Reads test records and execution results from the registry, builds the contract report in the requested formats, and returns report paths + summary.
| Name | Required | Description | Default |
|---|---|---|---|
| cwd | No | Project root directory (defaults to server cwd). | |
| formats | No | Output formats: subset of ["html", "junit", "json"]. Defaults to all three. | |
| open_html | No | Open the HTML report in the default browser (dev mode). | |
| project_id | No | Registry project ID (defaults to current directory name). |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It discloses that the tool reads from the registry, builds the report, and returns paths and summary. It does not mention any destructive actions (likely none), and the read-only nature is implied. However, it could explicitly state that the tool does not modify any data.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loading the purpose and then summarizing the process and output. It is concise, but the inclusion of 'Tool #12' is unnecessary and adds no value for the AI agent. Otherwise, every sentence earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity and the presence of an output schema, the description is largely complete. It explains that input comes from the registry and output includes report paths and a summary. However, it does not mention prerequisites (e.g., that test records must exist) or the structure of the summary.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the baseline is 3. The description does not add additional meaning beyond the schema; it only reinforces that formats are requested. The description's mention of 'requested formats' aligns with the schema but does not provide extra context.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool generates an '8-section contract compliance report', specifies it reads from the registry and produces multiple formats, and returns paths and summary. It is a specific verb+resource that distinguishes it from siblings like 'generate_tests' or 'validate_test_contract'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies the tool is used after test execution to build a compliance report, but it does not explicitly state when to use it versus alternatives (e.g., after tests are run, or when a comprehensive compliance report is needed). No when-not-to-use or alternative tooling guidance is provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Generate a complete test suite satisfying the 8-section test contract.
The framework automatically:
Detects language and framework from source code
Generates tests for happy path, failure paths, and boundary cases
Validates every test against the 8-section contract before returning
Builds a traceability matrix linking tests to requirements
Identifies coverage gaps and makes recommendations
| Name | Required | Description | Default |
|---|---|---|---|
| language | No | Override language detection. One of: python | typescript | javascript | java | go | cpp | |
| file_path | No | File path hint for language detection (e.g., 'src/auth.py') | |
| framework | No | Override framework detection. One of: pytest | jest | junit5 | go-test | playwright | k6 | |
| test_type | Yes | Type of tests to generate. One of: unit | integration | api | e2e | contract | performance | security | |
| project_dir | No | Absolute path to the caller's project root. The SQLite registry and reports will be stored under <project_dir>/.utf/. Defaults to the current working directory of the MCP client process. | |
| source_code | No | Source code to analyze (function/class/module). Optional but recommended. | |
| requirements_text | No | User stories, acceptance criteria, Jira tickets, or requirements. Include IDs like US-001, AC-2.1, REQ-042 for traceability. |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description discloses key behaviors: automatic language/framework detection, test generation for multiple path types, contract validation, traceability matrix building, and coverage gap identification. Since no annotations are provided, the description carries the full burden. However, it omits the side effect of storing results in <project_dir>/.utf/, which is mentioned only in the parameter description.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise with 6 bullet points and a clear first sentence. Every sentence adds value without redundancy. Slightly more structure (e.g., separating mandatory/optional steps) could improve it, but it is efficiently written.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description adequately explains the tool's automatic capabilities given the 7-parameter schema and existence of an output schema. However, it fails to mention the persistent storage side effect (registry and reports written to disk), which is important for understanding the tool's full impact. Output schema reduces the need to describe return values, but behavioral completeness is slightly lacking.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% with all parameters well-documented. The description does not add parameter-specific meaning beyond the schema; it provides high-level context (e.g., automatic detection) but does not elaborate on individual parameters. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Generate a complete test suite satisfying the 8-section test contract.' The verb is specific (Generate), the resource is a test suite with a defined scope (8-section contract). It distinguishes itself from sibling tools like 'validate_test_contract' and 'analyze_coverage' by focusing on generation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when to use this tool versus alternatives. The description lacks 'when to use', 'when not to use', or mentions of prerequisites. Siblings like 'validate_test_contract' or 'analyze_coverage' exist, but the description does not guide the agent on choosing this tool over them.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Health check for the Universal Test Framework MCP server.
Returns server status, version, and uptime. Suitable for use as a Docker HEALTHCHECK command, Kubernetes liveness/readiness probe, or basic availability verification.
Returns: status : "ok" when the server is healthy version : UTF semantic version string uptime_s : seconds the server process has been running tools : number of registered MCP tools
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Fully discloses return fields and implies read-only, safe operation. No annotations provided, but description compensates completely.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Short, front-loaded with purpose and use cases, then return fields. Every sentence is informative and concise.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Complete for a parameterless health check: defines purpose, usage context, and output. No gaps given the tool's simplicity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
No parameters; schema coverage 100% vacuously. Description adds value by detailing return values beyond schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Clearly states it is a health check returning status, version, uptime, and tool count. Distinct from siblings like generate_tests or run_tests.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly lists suitable use cases: Docker HEALTHCHECK, Kubernetes probes, and basic availability verification, providing clear guidance on when to use.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Import JUnit XML execution results into the UTF registry.
Use this when you have already run tests with pytest / Maven / Go and want to register the results so that generate_report shows real pass/fail rates.
| Name | Required | Description | Default |
|---|---|---|---|
| language | No | Language label (python|typescript|...). Default: python | python |
| framework | No | Framework label (pytest|jest|...). Default: pytest | pytest |
| test_type | No | Test type label (unit|integration|api|e2e|...). Default: e2e | e2e |
| project_id | No | Override project name in registry (defaults to project_dir name). | |
| project_dir | No | Absolute path to caller's project root. Registry stored at <project_dir>/.utf/utf.db. Defaults to UTF_PROJECT_DIR or cwd. | |
| junit_xml_path | Yes | Path to the JUnit XML file (absolute or relative to project_dir). |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must carry the full burden. It describes the core action and references the registry but does not disclose behavioral details like overwrite behavior, authorization needs, or error handling.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences: the first stating the action and the second providing usage context. It is efficient, though the term 'UTF registry' may be slightly obscure without context.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the rich input schema (100% coverage) and the presence of an output schema, the description adequately covers the tool's purpose, prerequisites, and downstream integration with generate_report. Minor gaps like behavioral specifics are acceptable.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the description does not need to add extra parameter meaning. It does not elaborate beyond what the schema already provides, achieving the baseline score.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Import' and the resource 'JUnit XML execution results into the UTF registry'. It also distinguishes from siblings like run_tests and generate_report by specifying that tests have already been run.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives explicit guidance on when to use: 'when you have already run tests... and want to register the results so that generate_report shows real pass/fail rates'. It implies but does not explicitly state when not to use or name alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Query the UTF persistent test registry.
| Name | Required | Description | Default |
|---|---|---|---|
| status | No | Filter by status (generated | executed | failed | gap) | |
| language | No | Filter by language (python | typescript | …) | |
| test_type | No | Filter by test type (unit | integration | api | …) | |
| project_id | No | Filter by project (defaults to current directory name) | |
| project_dir | No | Absolute path to the caller's project root. Points the registry query at <project_dir>/.utf/utf.db. Defaults to the current working directory of the MCP client process. | |
| requirement_id | No | Filter tests that cover a specific requirement ID |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided. The description does not disclose behavioral traits such as read-only nature, side effects, authentication requirements, or limitations. It only states it queries the registry.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, front-loaded sentence with no wasted words. It is concise, though perhaps overly brief for such a complex tool.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite an output schema existing, the description lacks context about what the registry is, what the query returns, and how results are structured. Given six parameters and a rich sibling set, more detail is warranted.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with each parameter having a description in the schema. The description adds no additional parameter information, so baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's function: 'Query the UTF persistent test registry.' It uses a specific verb ('Query') and resource ('test registry'), and distinguishes from sibling tools like generate_tests or run_tests.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives (e.g., analyze_coverage, generate_tests). No context about prerequisites or scenario appropriateness.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Parse test files and register per-method 8-section contract records.
This is the UTF registration bridge — it reads LLM-written test files, extracts the per-method 8-section comment blocks, and upserts status=generated rows into the UTF registry. Without this step, generate_report has no Per-Test Contract Detail cards.
IMPORTANT — Phase 1, Step ③ of the UTF 3-phase workflow: ① generate_tests (scaffold) ② Write real test methods with per-method TC-{PRJ}-{MODULE}-{NNN} blocks ③ register_contracts ← this tool ④ generate_report (verify contract detail cards) ⑤ pytest --junit-xml=... ⑥ import_test_results ⑦ generate_report (now shows both contract cards AND execution results)
| Name | Required | Description | Default |
|---|---|---|---|
| language | No | One of python|typescript|java|go|... Default: python | python |
| framework | No | One of pytest|jest|junit5|... Default: pytest | pytest |
| test_type | No | One of unit|integration|api|e2e|... Default: e2e | e2e |
| project_id | No | Registry project label. Defaults to project_dir basename. | |
| test_files | Yes | List of test file paths (absolute or relative to project_dir). | |
| project_dir | No | Absolute path to project root (.utf/utf.db lives here). Defaults to UTF_PROJECT_DIR env var or cwd. |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It explains the tool reads test files, extracts comment blocks, and upserts rows into the registry, which is transparent. However, it does not disclose error handling, permissions, or output schema details.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with a clear first line, an explanation paragraph, and a numbered workflow list. Every sentence is informative with no wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the complexity (6 params, output schema, many siblings), the description effectively covers the tool's role in the workflow. It omits edge cases or failure modes but is sufficient for typical use.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3. The description adds workflow context but does not elaborate on parameter usage beyond what the schema already provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it parses test files and registers per-method contract records. It distinguishes itself by placing it as a specific step in the UTF workflow, differentiating from siblings like generate_tests and generate_report.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly lists the 3-phase workflow with steps ①-⑦, positioning register_contracts as Step ③. It also states that without this step, generate_report has no contract detail cards, providing clear when-to-use context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run mutation testing on a source+test file pair and return coverage metrics.
Supports mutmut (Python), Stryker (JS/TS), PIT (Java), and gremlins (Go). The adapter is auto-detected based on language and tool availability.
| Name | Required | Description | Default |
|---|---|---|---|
| language | Yes | Language of the files (python | javascript | typescript | java | go) | |
| test_file | Yes | Path to the test file to run against mutants | |
| block_below | No | Score below which the result is flagged as blocked (default: 0.50) | |
| project_dir | No | Project root directory (defaults to cwd) | |
| source_file | Yes | Path to the source file to mutate | |
| minimum_score | No | Threshold to pass (default: 0.70 = 70% killed) | |
| timeout_seconds | No | Mutation run timeout in seconds (default: 300) |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It mentions auto-detection of adapters and return of coverage metrics, but lacks details on side effects, permissions, or failure behavior. The timeout parameter is in the schema, not the description.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences that front-load the purpose followed by supported tools. No wasted words, clear and efficient structure.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the 7 parameters (3 required) and existence of an output schema, the description covers the main purpose and tool support. It could elaborate on parameter roles like block_below and minimum_score, but the output schema compensates for return values.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the baseline is 3. The description does not add meaning beyond the schema; it only mentions supported tools and auto-detection, not parameter details.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'run' and the resource 'mutation testing on a source+test file pair'. It distinguishes from sibling tools like 'run_tests' and 'analyze_coverage' by focusing specifically on mutation testing and listing supported adapters.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for mutation testing but does not explicitly state when to use this tool versus alternatives. No when-not or alternative tool names are provided beyond the sibling list.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Execute test files and return structured results with CI annotations.
Execution is always optional — controlled by UTF config. Never blocks generation.
| Name | Required | Description | Default |
|---|---|---|---|
| language | Yes | Language of the tests (python | typescript | javascript | java | go) | |
| framework | No | Test framework hint (pytest | vitest | jest | junit5 | maven | gradle | go) | |
| test_files | Yes | List of test file paths to execute | |
| project_dir | No | Project root directory (defaults to cwd) | |
| timeout_seconds | No | Per-file execution timeout in seconds (default: 120) |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must disclose behaviors. It mentions 'Never blocks generation' which is a key behavioral trait, but omits details like failure modes, side effects, or required permissions. Minimal transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with no unnecessary words. Front-loaded with the primary action and output. Every word earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has 5 parameters and an output schema, but the description does not explain return values or error handling. For a test execution tool, details about partial failures or result structure would improve completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3. The description adds no additional meaning beyond the parameter descriptions already present in the schema (e.g., language, framework, etc.). Hence no extra value.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Execute test files') and the output ('structured results with CI annotations'). It does not explicitly differentiate from siblings like 'generate_tests', but the verb 'execute' makes its purpose distinct.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives like 'analyze_coverage' or 'validate_test_contract'. The only usage hint is that execution is optional and non-blocking, which is context but not a comparison.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Analyze code/requirements and suggest which test types to apply.
| Name | Required | Description | Default |
|---|---|---|---|
| source_code | No | Source code to analyze | |
| requirements_text | No | Requirements or user stories |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It mentions analyzing and suggesting but does not disclose whether the tool is deterministic, what side effects (if any) occur, required permissions, or the nature of suggestions (list of types, ranked scores, etc.). The term 'suggest' is vague.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, concise sentence that effectively communicates the core function. However, it lacks structure (e.g., no additional context or examples) and could benefit from being slightly more detailed without becoming verbose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite having an output schema (not shown) and fully described parameters, the description does not specify what form the suggestions take (e.g., a list of test type names, a prioritized output). It omits important contextual details that would help an agent use the tool effectively, such as whether both parameters are required or how they interact.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, with both parameters ('source_code', 'requirements_text') having clear descriptions. The tool description adds 'analyze code/requirements' which aligns with the parameters but does not provide additional clarity beyond the schema. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: analyzing code and/or requirements to suggest test types. This verb+resource structure effectively distinguishes it from sibling tools like 'generate_tests' (which generates actual tests) and 'analyze_coverage' (which analyzes code coverage).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance is provided on when to use this tool versus its alternatives. The description lacks any 'when to use' or 'when not to use' context, which is particularly problematic given the number of closely related sibling tools (e.g., 'run_tests', 'generate_tests', 'validate_test_contract').
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Validate a test or test suite against the 8-section contract.
Use this to check any existing test — generated or hand-written — for compliance. The contract requires all 8 sections: Test ID, Why Generated, Requirement Mapping, How it Exercises, Coverage Contribution, Expected Outcome, Gaps, Meaningfulness Check.
| Name | Required | Description | Default |
|---|---|---|---|
| test_dict | No | Structured test as dict with 8-section keys | |
| tests_list | No | List of raw markdown strings for suite-level validation | |
| test_content | No | Raw markdown test text (paste any test here for quick validation) |
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden. It describes the validation purpose and required sections but does not disclose details about side effects, permissions, or return format. The output schema exists but is not referenced.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences: first defines the action, second provides usage context and lists required sections. No filler, front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Complete enough for a validation tool with an output schema. It explains what the tool does and when to use it, though it could clarify which parameter to choose for different inputs.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so baseline is 3. The tool description does not add meaning beyond the schema descriptions, which already explain each parameter's role.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Clearly states the tool validates a test or test suite against an 8-section contract. The verb 'validate' and resource 'test or test suite' are specific and distinct from sibling tools like generate_tests or analyze_coverage.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly says 'Use this to check any existing test — generated or hand-written — for compliance,' indicating when to use. However, it does not specify when not to use or mention alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
v1.0.3analyze_coveragebuild_traceability_matrixdetect_language_frameworkfeedback_statusgenerate_reportgenerate_testshealthimport_test_resultsquery_registryregister_contractsrun_mutation_testsrun_testssuggest_test_typesvalidate_test_contractMost tools have distinct purposes, but analyze_coverage and feedback_status both address coverage gaps, causing slight ambiguity. Overall, the majority are clearly distinguishable.
Nearly all tools follow a consistent verb_noun snake_case pattern (e.g., generate_tests, run_mutation_tests). Only health deviates slightly but is still clear.
With 14 tools, the surface covers all necessary operations for a test framework server without being excessive. Each tool serves a well-defined role.
The tool set covers the entire workflow from test generation, validation, registry, execution, mutation testing, reporting, and import. No obvious gaps for the intended domain.
Unclaimed servers have limited discoverability.
If you are the server author, to access and configure the admin panel.
Generate a typed SDK, CLI, and MCP server from any OpenAPI or GraphQL spec, and keep them current.
3rd Generation Testing (3TG) — generate deterministic test suites from Markdown spec tables via MCP.
Deterministic validation for AI-generated artifacts: JSON Schema, OpenAPI response, SQL syntax.
One MCP endpoint for Claude, GPT & Gemini: 100+ tools + no-code connectors + agent workers.
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/phoenice-labs/Universal-Test-Framework'
If you have feedback or need assistance with the MCP directory API, please join our Discord server
