testgen-eval

An agent that tests another agent. testgen generates test suites from requirements using Claude; testgen-eval calls it over MCP, runs a deterministic gate plus an LLM-judge pass against a written quality rubric, and reports back what's good, what's weak, and what's outright broken.

npm run batch -- sample-requirements.json out/
open out/report.html

Why this exists

This is the second half of a two-project pair built to practice things a single CLI tool doesn't force you to learn: MCP as a real client/server boundary, spec-driven development (writing down what "good output" means before building the thing that judges it), and — the part most worth a second look — a framework for agents that evaluate other agents. As LLM agents show up more in test automation, "how do you know the agent's output is actually trustworthy" becomes the real question, and this project is a concrete answer.

testgen-eval doesn't reimplement testgen. It treats it as an external service, reachable only through the MCP tool generate_test_suite — the same constraint a real third-party client would have.

Related MCP server: Swagger Testcase MCP

How it works

flowchart TD
    R[Requirement + instructions] --> MCP[MCP client]
    R --> CLS[Classifier]
    MCP -->|generate_test_suite| TG[testgen HTTP API] --> TS[(TestSuite)]
    CLS -->|"security/NFR-sensitive?"| SENS[(sensitivity verdict)]
    TS --> GATE{Gate<br/>deterministic}
    GATE -->|fail| REJECT[Rejected - reported, not judged]
    GATE -->|pass| JUDGE[Judge - LLM, one call]
    SENS --> JUDGE
    JUDGE --> REPORT[Report<br/>markdown / json / html]
    REJECT --> REPORT
    REPORT --> SUMMARY[Batch summary<br/>pass rate, avg score, worst offenders]

Each requirement goes through the same five stages (full breakdown in ARCHITECTURE.md):

1. MCP client spawns testgen-eval's own MCP server and calls generate_test_suite — the only way this harness talks to testgen.

2. Classifier — a separate, small LLM call judging whether the requirement is security/NFR-sensitive (auth, payments, PII, rate limiting). Runs concurrently with generation; the result feeds the gate check below rather than being decided ad hoc.

3. Gate — plain TypeScript, no LLM. Rejects a suite only if it's genuinely unusable (empty IDs, cases with no steps, an empty feature name). A suite that passes the gate is usable; it isn't necessarily good.

4. Judge — one structured LLM call scoring a gate-passed suite against ten quality criteria (concreteness, realism, duplication, whether instructions were actually reflected, missing security coverage for sensitive requirements, scenario conflation, and more) — see SPEC.md for the full rubric and the reasoning behind it.

5. Report + batch summary — every requirement gets its own report (markdown, JSON, or HTML); a full batch also gets one aggregate summary (gate pass rate, average score, worst offenders, and any pipeline failures called out separately from low scores).

What "good" means

The rubric lives in SPEC.md, not in this README, because it's meant to be read on its own and argued with. The short version: the gate only protects against a test case being non-actionable — a tester literally can't execute it. Everything about completeness, realism, or polish is a scored quality signal, never a rejection reason.

Setup

Requires testgen running locally — this harness calls it, it doesn't embed it:

cd ../testgen
.venv\Scripts\activate      # macOS/Linux: source .venv/bin/activate
testgen serve                # stays up at http://127.0.0.1:8000

Then, in testgen-eval:

npm install

Both testgen and testgen-eval read ANTHROPIC_API_KEY from the environment.

Usage

Run a batch through the full harness

npm run batch -- <requirements.json> [outDir]

requirements.json is a JSON array:

[
  {
    "requirement": "Users must be able to reset their password via an emailed link.",
    "instructions": "focus on security and boundary cases"
  },
  { "requirement": "Users can toggle dark mode from the settings screen." }
]

(sample-requirements.json in this repo is a ready-to-run example.)

Each run writes three things to outDir (defaults to out/):

A pipeline failure on one requirement (a timeout, a malformed response) doesn't abort the batch — it's isolated, logged into results.jsonl and the error banner, and the rest of the batch keeps going.

Try just the MCP server

The harness's own MCP server (generate_test_suite) can also be driven directly, without the eval pipeline around it — useful for seeing the protocol itself:

npx @modelcontextprotocol/inspector tsx src/index.ts

Project structure

src/
  testgenClient.ts   testgen's HTTP API client
  server.ts           the MCP server and its one tool
  index.ts             MCP server entrypoint (stdio transport)
  harness/
    mcpClient.ts       MCP client - reuses one connection across a batch
    classifier.ts      requirement sensitivity classification
    gate.ts            deterministic structural checks
    judge.ts           LLM-judge quality scoring
    report.ts          per-requirement report (markdown / json / html)
    batch.ts           aggregate batch summary (markdown / json / html)
    htmlPage.ts         assembles the two html fragments into report.html
    htmlEscape.ts       XSS-safe escaping for LLM-sourced content
    run.ts              orchestrates everything, loops over a batch

Development

npm test

Covers everything deterministic — gate.ts, report.ts's and batch.ts's renderers — offline, no API calls. classifier.ts, judge.ts, mcpClient.ts, and run.ts call real LLMs/APIs, so they're smoke-tested by hand rather than covered by the automated suite.

Roadmap

• Resume a partially-completed batch instead of always starting over.

• Model test layer (UI vs API) in testgen so type-coverage can become a real scored criterion instead of an informational one (see SPEC.md).

• Additional input sources beyond a hand-written JSON file.

Related

• testgen — the agent under test.

• SPEC.md — what "good" means for a generated test suite.

• ARCHITECTURE.md — the harness's components and the decisions behind them.

License

MIT