Provides access to Gemini models for LLM evaluation, including generating responses and scoring faithfulness/relevance as a judge.
Provides access to GPT models for LLM evaluation, including generating responses and scoring faithfulness/relevance as a judge.
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@mcp-llm-eval Run evaluation on dataset qa_v1 with model gpt-4"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Status: stable. Used in CI gates by
mcp-content-pipeline. Live benchmarks at llmshot.vercel.app.
A local Model Context Protocol (MCP) server that packages LLM evaluation gates as reusable CI/CD primitives. Run datasets against multiple models, score responses with an LLM-as-judge, and enforce quality thresholds — all through MCP tools that AI agents can call.
flowchart LR
A[PR opened] --> B[Run dataset<br/>through models]
B --> C[Judge scores<br/>faithfulness + relevance]
C --> D{Thresholds met?}
D -->|Yes| E[PR passes]
D -->|No| F[PR blocked<br/>with diff comment]There's no unit test for LLM quality. Teams ship prompt changes, swap models, or update system prompts with no automated way to verify that output quality didn't regress. Manual spot-checking doesn't scale, and existing eval frameworks are heavy, opinionated, and hard to wire into CI/CD.
mcp-llm-eval gives AI agents structured access to a lightweight eval pipeline. Instead of building custom scripts for every project, you define a dataset, point the agent at it, and get scored results with pass/fail gates — the same workflow whether you're testing locally or gating a deployment.
Related MCP server: agentloop
mcp-llm-eval is the evaluation engine for a small ecosystem of repos:
Producers that consume this library: mcp-content-pipeline and this repo itself (dogfoods its own engine on a self-defined dataset)
Data layer: each producer writes benchmark JSON to llm-benchmarks
Visualization: LLMShot renders all three domains live at llmshot.vercel.app
This separation — engine here, golden datasets in the consuming repos, data and dashboard in dedicated public repos — means each producer defines its own quality bar without forking the engine.
Tool | Description |
| Load a dataset, query models via streaming, score with LLM-as-judge, return per-question scores and aggregate summary |
| Validate evaluation results against quality gates (faithfulness, relevance, TTFT, cost, retrieval, RAG) |
| List past evaluation runs with metadata (timestamp, models, cost, pass/fail) |
| Retrieve full details of a specific run (per-question scores, responses, judge reasoning) |
| Compare two evaluation runs and detect regressions beyond configurable tolerance |
| Generate a markdown PR comment from evaluation results with regression details and threshold status |
| Run retrieval metrics (recall@k, precision@k, MRR, nDCG@k) against a labelled chunk dataset; returns per-query metrics, aggregate, p50/p95 latency |
| Full RAG pipeline — retrieve, generate, score with |
| Compare two retrieval result files and flag metrics that regressed beyond tolerance |
| Reserved stub; schema is stable today and returns a not-implemented response |
Generation:
Faithfulness (0-1) — Is the response grounded in the provided context?
Relevance (0-1) — Does the response actually answer the question?
Time to First Token — Streaming latency in milliseconds
Cost per Query — Estimated cost based on token usage and provider pricing
Retrieval and RAG (v0.5.0):
Recall@k, Precision@k, MRR, nDCG@k — Standard IR metrics against labelled relevant_chunk_ids (binary relevance)
Context relevance (0-1) — LLM-as-judge score for each retrieved chunk against the query, averaged per query
Citation faithfulness (0-1) — LLM-as-judge score for whether the generated answer is supported by the retrieved chunks
p50 / p95 retrieval latency — Per-query timer wrapped around adapter.retrieve()
pip install mcp-llm-evalThen install the provider SDKs you need (they are not bundled):
# Pick what you use
pip install anthropic # for Claude models
pip install openai # for GPT models + judge
pip install google-genai # for Gemini modelsAdd this to your Claude Desktop MCP configuration file:
OS | Path |
macOS |
|
Windows |
|
Recommended — with uvx (no install required):
{
"mcpServers": {
"llm-eval": {
"command": "uvx",
"args": ["mcp-llm-eval"],
"env": {
"ANTHROPIC_API_KEY": "sk-ant-...",
"OPENAI_API_KEY": "sk-...",
"GOOGLE_API_KEY": "AIza..."
}
}
}
}Note: Only include API keys for the providers you plan to evaluate. For example, if you only use Anthropic and OpenAI (for the judge), omit
GOOGLE_API_KEY.
Note: Claude Desktop may not inherit your terminal's
$PATH. If the server fails to connect, use the absolute path touvx(find it withwhich uvx):{ "mcpServers": { "llm-eval": { "command": "/full/path/to/uvx", "args": ["mcp-llm-eval"], "env": { "ANTHROPIC_API_KEY": "sk-ant-...", "OPENAI_API_KEY": "sk-..." } } } }
Alternative — installed via pip:
{
"mcpServers": {
"llm-eval": {
"command": "mcp-llm-eval",
"env": {
"ANTHROPIC_API_KEY": "sk-ant-...",
"OPENAI_API_KEY": "sk-...",
"GOOGLE_API_KEY": "AIza..."
}
}
}
}Alternative — from source (virtualenv):
{
"mcpServers": {
"llm-eval": {
"command": "/absolute/path/to/mcp-llm-eval/.venv/bin/python",
"args": ["-m", "mcp_llm_eval.server"],
"env": {
"ANTHROPIC_API_KEY": "sk-ant-...",
"OPENAI_API_KEY": "sk-...",
"GOOGLE_API_KEY": "AIza..."
}
}
}
}Fully quit (Cmd+Q on macOS) and reopen. Look for the tools icon to confirm the server is connected.
"Run the eval dataset at
/path/to/dataset.jsonagainst Claude Sonnet and GPT-4o, then check if faithfulness is above 0.8."
Claude autonomously chains the tools — running the evaluation, then checking thresholds:
Running evaluation...
- Dataset: 9 questions (3 factual, 3 reasoning, 3 summarization)
- Models: claude-sonnet-4-6, gpt-4o-mini
- Judge: gpt-4o-mini
Results:
claude-sonnet-4-6: avg faithfulness=0.83, relevance=0.83, TTFT=1367ms, cost=$0.0035/q
gpt-4o-mini: avg faithfulness=0.94, relevance=0.94, TTFT=1194ms, cost=$0.0001/q
Threshold check:
avg_faithfulness >= 0.80: PASS (actual: 0.889)
avg_relevance >= 0.75: PASS (actual: 0.889)
p95_ttft_ms <= 2000: PASS (actual: 1367ms)
max_cost_per_query <= 0.01: PASS (actual: $0.0035)
Overall: PASS
> Full benchmark across 8 models lives at https://llmshot.vercel.app (Eval Gates dataset).Create an .eval-gate.yml in your project root for repeatable threshold configs:
dataset: eval/dataset.json
corpus: eval/corpus.jsonl # v0.5.0 — used by retrieval / RAG eval
output_dir: eval/results
models:
- provider: anthropic
model: claude-sonnet-4-6
input_cost_per_mtok: 3.0
output_cost_per_mtok: 15.0
- provider: openai
model: gpt-4o-mini
input_cost_per_mtok: 0.15
output_cost_per_mtok: 0.60
judge:
provider: openai
model: gpt-4o-mini
temperature: 0
retrieval: # v0.5.0
adapter: bm25
k: 5
thresholds:
# generation
avg_faithfulness: 0.80
avg_relevance: 0.75
p95_ttft_ms: 500
max_cost_per_query: 0.01
# retrieval (v0.5.0)
avg_recall_at_k: 0.75
avg_precision_at_k: 0.50
avg_mrr: 0.70
avg_ndcg_at_k: 0.75
p95_retrieval_latency_ms: 50
# RAG (v0.5.0)
avg_context_relevance: 0.70
avg_citation_faithfulness: 0.80All v0.5.0 keys are optional. Existing v0.4.x configs continue to load and run unchanged — missing thresholds simply skip their check.
The evaluation dataset is a JSON array of entries:
[
{
"id": "unique-id",
"category": "factual",
"context": "The system prompt / context provided to the model",
"question": "The question asked",
"expected_response": "Reference answer for the judge to compare against",
"tags": ["optional", "tags"]
}
]Required fields: id, category, context, question, expected_response. The tags field is optional.
v0.5.0 adds retrieval-only and end-to-end RAG evaluation alongside the v0.4.x generation eval. Datasets, configs, and tools from earlier versions continue to work unchanged.
One entry per line. Add relevant_chunk_ids to mark the ground-truth chunks for each question:
{"id": "r-001", "category": "factual", "context": "Answer questions about JWST.", "question": "When did the James Webb Space Telescope launch?", "expected_response": "December 25, 2021.", "relevant_chunk_ids": ["sp-001"]}
{"id": "r-002", "category": "factual", "context": "Answer questions about JWST.", "question": "Where does JWST orbit and how big is its primary mirror?", "expected_response": "L2 Lagrange point; 6.5 metre primary mirror.", "relevant_chunk_ids": ["sp-002", "sp-004"]}Entries without relevant_chunk_ids are skipped (with a stderr warning) by the retrieval and RAG commands.
One chunk per line:
{"chunk_id": "sp-001", "content": "The James Webb Space Telescope launched on December 25, 2021 from French Guiana on an Ariane 5 rocket.", "metadata": {"topic": "space"}}
{"chunk_id": "sp-002", "content": "JWST orbits the Sun at the L2 Lagrange point, approximately 1.5 million kilometres from Earth.", "metadata": {"topic": "space"}}
{"chunk_id": "sp-004", "content": "The primary mirror of JWST is 6.5 metres across and is made of 18 hexagonal beryllium segments coated in gold.", "metadata": {"topic": "space"}}chunk_id and content are required; metadata is an optional dict.
mcp-llm-eval evaluate-retrieval \
--dataset eval/retrieval_dataset.jsonl \
--corpus eval/retrieval_corpus.jsonl \
--k 5 \
--output-dir eval/resultsRetrieval evaluation: 20260425_120000
Queries: 4 | Errors: 0 | Adapter: bm25 | k=5
Recall@k Precision@k MRR nDCG@k p50 ms p95 ms
----------------------------------------------------------------
0.8750 0.4500 0.9167 0.8431 3.2 7.8Pass --config .eval-gate.yml to enforce thresholds post-run (exit code 1 on failure).
Shorthand form (one or more --model provider:model flags, repeatable):
mcp-llm-eval evaluate-rag \
--dataset eval/retrieval_dataset.jsonl \
--corpus eval/retrieval_corpus.jsonl \
--k 5 \
--model openai:gpt-4o-mini \
--model anthropic:claude-sonnet-4-6 \
--output-dir eval/resultsOr load models, judge, retrieval, and thresholds from .eval-gate.yml:
mcp-llm-eval evaluate-rag \
--dataset eval/retrieval_dataset.jsonl \
--corpus eval/retrieval_corpus.jsonl \
--config .eval-gate.ymlCLI --model flags fully override the config's models: list (no merging). Files written: {timestamp}_rag_summary.json, {timestamp}_rag_benchmark.json, latest_rag_summary.json.
Tool | Purpose |
| Run retrieval metrics against a labelled dataset; returns per-query metrics, aggregate, p50/p95 latency |
| Retrieve + generate + judge in one call; returns per-(query, model) results plus per-model aggregates |
| Compare two saved retrieval/RAG result files; flags metrics that regressed beyond tolerance |
| Reserved stub; schema is stable today and returns a not-implemented response |
v0.5.0 shipped an in-memory BM25Adapter (via rank_bm25); v0.7.0 adds three embedding-based adapters. All implement the same RetrievalAdapter Protocol — a single sync method retrieve(query, k) -> list[RetrievedChunk] — so they're interchangeable behind --adapter and the eval-gate thresholds.
Adapter | Backing model | Cost | Notes |
|
| $0 | Lexical keyword match. Deterministic, model-agnostic, fits unit tests. |
|
| ~$0.02/1M | Cheap dense vectors; corpus embeddings cached to |
|
| ~$0.13/1M | Higher-quality dense vectors; same cache layout as |
|
| ~$0.15/1M | Google's first-party embeddings. Batches automatically (100 inputs / req). |
Embedding adapters lazy-import openai / google-genai / numpy. Install via pip install "mcp-llm-eval[embeddings]" to pull all three. Plug your own (Azure AI Search, OpenSearch, Pinecone) by subclassing the protocol; no schema or threshold changes required.
The judge model is resolved in this order: explicit --judge-model CLI flag → MCP_LLM_EVAL_JUDGE_MODEL environment variable → built-in default gpt-4o-mini. All judge calls run at temperature=0.
The v0.5.0 judges (context_relevance, citation_faithfulness) prompt for an integer 1-5 score with anchor descriptions and normalise to a 0-1 float internally via (score - 1) / 4. Public APIs, eval-gate thresholds, and saved JSON all use the 0-1 form — the integer pipeline is an implementation detail for measurement reliability.
Connect to Claude Desktop or any MCP-compatible agent. The agent calls tools directly — run evals, check thresholds, browse past runs, compare runs, and generate PR comments.
The same mcp-llm-eval binary doubles as a CLI for CI/CD pipelines:
# Run a full evaluation
mcp-llm-eval run --config .eval-gate.yml --dataset eval/dataset.json --output-dir eval/results
# Check thresholds (exit code 1 on failure — blocks PRs)
mcp-llm-eval check --results eval/results/latest_summary.json --config .eval-gate.yml
# Compare against baseline (exit code 1 on regression)
mcp-llm-eval compare --baseline eval/results/main_summary.json --current eval/results/pr_summary.json
# Generate PR comment markdown
mcp-llm-eval comment --summary eval/results/latest_summary.json --config .eval-gate.yml --output pr-comment.md
# Run retrieval-only evaluation (v0.5.0)
mcp-llm-eval evaluate-retrieval --dataset eval/retrieval_dataset.jsonl --corpus eval/retrieval_corpus.jsonl --k 5 --output-dir eval/results
# Run end-to-end RAG evaluation (v0.5.0)
mcp-llm-eval evaluate-rag --dataset eval/retrieval_dataset.jsonl --corpus eval/retrieval_corpus.jsonl --k 5 --model openai:gpt-4o-mini --output-dir eval/resultsname: LLM Eval Gate
on:
pull_request:
jobs:
eval:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- run: pip install mcp-llm-eval anthropic openai google-genai
- run: mcp-llm-eval run --config .eval-gate.yml --dataset eval/dataset.json --output-dir eval/results
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
- run: mcp-llm-eval check --results eval/results/latest_summary.json --config .eval-gate.yml
- run: |
mcp-llm-eval comment --summary eval/results/latest_summary.json --config .eval-gate.yml --output pr-comment.md
gh pr comment ${{ github.event.number }} --body-file pr-comment.md
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}mcp-llm-eval's own dataset (eval/dataset.json) dogfoods the
evaluation engine across 8 models, 20 questions, 3 categories
(factual, reasoning, summarization). The results feed into
LLMShot as the Eval Gates benchmark.
Create a .env file in the project root with API keys for all
providers:
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...
GOOGLE_API_KEY=AIza...Then run:
make benchmark # Run eval against all 8 models
make benchmark-copy # Copy results to llm-benchmarks repoResults are written to eval/results/ (gitignored). The benchmark
output feeds into LLMShot via the
llm-benchmarks repo
at text-generation/eval-gates-summary.json and
text-generation/eval-gates-benchmark.json.
Server not appearing in Claude Desktop
Ensure Claude Desktop is fully restarted (quit with Cmd+Q, not just close the window).
Check your config JSON is valid — a trailing comma or typo will silently break it.
Use absolute paths if uvx or mcp-llm-eval aren't found.
"Provider SDK not installed" errors
Provider SDKs are optional. Install the ones you need:
pip install anthropic openai google-genai"Dataset file not found" errors
Use the full absolute path to your dataset file, not a relative path.
Judge scoring fails
The default judge uses OpenAI's gpt-4o-mini. Make sure the openai package is installed and OPENAI_API_KEY is set in your environment.
This is Claude Desktop only
MCP servers work with the Claude Desktop app, not claude.ai in your browser.
# Clone and set up
git clone https://github.com/berkayildi/mcp-llm-eval.git
cd mcp-llm-eval
make setup
# Run tests
make test
# Build distribution
make build
# Run the server locally (stdio)
make start
# Clean everything
make cleanMIT © Berkay Yildirim
Compare two retrieval evaluation result files and detect drift. Flags metrics that have regressed beyond configurable tolerance. Takes two result-set paths; does not persist history itself.
| Name | Required | Description | Default |
|---|---|---|---|
| baseline_path | Yes | ||
| current_path | Yes | ||
| tolerance | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden. It discloses that the tool compares files, detects drift, and does not persist history, implying no side effects. However, it does not mention error handling or output format, leaving some behavioral aspects unspecified.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three short, front-loaded sentences with no fluff. Every sentence adds value: purpose, behavior, and constraints. Perfectly concise.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, the description should hint at the return format (e.g., list of regressed metrics). It also assumes knowledge of 'retrieval evaluation result files' and does not elaborate on the complex tolerance object, leaving gaps for new users.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, yet the description only vaguely mentions 'configurable tolerance' and 'two result-set paths'. It does not explain each tolerance parameter or state that defaults exist, failing to compensate for the schema's lack of descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it compares two files and detects drift, with specific verb 'compare' and resource 'retrieval evaluation result files'. It also distinguishes from persistent tools by noting it does not persist history.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for comparing two result sets and mentions it does not persist history, but does not explicitly state when to prefer this over siblings like 'compare_runs' or 'evaluate_retrieval'. Absence of when-not-to-use guidance reduces clarity.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Check evaluation results against quality gate thresholds. Returns pass/fail per metric and overall gate status.
| Name | Required | Description | Default |
|---|---|---|---|
| results_path | Yes | Path to an evaluation results JSON file (summary). | |
| thresholds | Yes | Quality gate thresholds. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It discloses that it returns pass/fail per metric and overall gate status, implying a read-only check, but does not explicitly state side effects or auth requirements.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, concise sentence with no wasted words. It is front-loaded with the action and resource.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers the action and basic return format, but does not explain what 'overall gate status' entails or provide any example. Given no output schema, more detail on the return structure would improve completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% for both parameters, so the description adds no extra meaning beyond the schema. The description's context of 'checking against thresholds' is implicit in the schema's 'Quality gate thresholds' description.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it checks evaluation results against quality gate thresholds and returns pass/fail per metric and overall status. It uses a specific verb and resource, but does not differentiate from sibling tools like check_retrieval_drift or evaluate_rag_end_to_end, which also evaluate different aspects.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when to use this tool versus alternatives. It implies usage when evaluation results are available, but lacks exclusions or references to siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Compare two evaluation runs and detect regressions. Flags metrics that worsened beyond configurable tolerance.
| Name | Required | Description | Default |
|---|---|---|---|
| baseline_path | Yes | Path to baseline evaluation summary JSON. | |
| current_path | Yes | Path to current evaluation summary JSON. | |
| tolerance | No | Per-metric regression tolerance. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description should disclose behavioral traits like side effects or permissions. It only states the tool flags regressions but does not specify whether it modifies data, requires special access, or how results are presented.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One concise sentence that is front-loaded with the core action. Every word is necessary and no fluff.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has 3 parameters, a nested object, and no output schema. The description does not explain what the tool returns (e.g., list of flags, report object), leaving a significant gap for the agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3. The description adds the concept of 'configurable tolerance' which aligns with the tolerance parameter, but does not provide additional meaning beyond the schema's own descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool compares two evaluation runs and detects regressions, using specific verbs and resource. It distinguishes from siblings like check_retrieval_drift by focusing on pairwise run comparison.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when to use this tool versus alternatives. The description implies its use for comparing two runs but lacks exclusion criteria or mention of sibling tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run the full RAG pipeline: retrieve chunks, generate answers using the retrieved chunks as context, and score with context_relevance and citation_faithfulness judges. Returns retrieval metrics, generation metrics, and judge scores per query, plus an aggregate.
| Name | Required | Description | Default |
|---|---|---|---|
| dataset_path | Yes | ||
| corpus_path | Yes | ||
| models | Yes | Models to evaluate. | |
| k | No | ||
| adapter | No | bm25 | |
| judge | No | ||
| output_dir | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It does not disclose side effects, authorization needs, rate limits, or whether results are persisted. The output_dir parameter suggests potential file writes but is unmentioned.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two clear sentences cover the core actions and output. No redundant information. Front-loaded with process then outputs.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
High-level process and output are described, but given 7 parameters, nested objects, and no output schema or annotations, more detail on configuration and return format would be needed for full completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is only 14%, with only models having a description. The description adds no parameter-level details, leaving 6 of 7 parameters unexplained. It does not compensate for the low schema coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool runs the full RAG pipeline, including retrieval, generation, and scoring with specific judges. It distinguishes itself from siblings like evaluate_retrieval by explicitly mentioning the end-to-end process and named judges.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for end-to-end evaluation but does not provide explicit guidance on when to use this tool versus alternatives like evaluate_retrieval. No when-not or conditions are given.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run retrieval metrics (recall@k, precision@k, MRR, nDCG@k) against a labelled dataset with a configurable retrieval adapter. Returns per-query metrics, dataset-level aggregate, and p50/p95 retrieval latency.
| Name | Required | Description | Default |
|---|---|---|---|
| dataset_path | Yes | Path to JSONL dataset with relevant_chunk_ids on each entry. | |
| corpus_path | Yes | Path to JSONL corpus file. | |
| k | No | Top-k cutoff for all metrics (default 5). | |
| adapter | No | Retrieval adapter to use (default bm25). | bm25 |
| output_dir | No | Directory to save results (optional). |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Without annotations, the description adds some behavioral context (returns metrics, latency), but does not disclose side effects, permissions, or safety properties. A higher score would require explicit read-only or destructive hints.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two crisp sentences, front-loaded with the tool's purpose and output summary. No unnecessary words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Covers core metrics and outputs (per-query, aggregate, latency). Missing details on JSONL format expectations beyond 'relevant_chunk_ids', but sufficient for a run tool without output schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, baseline 3. The description adds context about 'relevant_chunk_ids' and configurable adapter but does not significantly deepen understanding beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description explicitly states running specific retrieval metrics (recall@k, precision@k, MRR, nDCG@k) against a labelled dataset, clearly distinguishing it from sibling tools like compare_runs or evaluate_rag_end_to_end.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when to use versus alternatives. The description implies usage for retrieval evaluation but does not define prerequisites or when to avoid this tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Generate a markdown PR comment from evaluation results. Includes results table, regression details, and threshold status.
| Name | Required | Description | Default |
|---|---|---|---|
| summary_path | Yes | Path to evaluation summary JSON. | |
| comparison_path | No | Path to compare_runs output JSON (optional). | |
| thresholds | No | Quality gate thresholds for pass/fail badges. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Describes outputs (results table, regression details, threshold status) but lacks details on side effects, permissions, or return format. With no annotations, the description carries full burden; some behavioral info is present but insufficient.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Single sentence that is direct and informative. No redundant information, front-loaded with key action and resource.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Sufficient for the tool's purpose given schema coverage, but lacks explanation of return value or behavior when optional parameters are omitted. With no output schema, more detail on what the function returns would improve completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with clear parameter descriptions. Tool description adds context about content of generated comment but doesn't clarify parameter usage beyond schema. Baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Clearly states the tool generates a markdown PR comment from evaluation results, listing key components like results table and threshold status. Distinct from sibling tools which focus on evaluation metrics or comparisons.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Implied use case for generating PR comments from evaluation outputs, but no explicit guidance on when to use vs siblings or when not to use. Does not exclude other use cases or provide context for selection.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Retrieve the full details of a specific evaluation run: per-question per-model scores, responses, and judge reasoning.
| Name | Required | Description | Default |
|---|---|---|---|
| results_path | Yes | Path to a specific evaluation result file. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It indicates a read operation but does not disclose potential side effects, permissions, or output format. It is not contradictory but could be more transparent about what the agent can expect.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, well-structured sentence that front-loads the action and delivers specific details without any waste. Every word adds value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple retrieval tool with one parameter and no output schema, the description adequately explains what the tool returns. It could optionally mention return format, but the described contents ('scores, responses, reasoning') provide sufficient context for an agent to decide to invoke it.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% for the single parameter 'results_path'. The description adds detail about what the tool returns (scores, responses, reasoning) but does not add new meaning to the parameter itself beyond the schema's description of it as a file path.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it retrieves full details of an evaluation run, including specific contents like per-question per-model scores, responses, and judge reasoning. It distinguishes itself from siblings like 'list_evaluations' (which lists runs) and 'run_evaluation' (which creates runs).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies use when you need detailed information about a specific evaluation run, but it does not provide explicit when-to-use or when-not-to-use guidance, nor does it mention alternatives like 'compare_runs' or 'list_evaluations' for different purposes.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
List past evaluation runs in a directory. Returns metadata for each run: timestamp, dataset, models, pass/fail, and cost.
| Name | Required | Description | Default |
|---|---|---|---|
| results_dir | Yes | Directory containing evaluation result files. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description bears the full burden. It correctly indicates a read operation and lists returned metadata, but omits details on error handling, permissions, pagination, or default behavior (e.g., ordering). No contradictions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loaded with the core action, and efficiently lists return fields. Every word adds value, no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple list tool with one parameter, the description covers the purpose, return fields, and required input. It could mention ordering or limits, but is complete enough for selection and basic use. Output schema is absent but described inline.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already describes 'results_dir' with 100% coverage. The description only reiterates that it lists runs 'in a directory', adding no new semantic information beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'List', the resource 'past evaluation runs', and the scope 'in a directory'. It distinguishes itself from siblings like 'run_evaluation' (creates) and 'compare_runs' (comparison) by focusing on listing historical runs.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description does not explicitly state when to use this tool versus alternatives. It implies usage for browsing historical runs by listing returned fields, but lacks guidance on when not to use it or how it differs from 'get_evaluation' or 'compare_runs'.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run an LLM evaluation: load a dataset, query models via streaming, score responses with an LLM-as-judge, and return per-question scores, aggregate summary, and pass/fail status.
| Name | Required | Description | Default |
|---|---|---|---|
| dataset_path | Yes | Path to the JSON evaluation dataset file. | |
| models | Yes | Models to evaluate. | |
| judge | No | Judge configuration (optional). | |
| output_dir | No | Directory to save results JSON files. | |
| tracing | No | Optional tracing configuration. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description discloses streaming queries and LLM-as-judge scoring, but does not mention side effects like writing results to output_dir, API costs, time consumption, or potential for partial results. Without annotations, these gaps reduce transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single sentence that is front-loaded with the core action. It efficiently lists the main steps and outputs. While it could be split for readability, it contains no fluff and earns its length.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers outputs (scores, summary, pass/fail) but omits operational context: it doesn't explain that results are saved to output_dir, the expected dataset format, or error handling. For a tool with 5 params, nested objects, and no output schema, these gaps are notable.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the schema already documents all 5 parameters. The description adds process context ('load a dataset, query models via streaming') but no significant parameter-level detail beyond what the schema provides. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it runs an LLM evaluation, lists the steps (load dataset, query models, score responses), and specifies outputs (per-question scores, summary, pass/fail). This differentiates it from sibling tools like evaluate_retrieval or check_retrieval_drift that focus on specific aspects.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool vs alternatives (e.g., when to use evaluate_rag_end_to_end or check_thresholds). The description does not provide context, prerequisites, or exclusions, leaving the agent to infer usage without clear boundaries.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
[STUB - not implemented in v0.5.0] Inject poisoned chunks into a corpus and re-run retrieval evaluation. Returns a clear not-implemented response.
| Name | Required | Description | Default |
|---|---|---|---|
| dataset_path | Yes | ||
| corpus_path | Yes | ||
| poisoning_strategy | Yes | ||
| poison_ratio | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description explicitly declares it is a stub not implemented in v0.5.0 and that it returns a clear not-implemented response. This is transparent about current behavior, though no annotations exist to contradict or supplement.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence with a stub note, front-loaded with the core action and a clear statement of current behavior. No unnecessary words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema and four undocumented parameters, the description fails to provide enough context for an agent to know how to invoke the tool correctly. Essential parameter details are missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0% and the description provides no information about the four parameters (e.g., dataset_path, corpus_path, poisoning_strategy, poison_ratio), leaving their meaning and usage entirely undocumented.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it injects poisoned chunks into a corpus and re-runs retrieval evaluation, which is specific and distinguishes it from sibling tools that focus on general evaluation or comparison.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool vs alternatives, nor any prerequisites or exclusions. The stub status is mentioned but not as usage advice.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
v0.9.2check_retrieval_driftcheck_thresholdscompare_runsevaluate_rag_end_to_endevaluate_retrievalformat_pr_commentget_evaluationlist_evaluationsrun_evaluationsimulate_poisoned_corpusMost tools have distinct purposes, but check_retrieval_drift and compare_runs both detect regressions, causing slight overlap. Descriptions help clarify, but an agent might confuse them.
Most tools use verb_noun pattern (e.g., check_thresholds, evaluate_retrieval), but evaluate_rag_end_to_end is more verbose and deviates slightly. Overall consistent.
10 tools is well-scoped for an evaluation server, covering retrieval, generation, comparison, and reporting without being overwhelming.
The set covers core evaluation workflows (retrieval, RAG, comparison, thresholds, reporting). Missing delete or batch management, but that's acceptable. The stub tool indicates a planned feature not yet implemented.
Unclaimed servers have limited discoverability.
If you are the server author, to access and configure the admin panel.
MCP server providing access to the Scorecard API to evaluate and optimize LLM systems.
MCP server for building and testing AI agents with multi-model experimentation and insights.
MCP server for AI agents to plan, verify, and deploy Cloudflare-native apps.
MCP server for secureFlows: token-free URL builders and integration-linting tools for AI agents.
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/berkayildi/mcp-llm-eval'
If you have feedback or need assistance with the MCP directory API, please join our Discord server
