Skip to main content
Glama

Santismm Knowledge — Harness Engineering, Agentic AI & Governance

Calculate an agent evaluation sample size

calculate_evaluation_sample_size
Read-onlyIdempotent

Calculate two different samples: how many independent evaluations are needed to detect at least one failure, and how many are needed to estimate its rate at a chosen margin. Use this when a user asks how many tests are enough; do not interpret zero observed failures as proof of zero risk.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
localeNoLanguage for interpretations, assumptions, formulas and warnings. The Labs execution service currently resolves fr/de/ja/zh to English and reports that fallback.
marginYesMargin for estimating the failure rate, in percentage points.
confidenceYesConfidence level, in percent.
populationYesNumber of distinct evaluable cases.
failureRateYesFailure rate to detect, in percent.

Output Schema

TableJSON Schema
NameRequiredDescriptionDefault
slugYes
unitsYes
inputsYes
sourceYes
api_urlYes
licenseYes
resultsYes
updatedYes
versionYes
fallbackYes
formulasYes
languageYes
warningsYes
assumptionsYes
canonical_urlYesCite this URL.
interpretationYes
schema_versionYes
methodology_urlYes
resolved_localeYes
requested_localeYes

Schema Changelog

Changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. First observed

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already establish readOnly, openWorld, idempotent, and destructiveHint=false, so the description does not need to cover mutation safety. It adds value beyond annotations by warning against interpreting zero observed failures as proof of zero risk and clarifying that the tool returns two separate sample-size calculations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences with no filler: the first states what the tool computes, and the second gives the use trigger plus a critical statistical warning. It is front-loaded with the most decision-relevant information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the rich input schema, clear annotations, and the presence of an output schema, the description covers what the tool does, when to invoke it, and the most important interpretation pitfall. Nothing essential is missing for correct selection and invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, and every parameter has its own description with constraints or enums, so the schema carries the semantic weight. The description references failure rate, margin, and the estimate context at a high level but does not need to duplicate the schema's parameter details.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific verb and resource ('Calculate two different samples' for agent evaluation sample size) and distinguishes two concrete outputs: detecting at least one failure versus estimating the failure rate at a chosen margin. This level of specificity separates it from the sibling calculate tool (calculate_agent_economics) even without naming it.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly states when to use the tool: 'when a user asks how many tests are enough.' It also supplies a key interpretive caveat about zero observed failures. It does not formally state when not to use it or name an alternative sibling, but the trigger condition is clear and actionable.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

TDQS

A4.1/5.0
Disambiguation4/5

Most tools are cleanly separated by content type and the list_/get_ pairs are predictable. The main ambiguity is among search, search_all, and search_articles: search claims to cover the 'whole corpus' while search_all actually expands to essays, labs, claims, and the Homeric Atlas, so an agent could select the narrower search and miss content.

Naming Consistency5/5

Every tool follows the same snake_case verb_noun pattern: calculate_*, get_*, list_*, and search_*. Even the three search variants are predictable from their suffixes, so there are no mixed naming conventions.

Tool Count2/5

30 tools is above the preferred MCP size and creates real selection burden for agents, even though the multi-surface knowledge scope explains the volume. The set is systematic rather than bloated, but 25+ tools is still too many for a typical server surface.

Completeness4/5

Each content surface has browse, fetch, and search coverage, and get_related plus get_overview provide cross-cutting navigation. The only meaningful gap is that the relationship between search and search_all is not fully disjoint, which can create a dead-end if the wrong search tool is chosen first.