Skip to main content
Glama
coval-ai

Coval MCP Server

Official
by coval-ai

Coval MCP Server

npm version

The official Model Context Protocol server for Coval - the AI evaluation platform.

This MCP server allows AI assistants like Claude, ChatGPT, Codex, and Cursor to interact with Coval's evaluation APIs, enabling you to:

  • Launch and monitor evaluation runs

  • Manage AI agents and test sets

  • Retrieve evaluation metrics and results

  • Inspect saved reports and recurring evaluation schedules

Hosted connector

The hosted connector is the recommended way to use Coval from Claude, ChatGPT, and other remote MCP clients. It uses OAuth, so users sign in to Coval and select the organization the connector can access. No Coval API key is copied into the client.

Standard MCP endpoint: https://mcp.coval.dev/mcp

Claude hosted endpoint: https://mcp.coval.dev/claude/mcp

Claude

  1. Open Customize > Connectors. Team and Enterprise owners add the connector first from Organization settings > Connectors.

  2. Add a custom web connector using https://mcp.coval.dev/claude/mcp.

  3. Select Connect, sign in to Coval, and choose the Coval organization to authorize.

  4. Enable the connector in a conversation and ask Claude to use Coval or Sofia.

See Claude's remote MCP connector guide for current plan and administration requirements.

ChatGPT

  1. Enable developer mode for your ChatGPT workspace account.

  2. From Settings > Apps, create an app using https://mcp.coval.dev/mcp and OAuth.

  3. Select Scan Tools, sign in to Coval, and choose the Coval organization to authorize.

  4. Create the app, then enable it in a conversation.

See ChatGPT's MCP app guide for current plan and workspace requirements.

Codex

Use Codex 0.144.2 or newer. Current Codex Desktop releases include a compatible client. For the CLI, update before connecting:

npm install -g @openai/codex@latest
codex mcp add coval --url https://mcp.coval.dev/mcp

When the browser opens, sign in to Coval, select the organization Codex should access, and select Allow. Codex Desktop, the Codex CLI, and the IDE extension share MCP configuration on the same machine, so the connection is available across those clients after they restart.

In Codex Desktop, you can also add Coval from Settings > MCP servers as a Streamable HTTP server using https://mcp.coval.dev/mcp, then select Authenticate.

If OAuth returns invalid_scope, first confirm codex --version is 0.144.2 or newer. Older clients registered a dynamic OAuth client without all the scopes they requested during consent. After updating, clear the stale registration and reconnect:

codex mcp logout coval
codex mcp remove coval
codex mcp add coval --url https://mcp.coval.dev/mcp

No API key, manual scope list, or oauth_resource override is required. See Codex's MCP documentation for current client setup details.

Related MCP server: VerifyAX MCP

Local installation

npx @coval/mcp-server

Use the local stdio server for service accounts, automation, and clients that do not support remote OAuth.

Claude Desktop

Add to ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "coval": {
      "command": "npx",
      "args": ["-y", "@coval/mcp-server"],
      "env": {
        "COVAL_API_KEY": "your_api_key_here"
      }
    }
  }
}

Cursor

Add to .cursor/mcp.json in your project:

{
  "mcpServers": {
    "coval": {
      "command": "npx",
      "args": ["-y", "@coval/mcp-server"],
      "env": {
        "COVAL_API_KEY": "your_api_key_here"
      }
    }
  }
}

Get your API key from app.coval.dev/settings

Available Tools

Tool

Description

list_agents

List all agents in your workspace

get_agent

Get details of a specific agent

create_agent

Create an agent configuration

update_agent

Update an agent configuration

list_runs

List evaluation runs

get_run

Get details of a specific run

create_run

Start a new evaluation run

list_test_sets

List available test sets

get_test_set

Get test set details

create_test_set

Create a test set

list_test_cases

List test cases in a test set

get_test_case

Get test case details

create_test_case

Add a test case to a test set

update_test_case

Update a test case

list_metrics

List evaluation metrics

get_metric

Get metric details

list_personas

List available personas

get_persona

Get persona details

list_reports

List saved evaluation reports

get_report

Get report details and a bounded page of result rows

create_report

Create an organization-private saved report

list_run_templates

List reusable evaluation configurations

list_scheduled_runs

List recurring evaluation schedules

get_scheduled_run

Get a schedule and paginated recent-run history

create_scheduled_run

Create a schedule, disabled by default

update_scheduled_run

Update selected schedule fields

consult_sofia

Delegate a read-only Coval evaluation question to Sofia

Example Usage

Once connected, you can ask Claude things like:

"Show me my recent evaluation runs" "List all my agents" "Run an evaluation of my customer-support-agent against the billing-inquiries test set" "What are the metrics for run abc123?" "Show the first 20 rows of my regression report for the task-success metric" "Create a private report comparing these completed runs by agent" "Show my enabled evaluation schedules and the latest runs from the nightly regression" "Create a disabled weekday schedule from my regression run template" "Use consult_sofia to analyze my latest failed run and recommend the most useful next test."

The same tools, including consult_sofia, are available through every supported endpoint and transport:

  • Standard remote Streamable HTTP: https://mcp.coval.dev/mcp using OAuth. This is the recommended connection for ChatGPT, Codex, and other hosted MCP clients.

  • Claude remote Streamable HTTP: https://mcp.coval.dev/claude/mcp using the same OAuth flow and tool surface, with Claude-specific risk annotations.

  • Local stdio: npx @coval/mcp-server with COVAL_API_KEY, for service accounts and local development.

The hosted connector can access only the Coval organization selected during OAuth consent. Remove the connector from the client or revoke its Coval access when it is no longer needed.

Development

# Install dependencies
npm install

# Build
npm run build

# Test locally with MCP Inspector
npm run inspector

# Run tests
npm test

# Check the public health and OAuth discovery endpoints
npm run check:remote

Environment Variables

Variable

Required

Default

Description

COVAL_API_KEY

Stdio

-

Coval API key for the local stdio transport

COVAL_API_BASE_URL

No

https://api.coval.dev/v1

API base URL

SOFIA_DELEGATION_ORIGIN

No

Derived from COVAL_API_BASE_URL

Overrides the expected Sofia origin used to validate delegation URLs

LOG_LEVEL

No

info

Logging level

MCP_ALLOWED_ORIGINS

No

Claude and OpenAI web origins

Comma-separated exact browser origins allowed to call /mcp or /claude/mcp; clients that omit Origin remain supported

OPENAI_APPS_CHALLENGE

No

-

OpenAI plugin-portal domain verification token served as plain text from /.well-known/openai-apps-challenge

DD_ENV

No

local

Deployment environment included in structured hosted-request logs

DD_SERVICE

No

coval-mcp-server

Service name included in structured hosted-request logs

DD_VERSION

No

COVAL_MCP_SOURCE_SHA or unknown

Immutable release identifier included in structured hosted-request logs

Documentation

License

MIT

Support

Available Tools

18 tools
create_agentB

Create a new agent configuration. Specify the model type (voice, chat, SMS, websocket) and connection details.

ParametersJSON Schema
NameRequiredDescriptionDefault
display_nameYesHuman-readable name for the agent
model_typeYesType of agent: MODEL_TYPE_VOICE, MODEL_TYPE_CHAT, MODEL_TYPE_SMS, etc.
phone_numberNoPhone number for voice agents (E.164 format)
endpointNoWebhook or WebSocket endpoint URL
promptNoSystem prompt or instructions for the agent
metadataNoCustom metadata for the agent

TDQS

B3.2/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided. The description does not disclose behavioral traits such as side effects, authentication requirements, or rate limits. It merely restates the obvious creation action without additional context.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two short sentences, front-loaded with the action. Every word is necessary and there is no extraneous information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the complexity (6 parameters, nested objects, no output schema), the description is too brief. It does not explain return values, the relationship between parameters (e.g., phone_number only for voice), or prerequisites like authentication. This leaves significant gaps for an agent to use correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has 100% description coverage for all 6 parameters. The description adds a high-level summary mentioning 'model type' and 'connection details', but does not provide deeper semantics beyond what the schema already offers. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb 'Create' and the resource 'agent configuration'. It lists specific attributes like model type and connection details, making it distinct from sibling tools that operate on runs, test cases, etc.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided on when to use this tool versus other create tools (e.g., create_run, create_test_case). The description simply states what it does without any context or exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

create_runA

Launch evaluation: agent + persona + test_set. Optionally add tags for filtering. Poll get_run until status=COMPLETED to see metrics.

ParametersJSON Schema
NameRequiredDescriptionDefault
agent_idYesThe unique ID of the agent to evaluate. Get this from list_agents.
persona_idYesThe unique ID of the persona to use. Get this from list_personas.
test_set_idYesThe unique ID of the test set to run against. Get this from list_test_sets.
metric_idsNoOptional list of metric IDs to evaluate. Uses agent defaults if omitted.
optionsNoRun configuration options
tagsNoTags for categorizing and filtering the run (max 20 tags, 200 chars each). Filter later with list_runs filter='tag="regression"'.
metadataNoCustom metadata for tracking purposes

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Without annotations, the description reveals that the tool launches an asynchronous process and advises polling get_run for completion. This provides behavioral transparency beyond just 'create a run', though it could mention resource limits or prerequisites.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences, front-loading the core purpose and then adding key usage hints. No redundant or unnecessary text.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (7 params, nested objects, no output schema), the description covers the essential action and follow-up (polling). It could elaborate on the return value or error conditions, but it's fairly complete for a launch tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema covers all parameters with descriptions (100% coverage), so the description adds minimal extra meaning. The note about tags for filtering is already in the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool launches an evaluation combining agent, persona, and test_set. It distinguishes itself from sibling tools like create_agent or create_test_set by focusing on the run action.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives implicit guidance by mentioning polling get_run for results, but it does not explicitly state when to use this tool versus alternatives or when not to use it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

create_test_caseA

Create test case in a test set. input_str: single scenario message OR JSON array [{role,content},...] for multi-turn conversations.

ParametersJSON Schema
NameRequiredDescriptionDefault
test_set_idYesThe 8-character ID of the test set to add this test case to.
input_strYesThe test input or scenario that will be presented to the agent.
expected_behaviorsNoList of expected agent behaviors or responses for evaluation.
descriptionNoHuman-readable description of what this test case validates.
simulation_metadata_inputNoAdditional context passed to the simulation environment.
metric_inputNoCustom inputs for metric evaluation.
user_notesNoInternal notes about this test case.

TDQS

A3.6/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description must disclose behavioral traits. It reveals that input_str can be a string or JSON array, but fails to mention whether the operation is destructive, requires specific permissions, or has rate limits. It does not indicate if the tool returns a value or has side effects.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise: one sentence for purpose, one phrase for parameter detail. No unnecessary words, and the key information (two input formats) is front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given 7 parameters (including nested objects), no output schema, and no annotations, the description is insufficient. It does not explain the role of simulation_metadata_input, metric_input, or what the tool returns after creation. The agent will lack critical context for proper use.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so baseline is 3. The description adds value by explaining the dual format of input_str, which is not obvious from the schema. For other parameters, it largely echoes the schema, but the additional context for input_str justifies a 4.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states 'Create test case in a test set.' It differentiates from siblings like create_test_set by specifying the operation on a test case. It also adds detail on input_str formats, making it specific and actionable.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description does not provide guidance on when to use this tool versus alternatives, nor does it mention prerequisites or when not to use it. While the purpose is clear, usage context is minimal.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

create_test_setA

Create a test set to organize test cases. After creating, use create_test_case to add scenarios.

ParametersJSON Schema
NameRequiredDescriptionDefault
display_nameYesHuman-readable name for the test set
slugNoURL-friendly identifier (auto-generated if not provided)
descriptionNoDescription of the test set
test_set_typeNoType of test set: DEFAULT, SCENARIO, TRANSCRIPT, WORKFLOW
test_set_metadataNoAdditional configuration metadata
parametersNoTest case parameterization (e.g., {"name": ["Alice", "Bob"]})

TDQS

A4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so the description carries full burden. It explains the creation behavior and the next step, but lacks details on side effects or permissions. Adequate but not thorough.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two concise sentences, front-loaded with the purpose, followed by a useful next-step instruction. No extraneous information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 6 parameters and no output schema, the description covers the core purpose and usage flow. It could benefit from more parameter usage context, but the schema compensates.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so parameters are well-documented in the schema. The description does not add additional meaning beyond the schema, meeting the baseline.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: creating a test set to organize test cases. It also distinguishes from sibling tools like 'create_test_case' by specifying that test cases are added separately.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides a clear context for use (to organize test cases) and a follow-up action (use create_test_case). It does not explicitly state when not to use it, but the context is sufficient.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_agentA

Get agent config: model_type, phone_number (voice), endpoint (websocket/chat), and display_name.

ParametersJSON Schema
NameRequiredDescriptionDefault
agent_idYesThe unique ID of the agent to retrieve. Get this from list_agents.

TDQS

A3.5/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries full burden for behavioral disclosure. It only lists returned fields, omitting traits like idempotency, error handling, authorization needs, or whether this triggers side effects. For a read tool, this is minimal but still insufficient.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence under 20 words that front-loads the purpose. It earns its place with specific return fields. Slightly under 5 because it could be even more concise, but overall excellent.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple getter with one parameter, no output schema, and a list of siblings, the description is nearly complete. It lists important return fields, helping the agent understand output shape. Could be improved by mentioning null handling or error cases, but sufficient.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and the parameter description already explains 'agent_id' (unique ID from list_agents). The tool description adds no extra meaning beyond what the schema provides, so baseline of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ('Get') and resource ('agent config'), and lists exact return fields (model_type, phone_number, endpoint, display_name). This clearly differentiates it from sibling tools like list_agents (list all) and create_agent (create new).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies use when needing a specific agent's configuration, but lacks explicit guidance on when to use this versus alternative tools like list_agents or other getters (get_persona, get_metric). No exclusions or when-not-to-use are provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_metricA

Get detailed information about a specific metric. Shows metric type, description, and configuration.

ParametersJSON Schema
NameRequiredDescriptionDefault
metric_idYesThe unique ID of the metric to retrieve. Get this from list_metrics.

TDQS

A4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description must fully disclose behavior. It states that the tool 'shows metric type, description, and configuration,' implying read-only access, but does not explicitly confirm no side effects, permissions, or rate limits.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise, consisting of two sentences that convey the core purpose and output without unnecessary verbiage.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity (single parameter, no output schema), the description adequately covers the input requirement and output details (type, description, configuration). It is sufficiently complete for an agent to use correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, providing a baseline of 3. The description adds value by noting that the metric_id comes from list_metrics, which aids in understanding the prerequisite.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Get detailed information about a specific metric.' It distinguishes itself from sibling tools like 'list_metrics' by focusing on a single metric's details.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage when a metric_id is available, and hints at a prerequisite by saying 'Get this from list_metrics,' but does not explicitly state when to use this tool versus alternatives or when not to use it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_personaA

Get persona details: voice_name, language_code (BCP-47), background_sound, persona_prompt (behavior), wait_seconds, conversation_initiation (speak_first/wait_for_user).

ParametersJSON Schema
NameRequiredDescriptionDefault
persona_idYesThe unique ID of the persona to retrieve. Get this from list_personas.

TDQS

A3.6/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden. It states that the tool retrieves details and lists fields, but does not disclose potential side effects, authentication needs, rate limits, or any constraints. For a simple read operation, this is adequate but not thorough.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence efficiently listing the returned fields. It is concise with no extraneous words, though a bulleted list could improve readability. Overall, it earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the low complexity (1 parameter, no output schema, no nested objects), the description is fairly complete by listing the returned fields. However, it does not specify the output format (e.g., JSON object) or any additional context, leaving room for minor ambiguity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% (persona_id is fully described in the schema). The description adds no additional meaning beyond what the schema already provides, so the score defaults to baseline 3.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states 'Get persona details' and enumerates the specific fields returned (voice_name, language_code, etc.). Among sibling tools like get_agent, get_metric, etc., it uniquely identifies the resource as persona details, making the purpose unmistakable.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage when needing persona details but does not explicitly state when to use this tool versus alternatives like list_personas. The schema description hints that list_personas provides the ID, but the tool description itself lacks clear when-to-use and when-not-to-use guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_runA

Get run status/results. Status: PENDING→RUNNING→COMPLETED. Completed runs include metrics (custom per org) and output_ids for transcripts.

ParametersJSON Schema
NameRequiredDescriptionDefault
run_idYesThe unique ID of the run to retrieve. Get this from list_runs.

TDQS

A4.1/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, but the description discloses the status lifecycle and what completed runs include (metrics, output_ids). This helps set expectations for the agent.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, each with clear purpose: first states action, second provides behavioral details. No redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple read tool with one parameter and no output schema, the description adequately explains the response (status, metrics, output_ids).

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The only parameter run_id is fully described in the schema. The tool description adds no additional meaning beyond what the schema provides, so baseline score of 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool retrieves run status and results, with a specific lifecycle (PENDING→RUNNING→COMPLETED). It distinguishes from sibling tools like create_run and list_runs.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No explicit guidance on when to use this tool vs alternatives. The schema mentions getting run_id from list_runs, but no context on when not to use or prerequisites.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_test_caseA

Get test case details: input_str (the scenario), expected_behaviors, and metadata.

ParametersJSON Schema
NameRequiredDescriptionDefault
test_case_idYesThe unique ID of the test case to retrieve.

TDQS

A3.6/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It does not disclose read-only status, idempotency, or any side effects. The description merely lists return fields, which is adequate for a simple retrieval but lacks explicit behavioral details.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence that is concise and front-loaded with the primary purpose. It could be improved with structured formatting, but it is efficient and avoids unnecessary words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description partially explains the return fields (input_str, expected_behaviors, metadata). This provides a basic expectation but does not cover error cases or response format details. It is minimally adequate for a simple retrieval tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The only parameter 'test_case_id' is fully described in the schema with a clear purpose. Schema coverage is 100%, so the baseline is 3. The description does not add additional meaning to the parameter beyond what the schema already provides.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states 'Get test case details' and lists specific fields: input_str, expected_behaviors, and metadata. This verb+resource combination distinguishes it from sibling tools like list_test_cases and create_test_case.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No explicit when-to-use or when-not-to-use guidance is provided. However, the tool name and context imply it should be used when a specific test_case_id is known, rather than listing all cases.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_test_setA

Get test set details: display_name, description, and test case count. Use list_test_cases to see individual scenarios.

ParametersJSON Schema
NameRequiredDescriptionDefault
test_set_idYesThe unique ID of the test set to retrieve. Get this from list_test_sets.

TDQS

A4.1/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided; description indicates it's a read operation but does not explicitly state read-only, permissions, or side effects. Adequate for a simple get but could be more explicit.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two concise sentences, front-loaded with purpose, no extraneous information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple tool with one parameter and no output schema, the description covers what is returned and provides guidance to a sibling. Complete given the simplicity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema has 100% coverage with a clear description for test_set_id. Description adds minimal value beyond schema, only a slight pointer to list_test_sets.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly states the verb (Get), resource (test set), and what details are returned (display_name, description, test case count). It also distinguishes from sibling tool list_test_cases.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides explicit direction to use list_test_cases for individual scenarios, but does not mention other relevant siblings like list_test_sets or update_test_set.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_agentsC

List agents (AI systems to evaluate). Model types: VOICE, OUTBOUND_VOICE, SMS, WEBSOCKET, CHAT. Use agent_id when creating runs.

ParametersJSON Schema
NameRequiredDescriptionDefault
page_sizeNoNumber of results per page (1-100, default 50)
page_tokenNoToken for retrieving the next page of results
order_byNoSort order (e.g., "-create_time" for newest first)
filterNoFilter expression (e.g., status="COMPLETED")

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden for behavioral disclosure. It only mentions model types and a creation hint, but fails to disclose behavior such as pagination (implied by schema but not stated), rate limits, authentication needs, or return format.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three sentences with no wasted words. The purpose is front-loaded, model types are listed efficiently, and a hint is appended. Perfectly concise for the content provided.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no output schema and a list tool with 4 parameters, the description is incomplete. It does not describe the return structure (e.g., list of agent objects), any default ordering, or what happens with empty results. The model types hint is tangential rather than addressing completeness needs.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3. The description adds no parameter-specific details beyond what the schema already provides. The mention of model types is not linked to any parameter, so no added value.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states 'List agents (AI systems to evaluate)' with a specific verb and resource. It adds context about model types, which helps clarify what agents are. However, it does not explicitly differentiate from sibling list tools like list_runs or list_test_cases.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides a tangential hint ('Use agent_id when creating runs') but no explicit guidance on when to use this tool versus alternatives (e.g., get_agent, create_agent) or when not to use it. No usage context is given.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_metricsA

List available evaluation metrics. Metrics define how agent performance is measured. Set include_builtin=true to see built-in metrics.

ParametersJSON Schema
NameRequiredDescriptionDefault
page_sizeNoNumber of results per page (1-100, default 50)
page_tokenNoToken for retrieving the next page of results
order_byNoSort order (e.g., "-create_time" for newest first)
filterNoFilter expression (e.g., status="COMPLETED")
include_builtinNoInclude built-in metrics in results (default false)

TDQS

A3.8/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations exist, and the description fails to disclose behavioral traits such as pagination behavior, default sorting, or what happens when parameters are omitted. The description adds minimal behavioral context beyond the literal listing action.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, directly states purpose and one key condition. No extraneous information. Front-loaded with action and resource.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given 5 parameters, no output schema, and no annotations, the description covers the core purpose and one key parameter. It could mention pagination behavior or default sorting, but the schema compensates. Acceptably complete for its complexity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, meaning each parameter is already documented in the schema. The description adds meaning only to 'include_builtin'. For other parameters, it adds no extra value, meeting the baseline 3.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb 'List' and the resource 'available evaluation metrics'. It differentiates from sibling list tools by specifying 'evaluation metrics', distinct from agents, runs, test cases, etc.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides a specific usage guideline with 'Set include_builtin=true to see built-in metrics'. While it doesn't explicitly mention when not to use or alternative tools, the context and name make it obvious this is for metrics only.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_personasB

List personas (simulated users). Each has voice_name, language_code, background_sound (off/office/crowd/airport/etc), and behavior prompt. Required for runs.

ParametersJSON Schema
NameRequiredDescriptionDefault
page_sizeNoNumber of results per page (1-100, default 50)
page_tokenNoToken for retrieving the next page of results
order_byNoSort order (e.g., "-create_time" for newest first)
filterNoFilter expression (e.g., status="COMPLETED")

TDQS

B3.2/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It mentions what a persona contains and that it's required for runs, but does not disclose behavioral traits like pagination, read-only nature, or any side effects.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise (two sentences) and front-loaded with the action. It efficiently conveys the purpose without wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description mentions key fields of personas and states 'Required for runs,' providing some context. However, it lacks information about return values or pagination behavior, especially given the absence of an output schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the parameter details are already in the schema. The description adds no additional meaning beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states it lists personas (simulated users) and specifies key fields like voice_name, language_code, etc. It distinguishes itself from sibling list tools (e.g., list_agents) by focusing on personas, but does not explicitly differentiate.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The phrase 'Required for runs' implies usage context but does not provide explicit guidance on when to use vs. alternatives or when not to use.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_runsA

List evaluation runs. Each run = agent + persona + test_set. Returns run_id, status, tags. Filter by tag: filter='tag="regression"'.

ParametersJSON Schema
NameRequiredDescriptionDefault
page_sizeNoNumber of results per page (1-100, default 50)
page_tokenNoToken for retrieving the next page of results
order_byNoSort order (e.g., "-create_time" for newest first)
filterNoFilter expression (e.g., status="COMPLETED")

TDQS

A4.1/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so description carries the burden. It discloses that the tool returns run_id, status, tags but does not mention pagination behavior despite schema having page_size/page_token. It implies a safe read operation but does not explicitly state read-only nature.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences conveying purpose, composition, return fields, and a usage example. No wasted words, efficient and to the point.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Lacks output schema; description partially explains return values (run_id, status, tags) but omits other potential fields. Pagination is not explicitly described despite being in schema. For a list tool, more context on pagination and sorting would improve completeness.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with descriptions for all parameters. The description adds a concrete filter example that goes beyond schema, aiding understanding. However, it does not elaborate on other parameters like order_by or page_token.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states 'List evaluation runs' and defines what constitutes a run (agent + persona + test_set). It distinguishes from sibling list_* tools by specifying the resource and the information returned (run_id, status, tags).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides a concrete filtering example with filter='tag="regression"' which guides usage of the filter parameter. However, it lacks explicit guidance on when to use this tool versus other list tools or when not to use it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_test_casesB

List test cases. Filter by test_set_id. Each has input_str (scenario text or JSON message array) and optional expected_behaviors.

ParametersJSON Schema
NameRequiredDescriptionDefault
page_sizeNoNumber of results per page (1-100, default 50)
page_tokenNoToken for retrieving the next page of results
order_byNoSort order (e.g., "-create_time" for newest first)
filterNoFilter expression (e.g., status="COMPLETED")

TDQS

B3.3/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, and the description does not disclose behavioral traits such as pagination, read-only nature, or rate limits. Schema implies pagination via page_size/page_token, but description doesn't confirm.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with the core action ('List test cases'), no superfluous content. Every sentence adds essential information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Adequate for a simple list tool with well-documented schema. However, lacks explanation of pagination behavior and response structure (e.g., array of test cases, next_page_token). Could be more complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with descriptions for all 4 parameters. Baseline is 3. Description adds marginal value by suggesting a typical filter (test_set_id) and mentioning output fields, but doesn't explain parameter semantics beyond schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb ('List') and resource ('test cases'), and distinguishes from sibling tools like get_test_case and create_test_case by specifying the listing nature and output fields.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool versus alternatives (e.g., get_test_case for a single test case, list_test_sets for test sets). Missing when-not-to-use and context for filter usage.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_test_setsB

List test sets (collections of test cases). Each contains scenarios to run against an agent. Use test_set_id when creating runs.

ParametersJSON Schema
NameRequiredDescriptionDefault
page_sizeNoNumber of results per page (1-100, default 50)
page_tokenNoToken for retrieving the next page of results
order_byNoSort order (e.g., "-create_time" for newest first)
filterNoFilter expression (e.g., status="COMPLETED")

TDQS

B3.4/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, and the description only adds that test sets contain scenarios. It does not disclose pagination behavior, rate limits, or response structure, which is insufficient for a listing tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two concise sentences. The first defines the tool, the second adds a practical usage hint. No extraneous information, well structured.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no output schema and 4 optional parameters, the description is incomplete. It does not specify what fields are returned, how filter works, or how to use pagination tokens, leaving the agent underinformed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with descriptions for all 4 parameters. The description adds no additional parameter-level meaning, so baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: listing test sets (collections of test cases). It distinguishes from siblings by mentioning test_set_id usage in runs, which is unique among sibling tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides a hint about using test_set_id when creating runs, but lacks explicit guidance on when to use this tool vs alternatives like list_test_cases, or context on filtering and pagination.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

update_agentA

Update an existing agent configuration. Only provided fields will be updated.

ParametersJSON Schema
NameRequiredDescriptionDefault
agent_idYesThe unique ID of the agent to update
display_nameNoNew display name
phone_numberNoNew phone number
endpointNoNew endpoint URL
promptNoNew system prompt
metadataNoNew metadata

TDQS

A3.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries full burden. It discloses partial update behavior (only provided fields updated) but omits other important traits such as authorization needs, rate limits, reversibility, or error states. The mutation nature is clear, but transparency is limited.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, direct sentence (10 words) that immediately states purpose and key behavior. It is optimally concise with no superfluous text, front-loading the essential information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given 6 parameters (with nested objects) and no output schema or annotations, the description is adequate but incomplete. It explains partial update behavior but does not clarify return values, validation rules for specific fields, or constraints on metadata. More detail could improve completeness.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so baseline is 3. The description's statement 'Only provided fields will be updated' reinforces schema cues but adds no new meaning beyond what parameter descriptions already provide. No extra context for nested objects (metadata) is given.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb 'Update' and the resource 'existing agent configuration', distinguishing it from sibling tools like create_agent (creation) and get_agent (retrieval). It directly conveys the tool's primary purpose without ambiguity.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description notes 'Only provided fields will be updated,' implying partial update behavior, but lacks explicit guidance on when to use this tool versus alternatives (e.g., avoid for creation). It does not mention prerequisites or exclusions, leaving usage partially implied.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

update_test_caseC

Update test case input_str, expected_behaviors, or other fields.

ParametersJSON Schema
NameRequiredDescriptionDefault
test_case_idYesThe unique ID of the test case to update.
input_strNoUpdated test input or scenario.
expected_behaviorsNoUpdated list of expected agent behaviors.
descriptionNoUpdated description.
simulation_metadata_inputNoUpdated simulation context.
metric_inputNoUpdated metric inputs.
user_notesNoUpdated internal notes.

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided; the description bears full burden. It only states 'Update' which implies mutation, but lacks details on destructive nature, authorization needs, rate limits, or side effects. Minimal additional value beyond the name.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Single sentence, no fluff. Efficiently conveys the core action and affected fields, but could be slightly more structured (e.g., noting required parameter).

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema and 7 parameters including nested objects. Description lacks details on return value, error handling, prerequisites, or behavior when updating partially. For a complex update tool, this is insufficient.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so each parameter already has a description. The description lists some fields but does not add significant meaning beyond what the schema provides. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states it updates a test case and lists specific fields (input_str, expected_behaviors, other fields). The verb 'Update' and resource 'test case' are explicit. However, it does not differentiate from sibling tools like update_agent, which also update resources.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool versus alternatives (e.g., creating a new test case via create_test_case vs updating). No mention of prerequisites or conditions for use.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. 18 tool updatesv0.1.0
    • First observedcreate_agent
    • First observedcreate_run
    • First observedcreate_test_case
    • First observedcreate_test_set
    • First observedget_agent
    • First observedget_metric
    • First observedget_persona
    • First observedget_run
    • First observedget_test_case
    • First observedget_test_set
    • First observedlist_agents
    • First observedlist_metrics
    • First observedlist_personas
    • First observedlist_runs
    • First observedlist_test_cases
    • First observedlist_test_sets
    • First observedupdate_agent
    • First observedupdate_test_case

TDQS

A3.7/5.0
Disambiguation5/5

Each tool targets a distinct resource (agent, run, test case, test set, metric, persona) with specific CRUD operations, leaving no ambiguity between tools.

Naming Consistency5/5

All tool names follow a consistent verb_noun pattern (create_, get_, list_, update_), making it easy to predict tool purpose from the name.

Tool Count5/5

18 tools cover the complete lifecycle of evaluating AI agents (agents, personas, test sets, runs, metrics) without being overly numerous or sparse.

Completeness4/5

Core workflows are covered, but missing delete operations for agents, test cases, and test sets, as well as update for test sets, leave minor gaps in the surface.

Maintenance

ActivityActive
ResponsivenessNo issues

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/coval-ai/mcp-server'

If you have feedback or need assistance with the MCP directory API, please join our Discord server