Coval MCP Server
OfficialThe Coval MCP Server integrates Coval's AI evaluation platform into MCP clients, enabling you to manage and run evaluations of AI agents.
Evaluation Runs
List, get, and create evaluation runs — launch evaluations by combining an agent, persona, and test set, with options for tags, concurrency, and iteration count
Agents
List, get, create, and update AI agents (VOICE, OUTBOUND_VOICE, SMS, WEBSOCKET, CHAT) with configuration details like endpoints, phone numbers, and prompts
Test Sets & Test Cases
List, get, and create test sets (collections of test scenarios)
List, get, create, and update individual test cases, including single-turn or multi-turn conversations, expected behaviors, and metadata
Metrics
List available evaluation metrics and get detailed information about specific metrics
Personas
List and get simulated user personas used in evaluations, including voice, language, background sound, and behavior settings
Covi Delegation
Use
consult_covito ask read-only evaluation and analysis questions to Covi
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Coval MCP ServerShow my recent evaluation runs"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Coval MCP Server
The official Model Context Protocol server for Coval - the AI evaluation platform.
This MCP server allows AI assistants like Claude, ChatGPT, Codex, and Cursor to interact with Coval's evaluation APIs, enabling you to:
Launch and monitor evaluation runs
Manage AI agents and test sets
Retrieve evaluation metrics and results
Inspect saved reports and recurring evaluation schedules
Hosted connector
The hosted connector is the recommended way to use Coval from Claude, ChatGPT, and other remote MCP clients. It uses OAuth, so users sign in to Coval and select the organization the connector can access. No Coval API key is copied into the client.
Standard MCP endpoint: https://mcp.coval.dev/mcp
Claude hosted endpoint: https://mcp.coval.dev/claude/mcp
Claude
Open Customize > Connectors. Team and Enterprise owners add the connector first from Organization settings > Connectors.
Add a custom web connector using
https://mcp.coval.dev/claude/mcp.Select Connect, sign in to Coval, and choose the Coval organization to authorize.
Enable the connector in a conversation and ask Claude to use Coval or Sofia.
See Claude's remote MCP connector guide for current plan and administration requirements.
ChatGPT
Enable developer mode for your ChatGPT workspace account.
From Settings > Apps, create an app using
https://mcp.coval.dev/mcpand OAuth.Select Scan Tools, sign in to Coval, and choose the Coval organization to authorize.
Create the app, then enable it in a conversation.
See ChatGPT's MCP app guide for current plan and workspace requirements.
Codex
Use Codex 0.144.2 or newer. Current Codex Desktop releases include a compatible client. For the
CLI, update before connecting:
npm install -g @openai/codex@latest
codex mcp add coval --url https://mcp.coval.dev/mcpWhen the browser opens, sign in to Coval, select the organization Codex should access, and select Allow. Codex Desktop, the Codex CLI, and the IDE extension share MCP configuration on the same machine, so the connection is available across those clients after they restart.
In Codex Desktop, you can also add Coval from Settings > MCP servers as a Streamable HTTP
server using https://mcp.coval.dev/mcp, then select Authenticate.
If OAuth returns invalid_scope, first confirm codex --version is 0.144.2 or newer. Older
clients registered a dynamic OAuth client without all the scopes they requested during consent.
After updating, clear the stale registration and reconnect:
codex mcp logout coval
codex mcp remove coval
codex mcp add coval --url https://mcp.coval.dev/mcpNo API key, manual scope list, or oauth_resource override is required. See
Codex's MCP documentation for current client setup
details.
Related MCP server: VerifyAX MCP
Local installation
npx @coval/mcp-serverUse the local stdio server for service accounts, automation, and clients that do not support remote OAuth.
Claude Desktop
Add to ~/Library/Application Support/Claude/claude_desktop_config.json:
{
"mcpServers": {
"coval": {
"command": "npx",
"args": ["-y", "@coval/mcp-server"],
"env": {
"COVAL_API_KEY": "your_api_key_here"
}
}
}
}Cursor
Add to .cursor/mcp.json in your project:
{
"mcpServers": {
"coval": {
"command": "npx",
"args": ["-y", "@coval/mcp-server"],
"env": {
"COVAL_API_KEY": "your_api_key_here"
}
}
}
}Get your API key from app.coval.dev/settings
Available Tools
Tool | Description |
| List all agents in your workspace |
| Get details of a specific agent |
| Create an agent configuration |
| Update an agent configuration |
| List evaluation runs |
| Get details of a specific run |
| Start a new evaluation run |
| List available test sets |
| Get test set details |
| Create a test set |
| List test cases in a test set |
| Get test case details |
| Add a test case to a test set |
| Update a test case |
| List evaluation metrics |
| Get metric details |
| List available personas |
| Get persona details |
| List saved evaluation reports |
| Get report details and a bounded page of result rows |
| Create an organization-private saved report |
| List reusable evaluation configurations |
| List recurring evaluation schedules |
| Get a schedule and paginated recent-run history |
| Create a schedule, disabled by default |
| Update selected schedule fields |
| Delegate a read-only Coval evaluation question to Sofia |
Example Usage
Once connected, you can ask Claude things like:
"Show me my recent evaluation runs" "List all my agents" "Run an evaluation of my customer-support-agent against the billing-inquiries test set" "What are the metrics for run abc123?" "Show the first 20 rows of my regression report for the task-success metric" "Create a private report comparing these completed runs by agent" "Show my enabled evaluation schedules and the latest runs from the nightly regression" "Create a disabled weekday schedule from my regression run template" "Use
consult_sofiato analyze my latest failed run and recommend the most useful next test."
The same tools, including consult_sofia, are available through every supported endpoint and
transport:
Standard remote Streamable HTTP:
https://mcp.coval.dev/mcpusing OAuth. This is the recommended connection for ChatGPT, Codex, and other hosted MCP clients.Claude remote Streamable HTTP:
https://mcp.coval.dev/claude/mcpusing the same OAuth flow and tool surface, with Claude-specific risk annotations.Local stdio:
npx @coval/mcp-serverwithCOVAL_API_KEY, for service accounts and local development.
The hosted connector can access only the Coval organization selected during OAuth consent. Remove the connector from the client or revoke its Coval access when it is no longer needed.
Development
# Install dependencies
npm install
# Build
npm run build
# Test locally with MCP Inspector
npm run inspector
# Run tests
npm test
# Check the public health and OAuth discovery endpoints
npm run check:remoteEnvironment Variables
Variable | Required | Default | Description |
| Stdio | - | Coval API key for the local stdio transport |
| No |
| API base URL |
| No | Derived from | Overrides the expected Sofia origin used to validate delegation URLs |
| No |
| Logging level |
| No | Claude and OpenAI web origins | Comma-separated exact browser origins allowed to call |
| No | - | OpenAI plugin-portal domain verification token served as plain text from |
| No |
| Deployment environment included in structured hosted-request logs |
| No |
| Service name included in structured hosted-request logs |
| No |
| Immutable release identifier included in structured hosted-request logs |
Documentation
License
MIT
Support
Available Tools
18 toolscreate_agentB
Create a new agent configuration. Specify the model type (voice, chat, SMS, websocket) and connection details.
| Name | Required | Description | Default |
|---|---|---|---|
| display_name | Yes | Human-readable name for the agent | |
| model_type | Yes | Type of agent: MODEL_TYPE_VOICE, MODEL_TYPE_CHAT, MODEL_TYPE_SMS, etc. | |
| phone_number | No | Phone number for voice agents (E.164 format) | |
| endpoint | No | Webhook or WebSocket endpoint URL | |
| prompt | No | System prompt or instructions for the agent | |
| metadata | No | Custom metadata for the agent |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided. The description does not disclose behavioral traits such as side effects, authentication requirements, or rate limits. It merely restates the obvious creation action without additional context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two short sentences, front-loaded with the action. Every word is necessary and there is no extraneous information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the complexity (6 parameters, nested objects, no output schema), the description is too brief. It does not explain return values, the relationship between parameters (e.g., phone_number only for voice), or prerequisites like authentication. This leaves significant gaps for an agent to use correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 100% description coverage for all 6 parameters. The description adds a high-level summary mentioning 'model type' and 'connection details', but does not provide deeper semantics beyond what the schema already offers. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Create' and the resource 'agent configuration'. It lists specific attributes like model type and connection details, making it distinct from sibling tools that operate on runs, test cases, etc.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus other create tools (e.g., create_run, create_test_case). The description simply states what it does without any context or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
create_runA
Launch evaluation: agent + persona + test_set. Optionally add tags for filtering. Poll get_run until status=COMPLETED to see metrics.
| Name | Required | Description | Default |
|---|---|---|---|
| agent_id | Yes | The unique ID of the agent to evaluate. Get this from list_agents. | |
| persona_id | Yes | The unique ID of the persona to use. Get this from list_personas. | |
| test_set_id | Yes | The unique ID of the test set to run against. Get this from list_test_sets. | |
| metric_ids | No | Optional list of metric IDs to evaluate. Uses agent defaults if omitted. | |
| options | No | Run configuration options | |
| tags | No | Tags for categorizing and filtering the run (max 20 tags, 200 chars each). Filter later with list_runs filter='tag="regression"'. | |
| metadata | No | Custom metadata for tracking purposes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Without annotations, the description reveals that the tool launches an asynchronous process and advises polling get_run for completion. This provides behavioral transparency beyond just 'create a run', though it could mention resource limits or prerequisites.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loading the core purpose and then adding key usage hints. No redundant or unnecessary text.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (7 params, nested objects, no output schema), the description covers the essential action and follow-up (polling). It could elaborate on the return value or error conditions, but it's fairly complete for a launch tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema covers all parameters with descriptions (100% coverage), so the description adds minimal extra meaning. The note about tags for filtering is already in the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool launches an evaluation combining agent, persona, and test_set. It distinguishes itself from sibling tools like create_agent or create_test_set by focusing on the run action.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives implicit guidance by mentioning polling get_run for results, but it does not explicitly state when to use this tool versus alternatives or when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
create_test_caseA
Create test case in a test set. input_str: single scenario message OR JSON array [{role,content},...] for multi-turn conversations.
| Name | Required | Description | Default |
|---|---|---|---|
| test_set_id | Yes | The 8-character ID of the test set to add this test case to. | |
| input_str | Yes | The test input or scenario that will be presented to the agent. | |
| expected_behaviors | No | List of expected agent behaviors or responses for evaluation. | |
| description | No | Human-readable description of what this test case validates. | |
| simulation_metadata_input | No | Additional context passed to the simulation environment. | |
| metric_input | No | Custom inputs for metric evaluation. | |
| user_notes | No | Internal notes about this test case. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must disclose behavioral traits. It reveals that input_str can be a string or JSON array, but fails to mention whether the operation is destructive, requires specific permissions, or has rate limits. It does not indicate if the tool returns a value or has side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise: one sentence for purpose, one phrase for parameter detail. No unnecessary words, and the key information (two input formats) is front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 7 parameters (including nested objects), no output schema, and no annotations, the description is insufficient. It does not explain the role of simulation_metadata_input, metric_input, or what the tool returns after creation. The agent will lack critical context for proper use.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description adds value by explaining the dual format of input_str, which is not obvious from the schema. For other parameters, it largely echoes the schema, but the additional context for input_str justifies a 4.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Create test case in a test set.' It differentiates from siblings like create_test_set by specifying the operation on a test case. It also adds detail on input_str formats, making it specific and actionable.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description does not provide guidance on when to use this tool versus alternatives, nor does it mention prerequisites or when not to use it. While the purpose is clear, usage context is minimal.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
create_test_setA
Create a test set to organize test cases. After creating, use create_test_case to add scenarios.
| Name | Required | Description | Default |
|---|---|---|---|
| display_name | Yes | Human-readable name for the test set | |
| slug | No | URL-friendly identifier (auto-generated if not provided) | |
| description | No | Description of the test set | |
| test_set_type | No | Type of test set: DEFAULT, SCENARIO, TRANSCRIPT, WORKFLOW | |
| test_set_metadata | No | Additional configuration metadata | |
| parameters | No | Test case parameterization (e.g., {"name": ["Alice", "Bob"]}) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so the description carries full burden. It explains the creation behavior and the next step, but lacks details on side effects or permissions. Adequate but not thorough.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences, front-loaded with the purpose, followed by a useful next-step instruction. No extraneous information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with 6 parameters and no output schema, the description covers the core purpose and usage flow. It could benefit from more parameter usage context, but the schema compensates.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so parameters are well-documented in the schema. The description does not add additional meaning beyond the schema, meeting the baseline.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: creating a test set to organize test cases. It also distinguishes from sibling tools like 'create_test_case' by specifying that test cases are added separately.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides a clear context for use (to organize test cases) and a follow-up action (use create_test_case). It does not explicitly state when not to use it, but the context is sufficient.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_agentA
Get agent config: model_type, phone_number (voice), endpoint (websocket/chat), and display_name.
| Name | Required | Description | Default |
|---|---|---|---|
| agent_id | Yes | The unique ID of the agent to retrieve. Get this from list_agents. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden for behavioral disclosure. It only lists returned fields, omitting traits like idempotency, error handling, authorization needs, or whether this triggers side effects. For a read tool, this is minimal but still insufficient.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficient sentence under 20 words that front-loads the purpose. It earns its place with specific return fields. Slightly under 5 because it could be even more concise, but overall excellent.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple getter with one parameter, no output schema, and a list of siblings, the description is nearly complete. It lists important return fields, helping the agent understand output shape. Could be improved by mentioning null handling or error cases, but sufficient.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and the parameter description already explains 'agent_id' (unique ID from list_agents). The tool description adds no extra meaning beyond what the schema provides, so baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Get') and resource ('agent config'), and lists exact return fields (model_type, phone_number, endpoint, display_name). This clearly differentiates it from sibling tools like list_agents (list all) and create_agent (create new).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies use when needing a specific agent's configuration, but lacks explicit guidance on when to use this versus alternative tools like list_agents or other getters (get_persona, get_metric). No exclusions or when-not-to-use are provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_metricA
Get detailed information about a specific metric. Shows metric type, description, and configuration.
| Name | Required | Description | Default |
|---|---|---|---|
| metric_id | Yes | The unique ID of the metric to retrieve. Get this from list_metrics. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must fully disclose behavior. It states that the tool 'shows metric type, description, and configuration,' implying read-only access, but does not explicitly confirm no side effects, permissions, or rate limits.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise, consisting of two sentences that convey the core purpose and output without unnecessary verbiage.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (single parameter, no output schema), the description adequately covers the input requirement and output details (type, description, configuration). It is sufficiently complete for an agent to use correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, providing a baseline of 3. The description adds value by noting that the metric_id comes from list_metrics, which aids in understanding the prerequisite.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Get detailed information about a specific metric.' It distinguishes itself from sibling tools like 'list_metrics' by focusing on a single metric's details.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage when a metric_id is available, and hints at a prerequisite by saying 'Get this from list_metrics,' but does not explicitly state when to use this tool versus alternatives or when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_personaA
Get persona details: voice_name, language_code (BCP-47), background_sound, persona_prompt (behavior), wait_seconds, conversation_initiation (speak_first/wait_for_user).
| Name | Required | Description | Default |
|---|---|---|---|
| persona_id | Yes | The unique ID of the persona to retrieve. Get this from list_personas. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden. It states that the tool retrieves details and lists fields, but does not disclose potential side effects, authentication needs, rate limits, or any constraints. For a simple read operation, this is adequate but not thorough.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence efficiently listing the returned fields. It is concise with no extraneous words, though a bulleted list could improve readability. Overall, it earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the low complexity (1 parameter, no output schema, no nested objects), the description is fairly complete by listing the returned fields. However, it does not specify the output format (e.g., JSON object) or any additional context, leaving room for minor ambiguity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% (persona_id is fully described in the schema). The description adds no additional meaning beyond what the schema already provides, so the score defaults to baseline 3.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Get persona details' and enumerates the specific fields returned (voice_name, language_code, etc.). Among sibling tools like get_agent, get_metric, etc., it uniquely identifies the resource as persona details, making the purpose unmistakable.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage when needing persona details but does not explicitly state when to use this tool versus alternatives like list_personas. The schema description hints that list_personas provides the ID, but the tool description itself lacks clear when-to-use and when-not-to-use guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_runA
Get run status/results. Status: PENDING→RUNNING→COMPLETED. Completed runs include metrics (custom per org) and output_ids for transcripts.
| Name | Required | Description | Default |
|---|---|---|---|
| run_id | Yes | The unique ID of the run to retrieve. Get this from list_runs. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, but the description discloses the status lifecycle and what completed runs include (metrics, output_ids). This helps set expectations for the agent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, each with clear purpose: first states action, second provides behavioral details. No redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple read tool with one parameter and no output schema, the description adequately explains the response (status, metrics, output_ids).
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The only parameter run_id is fully described in the schema. The tool description adds no additional meaning beyond what the schema provides, so baseline score of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool retrieves run status and results, with a specific lifecycle (PENDING→RUNNING→COMPLETED). It distinguishes from sibling tools like create_run and list_runs.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when to use this tool vs alternatives. The schema mentions getting run_id from list_runs, but no context on when not to use or prerequisites.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_test_caseA
Get test case details: input_str (the scenario), expected_behaviors, and metadata.
| Name | Required | Description | Default |
|---|---|---|---|
| test_case_id | Yes | The unique ID of the test case to retrieve. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It does not disclose read-only status, idempotency, or any side effects. The description merely lists return fields, which is adequate for a simple retrieval but lacks explicit behavioral details.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence that is concise and front-loaded with the primary purpose. It could be improved with structured formatting, but it is efficient and avoids unnecessary words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description partially explains the return fields (input_str, expected_behaviors, metadata). This provides a basic expectation but does not cover error cases or response format details. It is minimally adequate for a simple retrieval tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The only parameter 'test_case_id' is fully described in the schema with a clear purpose. Schema coverage is 100%, so the baseline is 3. The description does not add additional meaning to the parameter beyond what the schema already provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Get test case details' and lists specific fields: input_str, expected_behaviors, and metadata. This verb+resource combination distinguishes it from sibling tools like list_test_cases and create_test_case.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit when-to-use or when-not-to-use guidance is provided. However, the tool name and context imply it should be used when a specific test_case_id is known, rather than listing all cases.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_test_setA
Get test set details: display_name, description, and test case count. Use list_test_cases to see individual scenarios.
| Name | Required | Description | Default |
|---|---|---|---|
| test_set_id | Yes | The unique ID of the test set to retrieve. Get this from list_test_sets. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided; description indicates it's a read operation but does not explicitly state read-only, permissions, or side effects. Adequate for a simple get but could be more explicit.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences, front-loaded with purpose, no extraneous information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple tool with one parameter and no output schema, the description covers what is returned and provides guidance to a sibling. Complete given the simplicity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema has 100% coverage with a clear description for test_set_id. Description adds minimal value beyond schema, only a slight pointer to list_test_sets.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states the verb (Get), resource (test set), and what details are returned (display_name, description, test case count). It also distinguishes from sibling tool list_test_cases.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides explicit direction to use list_test_cases for individual scenarios, but does not mention other relevant siblings like list_test_sets or update_test_set.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_agentsC
List agents (AI systems to evaluate). Model types: VOICE, OUTBOUND_VOICE, SMS, WEBSOCKET, CHAT. Use agent_id when creating runs.
| Name | Required | Description | Default |
|---|---|---|---|
| page_size | No | Number of results per page (1-100, default 50) | |
| page_token | No | Token for retrieving the next page of results | |
| order_by | No | Sort order (e.g., "-create_time" for newest first) | |
| filter | No | Filter expression (e.g., status="COMPLETED") |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden for behavioral disclosure. It only mentions model types and a creation hint, but fails to disclose behavior such as pagination (implied by schema but not stated), rate limits, authentication needs, or return format.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences with no wasted words. The purpose is front-loaded, model types are listed efficiently, and a hint is appended. Perfectly concise for the content provided.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema and a list tool with 4 parameters, the description is incomplete. It does not describe the return structure (e.g., list of agent objects), any default ordering, or what happens with empty results. The model types hint is tangential rather than addressing completeness needs.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the baseline is 3. The description adds no parameter-specific details beyond what the schema already provides. The mention of model types is not linked to any parameter, so no added value.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'List agents (AI systems to evaluate)' with a specific verb and resource. It adds context about model types, which helps clarify what agents are. However, it does not explicitly differentiate from sibling list tools like list_runs or list_test_cases.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides a tangential hint ('Use agent_id when creating runs') but no explicit guidance on when to use this tool versus alternatives (e.g., get_agent, create_agent) or when not to use it. No usage context is given.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_metricsA
List available evaluation metrics. Metrics define how agent performance is measured. Set include_builtin=true to see built-in metrics.
| Name | Required | Description | Default |
|---|---|---|---|
| page_size | No | Number of results per page (1-100, default 50) | |
| page_token | No | Token for retrieving the next page of results | |
| order_by | No | Sort order (e.g., "-create_time" for newest first) | |
| filter | No | Filter expression (e.g., status="COMPLETED") | |
| include_builtin | No | Include built-in metrics in results (default false) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, and the description fails to disclose behavioral traits such as pagination behavior, default sorting, or what happens when parameters are omitted. The description adds minimal behavioral context beyond the literal listing action.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, directly states purpose and one key condition. No extraneous information. Front-loaded with action and resource.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 5 parameters, no output schema, and no annotations, the description covers the core purpose and one key parameter. It could mention pagination behavior or default sorting, but the schema compensates. Acceptably complete for its complexity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, meaning each parameter is already documented in the schema. The description adds meaning only to 'include_builtin'. For other parameters, it adds no extra value, meeting the baseline 3.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'List' and the resource 'available evaluation metrics'. It differentiates from sibling list tools by specifying 'evaluation metrics', distinct from agents, runs, test cases, etc.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides a specific usage guideline with 'Set include_builtin=true to see built-in metrics'. While it doesn't explicitly mention when not to use or alternative tools, the context and name make it obvious this is for metrics only.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_personasB
List personas (simulated users). Each has voice_name, language_code, background_sound (off/office/crowd/airport/etc), and behavior prompt. Required for runs.
| Name | Required | Description | Default |
|---|---|---|---|
| page_size | No | Number of results per page (1-100, default 50) | |
| page_token | No | Token for retrieving the next page of results | |
| order_by | No | Sort order (e.g., "-create_time" for newest first) | |
| filter | No | Filter expression (e.g., status="COMPLETED") |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It mentions what a persona contains and that it's required for runs, but does not disclose behavioral traits like pagination, read-only nature, or any side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise (two sentences) and front-loaded with the action. It efficiently conveys the purpose without wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description mentions key fields of personas and states 'Required for runs,' providing some context. However, it lacks information about return values or pagination behavior, especially given the absence of an output schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the parameter details are already in the schema. The description adds no additional meaning beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it lists personas (simulated users) and specifies key fields like voice_name, language_code, etc. It distinguishes itself from sibling list tools (e.g., list_agents) by focusing on personas, but does not explicitly differentiate.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'Required for runs' implies usage context but does not provide explicit guidance on when to use vs. alternatives or when not to use.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_runsA
List evaluation runs. Each run = agent + persona + test_set. Returns run_id, status, tags. Filter by tag: filter='tag="regression"'.
| Name | Required | Description | Default |
|---|---|---|---|
| page_size | No | Number of results per page (1-100, default 50) | |
| page_token | No | Token for retrieving the next page of results | |
| order_by | No | Sort order (e.g., "-create_time" for newest first) | |
| filter | No | Filter expression (e.g., status="COMPLETED") |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description carries the burden. It discloses that the tool returns run_id, status, tags but does not mention pagination behavior despite schema having page_size/page_token. It implies a safe read operation but does not explicitly state read-only nature.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences conveying purpose, composition, return fields, and a usage example. No wasted words, efficient and to the point.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Lacks output schema; description partially explains return values (run_id, status, tags) but omits other potential fields. Pagination is not explicitly described despite being in schema. For a list tool, more context on pagination and sorting would improve completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with descriptions for all parameters. The description adds a concrete filter example that goes beyond schema, aiding understanding. However, it does not elaborate on other parameters like order_by or page_token.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'List evaluation runs' and defines what constitutes a run (agent + persona + test_set). It distinguishes from sibling list_* tools by specifying the resource and the information returned (run_id, status, tags).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides a concrete filtering example with filter='tag="regression"' which guides usage of the filter parameter. However, it lacks explicit guidance on when to use this tool versus other list tools or when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_test_casesB
List test cases. Filter by test_set_id. Each has input_str (scenario text or JSON message array) and optional expected_behaviors.
| Name | Required | Description | Default |
|---|---|---|---|
| page_size | No | Number of results per page (1-100, default 50) | |
| page_token | No | Token for retrieving the next page of results | |
| order_by | No | Sort order (e.g., "-create_time" for newest first) | |
| filter | No | Filter expression (e.g., status="COMPLETED") |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, and the description does not disclose behavioral traits such as pagination, read-only nature, or rate limits. Schema implies pagination via page_size/page_token, but description doesn't confirm.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with the core action ('List test cases'), no superfluous content. Every sentence adds essential information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Adequate for a simple list tool with well-documented schema. However, lacks explanation of pagination behavior and response structure (e.g., array of test cases, next_page_token). Could be more complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with descriptions for all 4 parameters. Baseline is 3. Description adds marginal value by suggesting a typical filter (test_set_id) and mentioning output fields, but doesn't explain parameter semantics beyond schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb ('List') and resource ('test cases'), and distinguishes from sibling tools like get_test_case and create_test_case by specifying the listing nature and output fields.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives (e.g., get_test_case for a single test case, list_test_sets for test sets). Missing when-not-to-use and context for filter usage.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_test_setsB
List test sets (collections of test cases). Each contains scenarios to run against an agent. Use test_set_id when creating runs.
| Name | Required | Description | Default |
|---|---|---|---|
| page_size | No | Number of results per page (1-100, default 50) | |
| page_token | No | Token for retrieving the next page of results | |
| order_by | No | Sort order (e.g., "-create_time" for newest first) | |
| filter | No | Filter expression (e.g., status="COMPLETED") |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, and the description only adds that test sets contain scenarios. It does not disclose pagination behavior, rate limits, or response structure, which is insufficient for a listing tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two concise sentences. The first defines the tool, the second adds a practical usage hint. No extraneous information, well structured.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema and 4 optional parameters, the description is incomplete. It does not specify what fields are returned, how filter works, or how to use pagination tokens, leaving the agent underinformed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with descriptions for all 4 parameters. The description adds no additional parameter-level meaning, so baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: listing test sets (collections of test cases). It distinguishes from siblings by mentioning test_set_id usage in runs, which is unique among sibling tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides a hint about using test_set_id when creating runs, but lacks explicit guidance on when to use this tool vs alternatives like list_test_cases, or context on filtering and pagination.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
update_agentA
Update an existing agent configuration. Only provided fields will be updated.
| Name | Required | Description | Default |
|---|---|---|---|
| agent_id | Yes | The unique ID of the agent to update | |
| display_name | No | New display name | |
| phone_number | No | New phone number | |
| endpoint | No | New endpoint URL | |
| prompt | No | New system prompt | |
| metadata | No | New metadata |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden. It discloses partial update behavior (only provided fields updated) but omits other important traits such as authorization needs, rate limits, reversibility, or error states. The mutation nature is clear, but transparency is limited.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, direct sentence (10 words) that immediately states purpose and key behavior. It is optimally concise with no superfluous text, front-loading the essential information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 6 parameters (with nested objects) and no output schema or annotations, the description is adequate but incomplete. It explains partial update behavior but does not clarify return values, validation rules for specific fields, or constraints on metadata. More detail could improve completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so baseline is 3. The description's statement 'Only provided fields will be updated' reinforces schema cues but adds no new meaning beyond what parameter descriptions already provide. No extra context for nested objects (metadata) is given.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Update' and the resource 'existing agent configuration', distinguishing it from sibling tools like create_agent (creation) and get_agent (retrieval). It directly conveys the tool's primary purpose without ambiguity.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description notes 'Only provided fields will be updated,' implying partial update behavior, but lacks explicit guidance on when to use this tool versus alternatives (e.g., avoid for creation). It does not mention prerequisites or exclusions, leaving usage partially implied.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
update_test_caseC
Update test case input_str, expected_behaviors, or other fields.
| Name | Required | Description | Default |
|---|---|---|---|
| test_case_id | Yes | The unique ID of the test case to update. | |
| input_str | No | Updated test input or scenario. | |
| expected_behaviors | No | Updated list of expected agent behaviors. | |
| description | No | Updated description. | |
| simulation_metadata_input | No | Updated simulation context. | |
| metric_input | No | Updated metric inputs. | |
| user_notes | No | Updated internal notes. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided; the description bears full burden. It only states 'Update' which implies mutation, but lacks details on destructive nature, authorization needs, rate limits, or side effects. Minimal additional value beyond the name.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Single sentence, no fluff. Efficiently conveys the core action and affected fields, but could be slightly more structured (e.g., noting required parameter).
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema and 7 parameters including nested objects. Description lacks details on return value, error handling, prerequisites, or behavior when updating partially. For a complex update tool, this is insufficient.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so each parameter already has a description. The description lists some fields but does not add significant meaning beyond what the schema provides. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it updates a test case and lists specific fields (input_str, expected_behaviors, other fields). The verb 'Update' and resource 'test case' are explicit. However, it does not differentiate from sibling tools like update_agent, which also update resources.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives (e.g., creating a new test case via create_test_case vs updating). No mention of prerequisites or conditions for use.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
18 tool updates
v0.1.0- First observed
create_agent - First observed
create_run - First observed
create_test_case - First observed
create_test_set - First observed
get_agent - First observed
get_metric - First observed
get_persona - First observed
get_run - First observed
get_test_case - First observed
get_test_set - First observed
list_agents - First observed
list_metrics - First observed
list_personas - First observed
list_runs - First observed
list_test_cases - First observed
list_test_sets - First observed
update_agent - First observed
update_test_case
TDQS
Each tool targets a distinct resource (agent, run, test case, test set, metric, persona) with specific CRUD operations, leaving no ambiguity between tools.
All tool names follow a consistent verb_noun pattern (create_, get_, list_, update_), making it easy to predict tool purpose from the name.
18 tools cover the complete lifecycle of evaluating AI agents (agents, personas, test sets, runs, metrics) without being overly numerous or sparse.
Core workflows are covered, but missing delete operations for agents, test cases, and test sets, as well as update for test sets, leave minor gaps in the surface.
Maintenance
Resources
Unclaimed servers have limited discoverability.
Looking for Admin?
If you are the server author, to access and configure the admin panel.
Related MCP Connectors
Manage Speko voice-AI agents, sessions, calls, phone numbers, knowledge bases, evals, and docs.
Build, validate, and deploy multi-agent AI solutions from any AI environment.
Discover, inspect and run 63,000+ agent tools from one balance. Pay per call, no subscriptions.
1- mcp-serverOAuthcom.make
Give your AI agents the tools to build, manage, and run automation workflows.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceEnables automated LLM red teaming by submitting asynchronous test runs, retrieving aggregated metrics, and accessing artifacts.4CC BY-SA 4.0

VerifyAX MCPofficial
AlicenseAqualityAmaintenanceEnables conversational access to the VerifyAX agent-evaluation platform, exposing tools for agent evaluation workflows through natural language.121Apache 2.0- AlicenseNot gradedqualityBmaintenanceEnables LLM evaluation and observability by uploading documents, building test sets, running RAG pipelines, and automatically scoring answers for groundedness, hallucination risk, retrieval quality, latency, and cost, with tools exposed to MCP-compatible clients.1MIT
- AlicenseNot gradedqualityCmaintenanceEnables AI agents to access observability and evaluation data, including run history, span traces, LLM-as-judge evaluation results, and regression reports.MIT
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/coval-ai/mcp-server'
If you have feedback or need assistance with the MCP directory API, please join our Discord server