Skip to main content
Glama
Derrbal

TestRail MCP Server

by Derrbal

TestRail MCP Server

What is this?

This tool connects your TestRail test management system with AI assistants like Cursor or Claude Desktop. It allows your AI assistant to read and update test cases, add test results, and manage your testing workflow through simple conversations.

Related MCP server: TestRail MCP Server

Features

  • View Test Cases: Show test cases, projects, and test runs

  • Update Tests: Modify test case details, titles, and descriptions

  • Add Results: Record test results and upload screenshots

  • Search & Filter: Find specific tests or projects quickly

  • File Attachments: Upload documents and images to test cases

  • Full Integration: Works with Cursor, Claude Desktop, and other AI tools

Who is this for?

  • QA Engineers who want to manage tests through AI conversations

  • Project Managers who need quick access to test information

  • Developers who want to integrate testing into their AI-assisted workflow

  • Anyone who uses TestRail and wants to make it more accessible through AI

Quick Start

  1. Install Node.js from nodejs.org

  2. Get TestRail API key from your TestRail settings

  3. Install the server: npm install -g testrail-mcp-server

  4. Configure your AI assistant with your TestRail credentials

  5. Restart your AI assistant and start asking questions!

Prerequisites

You need these tools before installing:

  1. Node.js 18.17.0+ - Download from nodejs.org (LTS version)

  2. npm - Comes with Node.js automatically

  3. AI Assistant - Cursor (cursor.sh) or Claude Desktop (claude.ai)

  4. TestRail Account - With API access enabled

Verify installation:

node --version  # Should show 18.17.0 or higher
npm --version   # Should show a version number

Getting TestRail API Credentials

  1. Log into TestRail at your instance URL (e.g., https://yourcompany.testrail.com)

  2. Get API Key:

    • Click your profile picture → "My Settings"

    • Scroll to "API Keys" section

    • Click "Add API Key"

    • Name it (e.g., "MCP Server")

    • Copy and save the API key securely

  3. Note your credentials:

    • Username: Your TestRail login (usually email)

    • URL: Your TestRail web address

Installation

Step 1: Install the Server

Open Command Prompt/Terminal and run:

npm install -g testrail-mcp-server

Step 2: Configure Your AI Assistant

For Cursor Users:

  1. Open Cursor → Settings (Ctrl + , or Cmd + ,)

  2. Find "MCP" settings

  3. Add this configuration:

{
  "mcpServers": {
    "testrail": {
      "command": "npx",
      "args": ["testrail-mcp-server"],
      "env": {
        "TESTRAIL_USERNAME": "your_testrail_username",
        "TESTRAIL_API_KEY": "your_testrail_api_key",
        "TESTRAIL_URL": "https://your-instance.testrail.com"
      }
    }
  }
}

For Claude Desktop Users:

  1. Find config file:

    • Windows: %APPDATA%\Claude\claude_desktop_config.json

    • Mac: ~/Library/Application Support/Claude/claude_desktop_config.json

  2. Add the same configuration as above

Replace these values:

  • your_testrail_username: Your TestRail login (usually email)

  • your_testrail_api_key: The API key from TestRail

  • https://your-instance.testrail.com: Your TestRail URL

Step 3: Restart and Test

  1. Restart your AI assistant completely

  2. Test: Ask "Can you show me my TestRail projects?"

Usage

Once set up, simply talk to your AI assistant naturally:

Viewing Information:

  • "Show me all my TestRail projects"

  • "What test cases are in project 1?"

  • "Show me test case number 123"

Updating Tests:

  • "Update test case 123 with the title 'Login functionality test'"

  • "Change the priority of test case 456 to high"

  • "Add a comment to test case 789"

Adding Results:

  • "Mark test 456 as passed"

  • "Add a failed result to test 789 with comment 'Button not clickable'"

  • "Upload this screenshot to test case 123"

Searching:

  • "Find all test cases with 'login' in the title"

  • "Show me failed tests from the last test run"

Troubleshooting

"Command not found" or "npm is not recognized"

  • Install Node.js from nodejs.org

  • Restart Command Prompt/Terminal

  • Check with node --version

"Authentication failed" or "Invalid credentials"

  • Verify your TestRail username (usually email)

  • Copy API key again from TestRail

  • Check TestRail URL starts with https://

"Connection timeout" or "Cannot connect to TestRail"

  • Check internet connection

  • Verify TestRail URL is correct

  • Test TestRail in web browser

AI assistant doesn't respond to TestRail questions

  • Restart your AI assistant completely

  • Check configuration file format (valid JSON)

  • Verify all required fields are filled

"Permission denied" error

  • Run Command Prompt as Administrator (Windows) or with sudo (Mac/Linux)

  • Update to latest Node.js version

Still having trouble?

  • Check error messages for clues

  • Try installation steps again

  • Create an issue on GitHub with your error details

Security Best Practices

  • Never share your API key - treat it like a password

  • Use HTTPS URLs - ensure your TestRail URL starts with https://

  • Don't put API keys in code - always use configuration files

  • Rotate API keys regularly - change them every few months

  • Keep TestRail account secure - use strong passwords and 2FA

What NOT to do:

  • Don't share configuration files with others

  • Don't put API keys in public repositories

  • Don't use HTTP URLs (only HTTPS)

  • Don't share screenshots showing your API key

Additional Resources

Helpful Links:

Getting Support:

  • GitHub Issues - Report bugs or ask questions

  • Contact your TestRail administrator for API access issues

License

MIT License - see LICENSE file for details.

Available Tools

18 tools
add_attachment_to_caseAdd Attachment to TestRail CaseC

Upload a file attachment to a TestRail test case.

ParametersJSON Schema
NameRequiredDescriptionDefault
case_idYesTestRail case ID
file_pathYesPath to the file to upload as attachment

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden of behavioral disclosure. While 'Upload a file attachment' implies a write/mutation operation, it lacks critical details: required permissions (e.g., edit access to the case), file size/type limits, whether the attachment is public or private, or how failures are handled. This is inadequate for a mutation tool with zero annotation coverage.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence that front-loads the core action and target. There's no wasted wording or redundancy, making it easy to parse quickly while conveying the essential purpose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the complexity of a file upload operation (mutation with potential side effects), no annotations, and no output schema, the description is incomplete. It doesn't address behavioral aspects like error handling, response format, or constraints, leaving significant gaps for the agent to navigate blindly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, with clear parameter descriptions in the schema itself (e.g., 'TestRail case ID', 'Path to the file to upload as attachment'). The description adds no additional semantic context beyond what's already in the schema, such as example file paths or case ID formats. This meets the baseline for high schema coverage but doesn't enhance understanding.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Upload a file attachment') and target resource ('to a TestRail test case'), making the purpose immediately understandable. However, it doesn't differentiate from potential sibling tools like 'add_case' or 'add_result' that also involve adding content to TestRail, leaving room for confusion about when to choose this specific tool.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives. With sibling tools like 'add_case' (for creating cases) and 'add_result' (for adding test results), there's no indication of whether this is for supplementary files, required documentation, or specific contexts like bug reports. The agent must infer usage from the name alone.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

add_caseAdd TestRail CaseA

Create a new TestRail test case in a specific section. IMPORTANT: Before creating a case, gather required information using get_projects, get_suites, get_sections, and get_case_fields tools to ensure proper section_id, type_id, and custom field values. Or ask the user to provide the information if not provided.

ParametersJSON Schema
NameRequiredDescriptionDefault
titleYesTest case title - should be descriptive and unique within the section
section_idYesSection ID where the case will be created. REQUIRED: Use get_sections tool first to find valid section IDs for your project/suite. Different projects have different section structures.
type_idNoTest case type ID (e.g., 1=Acceptance, 2=Accessibility, 3=Automated, 4=Compatibility, 5=Destructive, 6=Functional, 7=Other, 8=Performance, 9=Regression, 10=Security, 11=Smoke & Sanity, 12=Usability). RECOMMENDED: Use get_cases tool to see what type_id values are used in existing cases in your target section.
priority_idNoPriority ID (1=Low, 2=Medium, 3=High, 4=Critical). RECOMMENDED: Use get_cases tool to see what priority_id values are used in existing cases.
refsNoReferences (e.g., requirement IDs, JIRA tickets, user story numbers). Can be comma-separated for multiple references.
customNoCustom fields (key-value pairs). REQUIRED: Use get_case_fields tool first to discover available custom fields and their valid values. Common fields include: custom_automation_type (1=None, 2=Playwright, 3=ChatGPT, 4=Non-Automated, 5=Partial), custom_environment (1=UAT Only, 2=UAT/Prod, 3=Demo UAT, 4=Live UAT), custom_preconds (preconditions text), custom_steps (test steps text), custom_expected (expected results text). Some custom fields are required by the project configuration.

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden. It clearly indicates this is a creation/mutation operation ('Create a new TestRail test case'), mentions important prerequisites (gathering data from other tools), and hints at complexity (custom fields may be required by project configuration). However, it doesn't explicitly mention authentication needs, rate limits, or what happens on failure.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is appropriately sized (3 sentences) and front-loaded with the core purpose. Every sentence adds value: first states what the tool does, second provides crucial prerequisite guidance, third offers alternative approach. Could be slightly more concise by combining the last two sentences.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a creation tool with 6 parameters (including nested objects) and no annotations/output schema, the description does well by emphasizing prerequisites and complexity. However, it doesn't describe what the tool returns (no output schema exists) or potential error conditions, leaving some gaps for a mutation operation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all 6 parameters thoroughly. The description doesn't add significant parameter semantics beyond what's in the schema - it references the same tools (get_sections, get_case_fields) that the schema descriptions mention. Baseline 3 is appropriate when schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description explicitly states 'Create a new TestRail test case in a specific section' - a clear verb ('Create') + resource ('TestRail test case') + location constraint ('in a specific section'). This distinguishes it from sibling tools like 'update_case' (modification) and 'get_case' (retrieval).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit guidance on when to use this tool: 'Before creating a case, gather required information using get_projects, get_suites, get_sections, and get_case_fields tools' and 'Or ask the user to provide the information if not provided.' It names specific prerequisite tools and offers an alternative approach when data is missing.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

add_resultAdd TestRail ResultC

Adds a new test result, comment, or assigns a test.

ParametersJSON Schema
NameRequiredDescriptionDefault
test_idYesThe ID of the test to which the result should be added
status_idYesThe ID of the test status (1=Passed, 2=Blocked, 4=Retest, 5=Failed)
commentNoThe comment or description for the test result
versionNoThe version or build against which the test was executed
elapsedNoThe time it took to execute the test (e.g., "30s" or "1m 45s")
defectsNoA comma-separated list of defects to link to the test result
assignedto_idNoThe ID of a user to whom the test should be assigned
custom_step_resultsNoArray of step results for structured testing
customNoCustom fields with custom_ prefix

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden of behavioral disclosure. While 'adds' implies a write operation, it doesn't specify permissions required, whether it's idempotent, error handling, or side effects (e.g., notifications). For a mutation tool with 9 parameters, this lack of behavioral context is a significant gap.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence that front-loads the core functionality without unnecessary words. Every part ('adds a new test result, comment, or assigns a test') directly contributes to understanding the tool's purpose, making it well-structured and concise.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a mutation tool with 9 parameters, no annotations, and no output schema, the description is incomplete. It lacks behavioral details (e.g., permissions, side effects), usage guidelines relative to siblings, and any information about return values or errors, leaving the agent with insufficient context for reliable invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description mentions 'test result, comment, or assigns a test', which loosely maps to parameters like status_id, comment, and assignedto_id, but adds minimal semantic value beyond what's in the schema. Baseline 3 is appropriate when schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose with specific verbs ('adds', 'assigns') and resources ('test result', 'comment', 'test'), making it easy to understand what the tool does. However, it doesn't explicitly differentiate from sibling tools like 'update_test' or 'add_case', which could also modify test-related data.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives. With sibling tools like 'update_test' and 'add_case' available, there's no indication of whether this is for initial results, updates, or specific scenarios, leaving the agent to guess based on context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_caseGet TestRail CaseC

Fetch a TestRail test case by ID.

ParametersJSON Schema
NameRequiredDescriptionDefault
case_idYesTestRail case ID

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden of behavioral disclosure. It states 'fetch', implying a read operation, but lacks details on permissions, rate limits, error handling, or response format. For a tool with no annotations, this leaves significant gaps in understanding its behavior and constraints.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence that directly states the tool's purpose without any unnecessary words. It is front-loaded and appropriately sized, making it easy to parse quickly while conveying the essential action.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the lack of annotations and output schema, the description is incomplete. It doesn't explain what data is returned (e.g., case details, fields), potential errors, or how it differs from sibling tools. For a retrieval tool in a context with multiple similar tools, more context is needed to ensure proper usage.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The description mentions 'by ID', which aligns with the single parameter 'case_id' in the schema. With 100% schema description coverage, the schema already documents the parameter as 'TestRail case ID' with type and constraints, so the description adds minimal value beyond reinforcing the ID-based lookup, meeting the baseline for high schema coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb 'fetch' and the resource 'TestRail test case by ID', making the purpose specific and understandable. However, it doesn't explicitly distinguish this tool from similar siblings like 'get_cases' (plural) or 'get_test', which also retrieve test-related data, leaving some ambiguity about when to use this specific tool versus others.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives. With siblings like 'get_cases', 'get_test', and 'get_case_fields', there is no indication of context, prerequisites, or exclusions, such as whether this is for single-case retrieval versus bulk operations or other related queries.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_case_fieldsGet TestRail Case FieldsB

Returns a list of available test case custom fields.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

B3.2/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden for behavioral disclosure. It states it 'Returns a list,' implying a read-only operation, but doesn't specify if it requires authentication, has rate limits, returns paginated results, or what format the list takes (e.g., JSON array of field objects). For a tool with zero annotation coverage, this is insufficient transparency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence that front-loads the core purpose ('Returns a list...') with no wasted words. Every part of the sentence earns its place by specifying what is returned and for what resource, making it optimally concise and well-structured.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity (0 parameters, no output schema, no annotations), the description is minimally adequate. It states what the tool does but lacks context on usage, behavior, or output format. For a read operation with no complex inputs, this is acceptable but leaves gaps that could hinder an agent's effective use, especially without annotations to fill in behavioral details.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has 0 parameters with 100% coverage, so the schema fully documents the lack of inputs. The description doesn't need to add parameter details, and it correctly implies no parameters are required by not mentioning any. This meets the baseline for zero-parameter tools, though it doesn't explicitly state 'no parameters needed,' which would have warranted a 5.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb ('Returns') and resource ('list of available test case custom fields'), making the purpose immediately understandable. It distinguishes this from siblings like get_case or get_cases by specifying it returns custom fields rather than cases themselves. However, it doesn't explicitly contrast with all siblings, so it falls short of a perfect 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., needing a project context), when it's appropriate (e.g., before creating cases with custom fields), or what siblings might be better for related tasks (like get_case for case details). This leaves the agent without contextual usage direction.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_casesGet TestRail CasesB

Get a list of test cases for a project or specific test suite with optional filtering and pagination.

ParametersJSON Schema
NameRequiredDescriptionDefault
project_idYesTestRail project ID
suite_idNoTestRail suite ID (optional if project is in single suite mode)
created_afterNoOnly return test cases created after this date (as UNIX timestamp)
created_beforeNoOnly return test cases created before this date (as UNIX timestamp)
created_byNoA list of creator user IDs to filter by
filterNoOnly return cases with matching filter string in the case title
limitNoThe number of test cases to return (max 250, default 250)
milestone_idNoA list of milestone IDs to filter by
offsetNoWhere to start counting the test cases from (pagination offset)
priority_idNoA list of priority IDs to filter by
refsNoA single Reference ID (e.g. TR-1, 4291, etc.)
section_idNoThe ID of a test case section
template_idNoA list of template IDs to filter by
type_idNoA list of case type IDs to filter by
updated_afterNoOnly return test cases updated after this date (as UNIX timestamp)
updated_beforeNoOnly return test cases updated before this date (as UNIX timestamp)
updated_byNoA user ID who updated test cases to filter by
label_idNoA list of label IDs to filter by

TDQS

B3.3/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It mentions 'optional filtering and pagination,' which hints at behavior, but doesn't disclose critical details like whether this is a read-only operation, potential rate limits, authentication needs, or what the return format looks like (e.g., list structure, error handling). For a tool with 18 parameters and no annotation coverage, this is a significant gap.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, well-structured sentence that efficiently conveys the core purpose and key features (filtering, pagination) without unnecessary words. It's front-loaded and every part earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the complexity (18 parameters, no annotations, no output schema), the description is incomplete. It covers the basic purpose but lacks behavioral details (e.g., read-only nature, response format) and deeper usage guidance. The high schema coverage helps, but for a tool with many parameters and no output schema, more context would be beneficial.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all 18 parameters thoroughly. The description adds minimal value beyond the schema by mentioning 'optional filtering and pagination,' which loosely maps to some parameters like 'limit' and 'offset,' but doesn't provide additional syntax, format, or usage details. Baseline 3 is appropriate when the schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Get a list of test cases for a project or specific test suite with optional filtering and pagination.' It specifies the verb ('Get'), resource ('test cases'), and scope ('project or specific test suite'), but doesn't explicitly distinguish it from sibling tools like 'get_case' (singular) or 'get_sections'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage context by mentioning 'project or specific test suite' and 'optional filtering and pagination,' but doesn't provide explicit guidance on when to use this tool versus alternatives like 'get_case' (for a single case) or 'get_sections' (for sections). No exclusions or prerequisites are stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_projectGet TestRail ProjectB

Get details for a specific TestRail project by ID.

ParametersJSON Schema
NameRequiredDescriptionDefault
project_idYesTestRail project ID

TDQS

B3.1/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It states it 'Get details' but doesn't clarify if this is a read-only operation, what permissions are required, or what the response format includes (e.g., project name, settings). This leaves significant gaps for a tool that likely interacts with a database or API.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence with no wasted words. It is front-loaded with the core action and resource, making it easy to parse quickly.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's low complexity (single parameter, no output schema, no annotations), the description is minimally adequate. It covers the basic purpose but lacks details on behavior, usage context, and output, which are needed for full understanding in the absence of annotations.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has 100% description coverage, with the parameter 'project_id' well-documented as a positive integer. The description adds minimal value beyond the schema by specifying 'by ID', but it doesn't explain format constraints or provide examples, so it meets the baseline for high schema coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb ('Get details') and resource ('specific TestRail project by ID'), making the purpose unambiguous. However, it doesn't explicitly differentiate from sibling tools like 'get_projects' (plural) or 'get_case', which also retrieve details but for different resources.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives. It doesn't mention sibling tools like 'get_projects' (for listing projects) or specify prerequisites such as needing a valid project ID, leaving usage context implied rather than explicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_projectsGet TestRail ProjectsB

List all TestRail projects.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

B3.1/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries full burden but only states the action without behavioral details. It doesn't disclose whether this is paginated, requires authentication, has rate limits, returns structured data, or what happens with large project sets—critical for a list operation.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence with zero waste—front-loaded and to the point. Every word contributes directly to stating the tool's purpose without redundancy or fluff.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no annotations, no output schema, and a simple list operation, the description is incomplete. It lacks details on return format, pagination, error handling, or how it fits with siblings like 'get_project', leaving gaps for an agent to use it effectively in context.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has 0 parameters with 100% schema coverage, so no parameter documentation is needed. The description doesn't add parameter info, which is appropriate, but it also doesn't imply any hidden parameters or constraints, keeping it straightforward.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb ('List') and resource ('TestRail projects'), making the purpose immediately understandable. However, it doesn't differentiate from sibling tools like 'get_project' (singular) or explain scope beyond 'all', missing full sibling distinction.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided on when to use this tool versus alternatives like 'get_project' (singular) or 'get_cases' (which might be project-specific). The description implies a broad listing but offers no context about prerequisites, filtering, or comparison to siblings.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_runGet TestRail RunB

Returns an existing test run. Please see get tests for the list of included tests in this run.

ParametersJSON Schema
NameRequiredDescriptionDefault
run_idYesThe ID of the test run

TDQS

B3.3/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It states this is a read operation ('Returns'), which is clear, but doesn't disclose behavioral traits such as error handling (e.g., what happens if the run_id is invalid), authentication needs, rate limits, or response format. The reference to 'get tests' adds some context but is insufficient for a mutation-free tool with no annotation coverage.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences, front-loaded with the core purpose and efficiently references another tool for additional context. Every sentence earns its place with no wasted words, making it appropriately sized and easy to parse.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity (1 parameter, no output schema, no annotations), the description is minimally adequate. It covers the basic purpose and points to related information, but as a read tool with no annotations, it should ideally include more on response behavior or error cases to be fully complete for agent use.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, with the single parameter 'run_id' well-documented in the schema as 'The ID of the test run'. The description doesn't add any meaning beyond this, such as format examples or constraints not in the schema. Baseline 3 is appropriate since the schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb ('Returns') and resource ('an existing test run'), making the purpose understandable. However, it doesn't explicitly differentiate from sibling tools like 'get_runs' (plural) or 'get_test', which could retrieve similar resources. The mention of 'get tests' is helpful but not a direct sibling distinction.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage by referencing 'get tests' for related information, suggesting it's for retrieving a specific run's details. However, it lacks explicit guidance on when to use this versus alternatives like 'get_runs' (for multiple runs) or 'get_test' (for individual tests), and doesn't mention prerequisites or exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_runsGet TestRail RunsA

Get a list of test runs for a project with optional filtering and pagination. Only returns test runs that are not part of a test plan.

ParametersJSON Schema
NameRequiredDescriptionDefault
project_idYesTestRail project ID
created_afterNoOnly return test runs created after this date (as UNIX timestamp)
created_beforeNoOnly return test runs created before this date (as UNIX timestamp)
created_byNoA comma-separated list of creators (user IDs) to filter by
is_completedNo1 to return completed test runs only. 0 to return active test runs only
limitNoThe number of test runs to return (max 250, default 250)
milestone_idNoA comma-separated list of milestone IDs to filter by
offsetNoWhere to start counting the test runs from (pagination offset)
refs_filterNoA single Reference ID (e.g. TR-a, 4291, etc.)
suite_idNoA comma-separated list of test suite IDs to filter by

TDQS

A3.5/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions 'optional filtering and pagination,' which hints at functionality, but lacks details on permissions, rate limits, error handling, or the return format (e.g., structure of the list). For a tool with 10 parameters and no annotations, this leaves significant gaps in understanding its behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence that front-loads the core purpose ('Get a list of test runs for a project') and adds key constraints ('with optional filtering and pagination. Only returns test runs that are not part of a test plan.'). Every part earns its place without redundancy or waste.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the complexity (10 parameters, no annotations, no output schema), the description is adequate but incomplete. It covers the basic purpose and a key constraint (excludes test plan runs), but lacks details on behavioral aspects like response format, error cases, or usage nuances. This makes it minimally viable but with clear gaps for effective tool invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, meaning all parameters are documented in the input schema. The description adds minimal value beyond the schema by mentioning 'optional filtering and pagination,' which aligns with parameters like created_after, limit, and offset, but doesn't provide additional syntax or usage details. This meets the baseline of 3 when schema coverage is high.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb ('Get a list of') and resource ('test runs for a project'), making the purpose specific and understandable. It distinguishes this tool from 'get_run' (singular) by indicating it returns multiple runs, but doesn't explicitly differentiate from other list tools like 'get_cases' or 'get_tests' beyond the resource type.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context by specifying 'Only returns test runs that are not part of a test plan,' which helps guide when to use this tool. However, it doesn't explicitly mention when to use alternatives like 'get_run' (for a single run) or other filtering tools, nor does it state any prerequisites beyond the required project_id.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_sectionsGet TestRail SectionsC

Get a list of sections for a project and test suite with optional pagination.

ParametersJSON Schema
NameRequiredDescriptionDefault
project_idYesTestRail project ID
suite_idNoTestRail suite ID (optional if project is in single suite mode)
limitNoThe number of sections to return (max 250, default 250)
offsetNoWhere to start counting the sections from (pagination offset)

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions 'optional pagination' which hints at list behavior, but doesn't describe return format, error conditions, rate limits, authentication needs, or what happens with invalid inputs. For a read operation with 4 parameters, this lacks critical operational context.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, well-structured sentence that efficiently conveys the core action, scope, and key feature. It's front-loaded with the main purpose and avoids redundancy or unnecessary details. Every word earns its place, making it highly concise and readable.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given 4 parameters, no annotations, and no output schema, the description is incomplete. It lacks details on return values, error handling, and practical usage scenarios. For a tool that fetches structured data, more context is needed to help an agent use it effectively, especially without annotations to fill gaps.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema fully documents all 4 parameters. The description adds minimal value beyond the schema, mentioning 'optional pagination' which loosely relates to 'limit' and 'offset', but doesn't clarify semantics like the relationship between project_id and suite_id or default behaviors. Baseline 3 is appropriate as the schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb ('Get a list of') and resource ('sections'), specifying the scope ('for a project and test suite') and optional feature ('with optional pagination'). It distinguishes from siblings like 'get_cases' or 'get_suites' by focusing on sections, but doesn't explicitly contrast with them. The purpose is specific and actionable, though not maximally differentiated.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives. It mentions the context ('for a project and test suite') but doesn't specify use cases, prerequisites, or exclusions. With siblings like 'get_cases' or 'get_suites' available, there's no indication of when sections are needed over those resources, leaving usage ambiguous.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_suiteGet TestRail SuiteC

Get details for a specific TestRail test suite by ID.

ParametersJSON Schema
NameRequiredDescriptionDefault
suite_idYesTestRail suite ID

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It states it 'gets details' but doesn't specify what details are returned (e.g., suite name, description, project association), whether it's a read-only operation, or any error handling (e.g., for invalid IDs). This leaves significant gaps in understanding the tool's behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, clear sentence that efficiently conveys the core purpose without unnecessary words. It's front-loaded and wastes no space, making it easy for an agent to parse quickly.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the lack of annotations and output schema, the description is incomplete. It doesn't explain what 'details' are returned, which is critical for a tool with no structured output documentation. For a simple read operation, more context on the response format would help the agent use it effectively.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The description mentions 'by ID', which aligns with the single parameter 'suite_id' in the input schema. Since schema description coverage is 100%, the schema already documents the parameter as 'TestRail suite ID' with type and constraints. The description adds minimal value beyond this, meeting the baseline for high schema coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Get details') and resource ('TestRail test suite by ID'), making the purpose immediately understandable. However, it doesn't differentiate from sibling tools like 'get_suites' (plural) or 'get_case' which suggests this is for a single suite, but this distinction isn't explicitly stated.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives like 'get_suites' (for listing multiple suites) or other 'get_' tools for different resources. It lacks context about prerequisites, such as needing a valid suite ID, or exclusions, leaving the agent to infer usage from the tool name alone.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_suitesGet TestRail SuitesB

Get all test suites for a specific TestRail project by ID.

ParametersJSON Schema
NameRequiredDescriptionDefault
project_idYesTestRail project ID

TDQS

B3.3/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It states the action ('Get all test suites') but lacks behavioral details such as whether this is a read-only operation, if it requires authentication, any rate limits, pagination behavior, or error handling. For a tool with zero annotation coverage, this is a significant gap in disclosing operational traits.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence that front-loads the core purpose ('Get all test suites') and specifies the context ('for a specific TestRail project by ID'). There is zero waste, making it highly concise and well-structured for quick understanding.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's low complexity (one parameter, no output schema, no annotations), the description is minimally adequate. It covers the basic purpose and parameter context, but lacks completeness in behavioral aspects like safety or output details, which are important for a tool with no annotations or output schema to guide the agent.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema description coverage is 100%, with the single parameter 'project_id' documented as 'TestRail project ID'. The description adds no additional meaning beyond this, such as format examples or constraints not in the schema. With high schema coverage, the baseline score of 3 is appropriate as the description doesn't compensate but doesn't detract either.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb ('Get') and resource ('all test suites'), specifying the scope ('for a specific TestRail project by ID'). It distinguishes from siblings like 'get_suite' (singular) by indicating it retrieves multiple suites, though it doesn't explicitly contrast with other list tools like 'get_cases' or 'get_projects' beyond the resource type.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage when needing suites for a project, but provides no explicit guidance on when to use this versus alternatives like 'get_suite' (for a single suite) or other list tools. It mentions the required 'project_id' parameter, which hints at prerequisites, but lacks context on exclusions or comparisons with siblings.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_testGet TestRail TestC

Returns an existing test.

ParametersJSON Schema
NameRequiredDescriptionDefault
test_idYesThe ID of the test
with_dataNoThe parameter to get data

TDQS

C2.7/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool returns data (a read operation), but doesn't cover aspects like authentication needs, rate limits, error handling, or what the return format looks like (e.g., JSON structure). For a read tool with zero annotation coverage, this is a significant gap in transparency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise—a single sentence with zero waste. It's front-loaded with the core action ('Returns'), making it easy to scan. Every word earns its place, though this brevity contributes to gaps in other dimensions like guidelines and transparency.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (a read operation with 2 parameters) and lack of annotations and output schema, the description is incomplete. It doesn't explain what 'test' data is returned, how to interpret 'with_data,' or any behavioral traits. For a tool with no structured output and full parameter reliance on the schema, more context is needed to be fully helpful.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The description adds no meaning beyond what the input schema provides. Schema description coverage is 100%, with clear documentation for 'test_id' (ID of the test) and 'with_data' (parameter to get data). Since the schema does the heavy lifting, the baseline score of 3 is appropriate, as the description doesn't compensate or add extra context.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose3/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description 'Returns an existing test' states the basic action (return) and resource (test), but it's vague—it doesn't specify what a 'test' entails in TestRail (e.g., test case details, results) or differentiate it from sibling tools like 'get_tests' (plural). It avoids tautology by not just restating the name, but lacks specificity.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided on when to use this tool versus alternatives. For example, it doesn't explain if this is for retrieving a single test by ID (vs. 'get_tests' for multiple tests) or mention prerequisites like needing a valid test ID. The description implies usage only through the action 'returns,' but no explicit context or exclusions are stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_testsGet TestRail TestsC

Returns a list of tests for a test run.

ParametersJSON Schema
NameRequiredDescriptionDefault
run_idYesThe ID of the test run
status_idNoA comma-separated list of status IDs to filter by
limitNoThe number that sets the limit of tests to be shown on the response (max 250, default 250)
offsetNoThe number that sets the position where the response should start from (pagination offset)
label_idNoIDs of labels as comma separated values to filter by

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden for behavioral disclosure but offers minimal information. It doesn't mention authentication requirements, rate limits, pagination behavior beyond parameters, response format, error conditions, or whether this is a read-only operation (though 'Returns' implies it). For a tool with 5 parameters and no annotation coverage, this is inadequate.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence that states the core purpose without unnecessary words. It's appropriately sized for a list-retrieval tool and front-loads the essential information. Every word earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 5 parameters, no annotations, and no output schema, the description is insufficiently complete. It doesn't address key contextual aspects like authentication needs, rate limits, pagination behavior, response format, or error handling. The agent would need to guess about important behavioral characteristics when invoking this tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description adds no additional parameter semantics beyond what's in the schema - it doesn't explain relationships between parameters (like how status_id and label_id interact) or provide examples. This meets the baseline for high schema coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Returns a list') and resource ('tests for a test run'), making the purpose immediately understandable. It distinguishes from sibling tools like 'get_test' (singular) by specifying it returns multiple tests, but doesn't explicitly differentiate from other list tools like 'get_cases' or 'get_runs' beyond the resource type.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives. It doesn't mention when filtering by status or label is appropriate, when pagination is needed, or how this differs from similar tools like 'get_cases' which might return related data. The agent must infer usage from parameter names alone.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

update_caseUpdate TestRail CaseC

Update a TestRail test case by ID with new field values.

ParametersJSON Schema
NameRequiredDescriptionDefault
case_idYesTestRail case ID
titleNoTest case title
section_idNoSection ID
type_idNoTest case type ID
priority_idNoPriority ID
refsNoReferences (e.g., requirement IDs)
customNoCustom fields (key-value pairs)

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It states this is an update operation, implying mutation, but doesn't mention permissions required, whether changes are reversible, rate limits, error handling, or what the response looks like (since there's no output schema). This leaves significant gaps in understanding the tool's behavior beyond the basic action.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence that front-loads the core purpose without unnecessary words. Every part ('Update a TestRail test case by ID with new field values') contributes directly to understanding the tool, making it appropriately sized and well-structured for quick comprehension.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (7 parameters, mutation operation, no annotations, no output schema), the description is insufficient. It lacks details on behavioral traits (e.g., side effects, authentication), doesn't explain return values or errors, and provides minimal context for usage. For a mutation tool with rich parameters, this leaves the agent under-informed about critical aspects.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all 7 parameters thoroughly. The description adds no additional meaning beyond implying that parameters represent 'new field values,' which is redundant with the schema. This meets the baseline of 3 for high schema coverage, but doesn't compensate with extra insights like field interdependencies or examples.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Update'), resource ('TestRail test case'), and mechanism ('by ID with new field values'), which is specific and unambiguous. However, it doesn't explicitly differentiate from sibling tools like 'update_run' or 'update_test', which also update TestRail entities, leaving some ambiguity about when to choose this specific tool.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives like 'add_case' (for creation) or other update tools (e.g., 'update_run'). It mentions updating by ID but doesn't specify prerequisites (e.g., needing an existing case ID) or exclusions (e.g., not for bulk updates), leaving usage context implied rather than explicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

update_runUpdate TestRail RunC

Updates an existing test run. Partial updates are supported.

ParametersJSON Schema
NameRequiredDescriptionDefault
run_idYesThe ID of the test run to be updated
nameNoThe name of the test run
descriptionNoThe description of the test run
milestone_idNoThe ID of the milestone
include_allNoTrue for including all test cases and false for a custom case selection
case_idsNoAn array of case IDs for the custom case selection
configNoA comma-separated list of configuration IDs
config_idsNoAn array of configuration IDs
refsNoA string of external requirements
start_onNoThe start date (Unix timestamp)
due_onNoThe due date (Unix timestamp)
customNoCustom fields (key-value pairs)

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden for behavioral disclosure. While it mentions partial updates are supported (useful context), it doesn't address critical aspects like required permissions, whether changes are reversible, rate limits, or what happens to unspecified fields during partial updates. For a mutation tool with 12 parameters, this leaves significant gaps.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise with just two sentences that both earn their place. The first sentence states the core purpose, and the second adds important behavioral context about partial updates. No wasted words or redundant information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a mutation tool with 12 parameters, no annotations, and no output schema, the description is insufficient. It doesn't explain what the tool returns, error conditions, authentication requirements, or how it differs from similar update tools. The mention of partial updates is helpful but doesn't compensate for the broader gaps.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The description mentions 'partial updates are supported' which provides context about how parameters work together, but doesn't add specific meaning to individual parameters beyond what's already in the schema (which has 100% coverage). The baseline of 3 is appropriate since the schema does the heavy lifting of documenting all parameters.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Updates') and resource ('an existing test run'), and specifies that partial updates are supported. However, it doesn't differentiate this tool from sibling tools like 'update_case' or 'update_test' beyond the resource type, which prevents a perfect score.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives like 'update_case' or 'update_test'. It mentions partial updates are supported, but doesn't explain when this might be preferred over creating a new run or using other update tools.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

update_testUpdate TestRail TestC

Updates the labels assigned to an existing test.

ParametersJSON Schema
NameRequiredDescriptionDefault
test_idYesThe ID of the test to be updated
labelsNoThe ID of a label, the title of a label or both, in array form
customNoCustom fields (key-value pairs)

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden for behavioral disclosure. It states this is an update operation (implying mutation) but doesn't cover critical aspects like required permissions, whether changes are reversible, rate limits, or what happens to existing labels not mentioned. For a mutation tool with zero annotation coverage, this is inadequate.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence that directly states the tool's purpose without unnecessary words. It's appropriately sized and front-loaded, with every word earning its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a mutation tool with no annotations and no output schema, the description is insufficient. It doesn't explain what the update operation returns, error conditions, or behavioral constraints. Given the complexity (3 parameters including nested objects) and lack of structured coverage, more context is needed for effective agent use.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema fully documents all three parameters. The description mentions 'labels' but doesn't add meaningful semantic context beyond what the schema provides (e.g., explaining label types or custom field usage). Baseline 3 is appropriate when the schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Updates') and resource ('labels assigned to an existing test'), providing a specific verb+resource combination. However, it doesn't explicitly differentiate from sibling tools like 'update_case' or 'update_run', which likely update different aspects of TestRail entities.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., needing an existing test), exclusions, or comparisons to sibling tools like 'update_case' or 'update_run', leaving the agent to infer usage context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. 18 tool updatesv1.0.0
    • First observedadd_attachment_to_case
    • First observedadd_case
    • First observedadd_result
    • First observedget_case
    • First observedget_case_fields
    • First observedget_cases
    • First observedget_project
    • First observedget_projects
    • First observedget_run
    • First observedget_runs
    • First observedget_sections
    • First observedget_suite
    • First observedget_suites
    • First observedget_test
    • First observedget_tests
    • First observedupdate_case
    • First observedupdate_run
    • First observedupdate_test

TDQS

B3.4/5.0
Disambiguation4/5

Most tools have distinct purposes targeting specific TestRail resources (cases, runs, tests, projects, suites, sections, fields), but there is some overlap between get_case/get_cases and get_test/get_tests that could cause confusion if an agent misinterprets singular vs. plural usage. The descriptions help clarify, but the naming similarity creates minor ambiguity.

Naming Consistency5/5

All tool names follow a consistent verb_noun pattern with snake_case, using clear action verbs like get, add, and update paired with specific nouns (e.g., get_case, add_attachment_to_case, update_run). This uniformity makes the tool set predictable and easy to navigate.

Tool Count4/5

With 18 tools, the count is slightly high but reasonable for a comprehensive TestRail API coverage, including CRUD operations for cases, runs, tests, and supporting resources like projects and suites. It feels slightly heavy but well-scoped to the domain without being excessive.

Completeness5/5

The tool set provides complete CRUD/lifecycle coverage for TestRail's core domain, including projects, suites, sections, cases, runs, tests, and attachments. It supports creation (add_case), retrieval (get_* tools), updates (update_* tools), and result management (add_result), with no obvious gaps for essential workflows.

Maintenance

ActivityInactive
ResponsivenessSyncing

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

  • A
    license
    B
    quality
    D
    maintenance
    Enables AI assistants to interact directly with TestRail instances for managing test projects, suites, cases, runs, results, plans, milestones, and attachments through the TestRail API with secure authentication.
    77
    200
    1
    MIT
  • A
    license
    A
    quality
    F
    maintenance
    Enables management of TestRail projects, test cases, runs, and results directly through MCP-supported clients. It provides a comprehensive set of tools to interact with the TestRail API for seamless test cycle management within AI environments.
    42
    1,580
    44
    MIT
  • A
    license
    B
    quality
    D
    maintenance
    Enables AI assistants to interact with TestRail test management system, supporting full CRUD operations on projects, suites, sections, test cases, runs, results, plans, and milestones.
    35
    3,508
    1
    MIT

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/Derrbal/testrail-mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server