Server Quality Checklist

Tool Scores

• get_repo_healthA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint, idempotentHint, and destructiveHint, but the description adds valuable context: side effects are explicitly none, auth requirements are stated (public repos need no auth, token used if available), rate limits are quantified, and the underlying endpoint is identified. No contradictions with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-organized with the primary purpose front-loaded and the rest in scannable bullet points. However, the 'Return shape' section largely duplicates the output schema, which adds length without new information. The sibling exclusions are valuable and justify the overall length.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers data source, auth requirements, rate limits, return shape, and explicit routing to alternatives. Combined with the annotations and output schema, nothing critical is missing for an agent to select and invoke this tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% coverage with clear descriptions for both 'owner' and 'repo' parameters. The description adds no parameter-specific meaning beyond what the schema already provides, so the baseline 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Fetches') and resource ('basic repository metadata and statistics from the public GitHub API'), clearly identifying what the tool does. The usage guidelines further differentiate it from siblings by noting it returns raw metadata rather than a computed health score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use the tool ('ONLY to retrieve basic raw metadata') and when not to, naming each sibling tool and the exact purpose for which it should be used instead. This leaves no ambiguity about tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• analyze_code_scanningA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the annotations, the description fully discloses the conditional side effect: 'Read-only by default. If trigger_scan=true, writes to GitHub Actions by creating a workflow_dispatch event.' It also adds auth requirements, rate limit awareness, and return shape, which is substantial behavioral context. The annotations are consistent: readOnlyHint=false correctly reflects the optional write behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is structured with clear bullets and front-loads the core purpose before side effects, data sources, auth, return shape, and usage guidance. Every section earns its place and is written efficiently for an agent to consume.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 5 parameters, annotations, an output schema, and sibling tools, the description is exceptionally complete. It covers side effects, auth, rate limits, return shape, polling behavior, and explicit routing to alternatives. No critical operational context is missing for an agent to select and invoke this tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 100% parameter description coverage, so the baseline is appropriately 3. The description adds minimal extra parameter meaning beyond what the schema already provides, though the side-effect explanation for trigger_scan is helpful. It does not meaningfully enhance understanding of owner, repo, poll_timeout_seconds, or poll_interval_seconds beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's specific verb and resource: 'Fetches or triggers open Code Scanning (CodeQL) alerts for a GitHub repository.' This distinguishes it from the available sibling tools, especially when combined with the explicit usage guidelines naming alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly narrows usage to 'deep static code vulnerability scanning (CodeQL)' and gives direct exclusions with sibling alternatives: analyze_dependencies for package vulnerabilities, get_health_score for A-F grading, and check_ci_status for CI/CD statuses. This is explicit when-to-use and when-not-to-use guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• analyze_dependenciesA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already mark the tool as read-only and non-destructive, but the description adds valuable behavioral context: no side effects, data source (GitHub REST API dependabot/alerts), auth requirements, rate limits, and return shape. No contradiction exists between the description and annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose and then uses compact labeled bullets for side effects, data source, auth, rate limits, return shape, and usage guidelines. Every sentence adds useful information without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite lacking an output schema, the description explicitly states the return shape including fields like summary, severity, package_name, state, and html_url. It also covers auth, rate limits, and sibling tool exclusions, making it complete for an agent to invoke correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 67%, so the schema already documents most parameters. The description does not add parameter-level guidance, such as how the severity parameter filters results. It mentions severity only as a return field, which could be slightly ambiguous, so the description provides no meaningful parameter clarification beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Fetches Dependabot alerts for a GitHub repository to analyze vulnerable package dependencies.' It clearly distinguishes this tool from siblings like analyze_code_scanning by scoping it to vulnerable package dependencies rather than static code vulnerabilities.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use the tool ('ONLY to find vulnerable package dependencies') and when not to use it, naming alternatives: analyze_code_scanning for CodeQL and get_health_score for health grading. This provides unambiguous routing guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_ci_statusA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description thoroughly discloses side effects, data sources, auth requirements, rate limits, and return shape. This goes well beyond the annotations, which only indicate readOnly, openWorld, idempotent, and non-destructive behavior. No contradiction with annotations exists.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a front-loaded purpose, followed by concise bullet points for side effects, data sources, auth, rate limits, return shape, and usage guidelines. Each section adds distinct value, and the exclusion list is efficient rather than redundant.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no output schema, the description compensates by specifying the return shape precisely. It also covers auth, rate limits, side effects, and alternatives, making the description fully complete for an agent to select and invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the input schema already fully documents the three parameters. The description adds context about the returned fields but does not add much meaning about the parameters themselves, which matches the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Fetches recent CI/CD workflow runs (GitHub Actions) for a GitHub repository.' It clearly distinguishes the tool from siblings by explicitly naming what it is for and what it is not for, such as get_health_score and get_dora_metrics.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage guidelines: 'Use this tool ONLY to check raw GitHub Actions workflow history and CI build statuses.' It then lists specific alternative tools for other analyses, leaving no ambiguity about when to choose this tool versus siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_dora_metricsA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the readOnly/idempotent/destructive annotations, the description adds concrete behavioral detail: no side effects, GitHub REST API data sources, public vs. private auth requirements, standard API rate limits, and heavy multi-endpoint API usage. This is exactly the kind of context that helps the agent predict cost and failure modes.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description uses compact bullet points with the core purpose first, followed by behavior, constraints, return shape, and usage routing. Each section adds distinct information; the only mild redundancy is 'Side effects: None' alongside 'strictly read-only,' but this does not meaningfully bloat the definition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a moderately complex metrics tool with an output schema, the description covers purpose, exclusions, auth, rate limits, data sources, side effects, and return shape. Combined with the fully described input schema and annotations, there are no significant gaps an agent would need to guess about.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, and each property (days, repo, owner) already has a clear schema description with defaults and examples. The tool description does not add parameter-specific detail, but with full schema coverage it has no obligation to; baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Calculates DORA proxy metrics (deployment frequency, lead time, change failure rate, MTTR) for a GitHub repository.' It names the exact metrics and scope, and the usage-guidelines section explicitly separates it from sibling tools, so an agent can distinguish it immediately.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives unambiguous when-to-use guidance: 'Use this tool ONLY to evaluate DORA metrics and team delivery performance.' It also provides explicit not-this-tool routing for check_ci_status, get_health_score, and get_repo_health, making the selection decision clear without requiring the agent to infer.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_health_scoreA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses side effects ('Writes a trend snapshot to local disk'), data sources (GitHub REST API and OpenSSF Scorecard API), auth requirements, and rate limits. It also details the exact return shape with category breakdown, suggestions, and historical trend data. This goes well beyond the sparse boolean annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is organized into labeled bullet sections (Side effects, Data sources, Auth requirements, Rate limits, Return shape, Usage guidelines) with the core purpose front-loaded. Every section conveys non-redundant, decision-relevant information, so the length is justified.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers purpose, alternatives, side effects, data sources, auth, rate limits, and return shape, leaving no operational gaps. The presence of an output schema further reduces the need to explain return values, yet the description still provides a helpful summary.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Both parameters repo and owner are fully described in the input schema with examples, yielding 100% schema description coverage. The description adds context about public vs. protected repositories but no additional parameter-level detail, so the baseline 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The opener 'Calculates a 0-100 health score and A-F grade for a GitHub repository' names a specific verb, resource, and output. The later guidance 'Use this tool ONLY for deep analytical grading and overall repository health assessment' explicitly differentiates it from siblings like get_repo_health and check_ci_status. Unambiguous and distinctive.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit when-to-use guidance: 'Use this tool ONLY for deep analytical grading... DO NOT use this tool for quick metadata checks.' It then lists four sibling alternatives with precise conditions, e.g., 'For basic raw metadata (stars, language, etc.), use get_repo_health instead.' No inference is needed.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• compare_reposA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description thoroughly discloses behavior beyond the annotations: explicitly states 'Side effects: None', specifies data sources (GitHub REST API and OpenSSF Scorecard API), explains auth requirements, and warns about rate-limit multiplication. This aligns with annotations (readOnlyHint=true, destructiveHint=false) and adds meaningful operational context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is structured with clear bullet points and front-loads the core purpose in the first sentence. It is longer than average, but nearly every line carries useful information about side effects, data sources, auth, rate limits, return shape, and usage boundaries. Minor redundancy exists between 'Side effects: None' and 'strictly read-only', but it does not detract significantly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool that calls external APIs and compares multiple repositories, the description is complete: it covers data sources, authentication, rate-limit implications, exact repository count limits, return shape, and clear routing to sibling tools. Nothing needed to invoke it correctly is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description carries the burden. It compensates by stating the accepted count range (2-5 repos), the comparison/ranking behavior, and the output shape. The nested owner/repo properties are documented in the schema itself, so the description adds enough context for correct parameter use.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Compares health scores of multiple GitHub repositories (2-5 repos) and ranks them.' It clearly identifies the operation and scope, and the explicit comparison/ranking focus distinguishes it from sibling tools like get_health_score and get_repo_health.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit when-to-use guidance: 'Use this tool ONLY when you need to compare or rank multiple repositories.' It also provides strong guardrails with 'DO NOT use this tool for analyzing a single repository' and names the correct alternatives: get_health_score for single-repo health and get_repo_health for raw metadata comparison.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.