Server Quality Checklist

Tool Scores

• get_case_queueB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate destructiveHint=true and readOnlyHint=false, but the description's 'Return...' reads as a safe read operation and adds no context about side effects, what may be destroyed, or why the tool is marked destructive. This mismatch makes the safety profile confusing and under-disclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is one short sentence with no filler. It front-loads the core purpose, and every word contributes to understanding what the tool returns.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The schema covers parameters and an output schema exists, so return structure is not the description's burden. However, the description is too thin to fully explain the disruptive destructive hint, the reconciliation context, or when an agent should prefer resolve_case, leaving the overall guidance minimally viable but gapped.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so repoPath and interactive are already documented in the schema. The description adds no extra parameter meaning beyond the schema and does not address the interactive behavior or its consequences.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Return unresolved ambiguity cases from reconciliation' uses a specific verb and resource, making the tool's main output clear. It is distinguishable from siblings like resolve_case, but it does not explicitly call out that distinction.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description only implies when to use the tool: when unresolved ambiguity cases from reconciliation need to be retrieved. It gives no guidance about alternatives such as resolve_case, nor any exclusions, leaving the agent to infer selection criteria from the name and schema.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• crawl_siteB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already cover read-only, idempotency, and destructive hints. The description adds some behavioral detail by noting it runs headless and emits periodic progress notifications, but it does not clarify what side effects the crawl may produce beyond visiting pages, even though readOnlyHint is false and repoPath suggests saving evidence. No contradiction with annotations was found.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two short sentences with no filler, and the core action is front-loaded. It is concise and readable, though it could have used the extra space to provide more usage context.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the schema fully documents all parameters and an output schema exists, the core technical details are covered. However, the description alone does not address when to use the tool, what side effects the crawl might have, or how it relates to the sibling tools. It is adequate but has clear gaps for an agent deciding whether to invoke it.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already explains url, maxPages, and repoPath. The description does add a small hint that the crawl follows reachable routes from the base URL, but it does not materially improve on the parameter descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the action ('crawl'), the resource ('site'), and the method ('Playwright headless'), and specifies the scope as 'reachable routes.' This distinguishes it from the sibling tools, which perform different operations like ingesting, flagging, or resolving.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to choose this tool over alternatives, no prerequisites, and no exclusions. The intended context is only implied by the word 'crawl,' not explicitly stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• resolve_caseA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already carry the safety profile (destructiveHint=true, idempotentHint=true), and the description adds the useful operational trait that the tool is always available and requires no elicitation capability. It does not, however, disclose what resolution actually changes (e.g., case status or removal from the queue), leaving the side effect only implied by the destructive hint.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two short sentences, front-loaded with the primary purpose and followed by a concise availability note. Every word earns its place; there is no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool benefits from rich annotations, 100% parameter documentation, and an output schema, so the description need not explain return values. Still, it omits the practical effect of resolving a case (e.g., the case disappearing from get_case_queue) and provides no guidance about when to prefer this over the closely related sibling flag_known_bug, leaving a small but real completeness gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema description coverage, the baseline is 3. The description adds the key semantic that the decision must be a human decision, which is not stated in the schema's decision property text and helps prevent an agent from fabricating a decision on its own. This one meaningful addition justifies a score above baseline.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: 'resolve one open case' with the key qualifier 'with a human decision.' It is not a tautology and clearly outlines the core action, but it does not explicitly contrast with sibling tools like flag_known_bug or get_case_queue, so it falls short of full differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'Always available, no elicitation capability required' gives some operational context about when the tool can be invoked, implying it is the standard path for resolving a case. However, it never names alternatives or conditions when another sibling should be used instead, so guidance is mostly implicit rather than explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ingest_repoA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already provide idempotentHint=true and destructiveHint=false. The description adds meaningful behavioral context with 'static analysis' and 'No LLM call', signaling deterministic, non-LLM execution beyond what annotations state.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two short sentences with no filler. The first states the operation and scope, and the second adds a key behavioral constraint. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is low complexity, has full schema coverage, an output schema, and annotations covering idempotency and destructiveness. The description supplies the remaining essential facts: what files are parsed and that no LLM call is made.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so both path and interactive are already well documented in the input schema. The description does not add parameter-specific meaning, which is acceptable given the schema already carries the burden.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb, 'Parse', and names concrete resources: package.json, tailwind/vite config, route files, and existing tests. An agent can tell what the tool operates on, though it does not explicitly contrast itself with siblings like generate_spec or crawl_site.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The static-analysis phrasing and 'No LLM call' imply this is a deterministic, lower-cost ingestion step, but the description does not explicitly say when to use this tool versus alternatives. Sibling names provide context, yet no direct routing or exclusion guidance is given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• flag_known_bugA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations only indicate that this is a non-read-only, non-idempotent mutation. The description adds the crucial non-obvious behavior that a flagged known bug always wins over auto-resolve regardless of evidence. This is valuable context that annotations cannot communicate. No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two short sentences with no filler. The primary action is front-loaded, followed immediately by the single most important behavioral rule. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple two-parameter write tool, the description covers the action and the essential override behavior, and the schema documents the parameters. An output schema exists, so return-value details are not needed. The only small gap is that when-to-use guidance is implied rather than explicit.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, and both parameters (repoPath and description) are already well documented in the schema. The main description adds no additional parameter semantics, so the baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action ('Record a known bug') and immediately supplies the core differentiator: it overrides auto-resolve. This distinguishes it from sibling resolution/auto-resolve tools without needing to inspect the schema.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The second sentence gives a clear behavioral context: use this when a known bug should supersede any auto-resolve conclusion, even when other evidence points elsewhere. It does not explicitly list when not to use it or name sibling tools, but the precedence rule strongly implies the intended usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• generate_specA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With annotations already marking this as destructive and non-read-only, the description adds substantial behavioral context: the tool is only callable with an empty case queue, the vision mode is off by default and changes nothing when unset, the tool never logs in or handles credentials itself, and the auth state file is copied into build output and gitignored. These details meaningfully extend beyond the annotation hints and help an agent predict side effects and prerequisites.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core action, then moves from precondition to optional enhancements in a logical order. Every sentence carries operational weight: the initial write target, the queue precondition, the vision-mode toggle and tradeoff, and the auth-state option. Although it is longer than a one-liner, the length is justified by the conditional behavior it must convey.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description, annotations, and rich schema collectively cover prerequisites, optional configurations, credential handling, side-effect locations, and output scope. Since an output schema exists, the description does not need to detail return values. There is no obvious gap an agent would need to guess about in order to call this tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the input schema already fully documents repoPath and authStorageStatePath. The tool description adds only a cross-reference to authStorageStatePath and an optional storage-state usage note, but does not go beyond what the schema fields themselves say. With high schema coverage, baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: it writes CLAUDE.md, .claude/, spec/, tests/, and kickoff-prompt.txt to a <repo>-rebuild/ directory. This clearly distinguishes it from sibling tools like ingest_repo or crawl_site, which perform other pipeline stages. The title alone would be vague, but the description removes all ambiguity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It states an explicit precondition: 'Only callable once the case queue is empty,' which tells the agent when it may and may not be invoked. It also provides conditional guidance for two optional modes: when to set the vision-classification env vars, when to ask the user for a Groq key, and when to pass authStorageStatePath. This is direct, operational usage guidance rather than left to inference.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.