Server Quality Checklist

Tool Scores

• q_plan_startA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses background execution, immediate return, parallel seats, bias-free starting point, the anti-bias gate, optional role assignment, gemini_model substitution, and expected wall-clock time. It does not explicitly state side effects or permissions, but the behavior is largely transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a coherent paragraph with clear sections: execution model, the anti-bias gate, roles, gemini_model, and timing. It is somewhat verbose (e.g., 'structurally bias-free') but every sentence adds useful information and is appropriately front-loaded with the core behavior.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 9 parameters, no annotations, and an output schema, the description covers the high-level workflow and two parameters but omits critical invocation details like the meaning of 'cwd', 'task', 'host', 'agents', 'skip_gh', and 'no_context'. An agent would struggle to invoke it correctly without further documentation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description explains 'roles' (format and defaults) and 'gemini_model' (format and purpose), but ignores the other 7 parameters including required ones like 'task' and 'cwd'. With 0% schema description coverage, the description must compensate, but it covers only about 22% of parameters, leaving most ambiguous.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool starts a q-plan run in the background, returns a job_id immediately, and explains the anti-bias gate. It is distinct from siblings like q_brainstorm_start or q_validate_start by its explicit focus on structural anti-bias and the background execution model.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives concrete workflow guidance: start the run, then call q_await with the job_id, and it emphasizes the anti-bias gate as a reason to use this tool. However, it does not explicitly say when not to use it or contrast with alternative start tools like q_brainstorm_start.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• q_validate_startA
  4.1/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It comprehensively explains the background execution, immediate return of job_id, the bias-free first round and later rounds embedding priors, the extended mode's 4 rounds and stance rotation, the verbose output toggle, and the gemini_model override for this invocation. It also discloses expected wall-clock times and the anti-bias gate expectation, making the tool's behavior highly transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is detailed but well-structured, front-loading the core behavior and return value before diving into options. It uses clear paragraphs and imperative sentences that earn their place. While somewhat long, the length is justified by the tool's complexity, and it could be slightly more concise without losing essential information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (11 parameters, no schema descriptions), the description covers the main workflow and several key parameters, and provides expected wall-clock times. However, it omits explanations for critical parameters like plan_path and cwd, and does not clarify the exact output format beyond job_id (though q_await likely handles results). The description is strong on purpose and behavior but incomplete in parameter coverage, so it is not fully complete for an agent to confidently invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate for parameter documentation. It explains extended, mode, verbose, roles, and gemini_model in detail, but leaves essential parameters like plan_path, cwd, host, agents, skip_gh, and no_context unexplained. The required plan_path is only referred to generically as 'plan file', which is insufficient for an agent to know exactly what to provide. The description only partially covers the 11 parameters, making parameter semantics incomplete.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it starts a q-validate run in the background and returns a job_id immediately. It describes the process of agents reviewing the plan file and deliberating, which distinguishes it from sibling tools like q_plan_start, q_brainstorm_start, and q_review_start. The tool's purpose is unambiguous and specific.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on retrieving results via q_await with the returned job_id and explains the structural anti-bias gate that the caller should form its own review between start and await. It also advises when to use extended mode ('Pick extended deliberately') and explains the default vs extended rounds. However, it does not explicitly contrast this tool with sibling start tools, relying on tool names for differentiation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• q_brainstorm_startA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden, and it delivers comprehensively. It discloses the async nature (runs in background, returns job_id immediately), the output format, the expected timing (30s-4min), and the behavioral constraints (agents contribute 3-5 ideas, no synthesis, anti-bias gate). It also explains the semantic difference between research and prior_ideas. No behavioral aspect is left opaque.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long but well-organized into focused paragraphs. It front-loads the core behavior and return type, then explains each optional parameter and composition. Each paragraph earns its place, but the length is near the upper bound of acceptable. It is structured rather than rambling, so it scores a 4.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (12 params, no schema coverage, no annotations, and an output schema), the description covers the essential workflow and the most impactful parameters. It explains the anti-bias gate, the difference between research and prior_ideas, and the grounding pass. Missing: descriptions of verbose, skip_gh, no_context, host, and a few error-handling details, but those are minor. It is nearly complete for a working understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must explain parameters. It does explain the significant ones: research, prior_ideas, grounding, roles, gemini_model. However, it omits explanations for agents, skip_gh, verbose, no_context, host, and even cwd/topic are only implicit. For a tool with 12 parameters, the description adds value but leaves several parameters undocumented, forcing the agent to infer from names alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb and resource ('Start a q-brainstorm run'), states the async behavior, and names the sibling tools it complements (q_await, q_research). It clearly distinguishes this from q_plan_start, q_validate_start, etc., and explains the composition with q_research and q_await. An agent can immediately understand what this tool does and how it fits.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit when-to-use guidance: 'Between the start and the await, the caller is expected to list its own ideas — this is the structural anti-bias gate.' It also names alternatives and conditions: 'Distinct from prior_ideas, whose do-not-repeat framing...' and mentions specific flows like '--extended' and 'q-skystorm Stage 2' that select the grounding mode. This is unambiguous.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• q_review_startA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden and it delivers richly. It discloses that the call returns immediately instead of blocking (async job), gives expected wall-clock timing (1-8min default, 4-15min extended), spells out the empty-diff short-circuit that returns a 'nothing to review' message without running the council, details verbose output behavior (HELD lines, agreement counts), and explains the stance rotation at round 3 for extended mode. This far exceeds baseline behavioral disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long, but every paragraph earns its place for a 12-parameter async background tool. It is front-loaded with purpose and the return contract, then flows logically through the anti-bias gate, target values, scope, execution modes, output verbosity, and timing. The final timing note is a genuine value-add. It could be tightened slightly, but the length is justified by complexity rather than padding.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool of this complexity — 12 params, async behavior, background execution, no annotations, 0% schema coverage — the description is remarkably complete, covering the workflow, the caller's required participation (anti-bias gate), short-circuit behavior, per-parameter semantics, and timing expectations. The output schema exists so return values need no further explanation, and job_id is already surfaced. The only completeness gap is the five undocumented parameters (agents, skip_gh, no_context, host, cwd) that the description leaves to the schema, which is empty.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate — and it does for the majority. target gets a detailed breakdown of every value ('working', 'pr:N'/URL, 'A..B'/'A...B', 'all', default branch-vs-main), and scope_path, extended, mode, verbose, roles, and gemini_model are each explained. However, five of the twelve parameters (cwd, host, agents, skip_gh, no_context) are not touched in the description and, with zero schema descriptions, remain entirely undocumented — a real gap for a tool where agents and skip_gh likely matter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The opening line states a specific verb and resource — "Start a q-review run in the background. Returns {"job_id": str} immediately" — and explains the mechanism (independent agent review then convergence across rounds). This clearly distinguishes a review tool from its siblings q_plan_start, q_brainstorm_start, and q_validate_start, which by naming convention cover different activities. No ambiguity about what this tool does.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives strong usage context: it is for reviewing real code changes, and it explicitly routes retrieval to a sibling — "Retrieve results by calling `q_await` with the returned job_id" — which orients the agent within the family of q_* tools. It also defines the structural anti-bias gate the caller must pass between start and await. However, it never explicitly states when NOT to use this tool versus q_validate_start or the other review siblings; exclusions are implied rather than stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• q_researchA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden, and it excels. It discloses internal retries ('Mechanically-fixable failures ... are already retried inside the tool'), error handling ('Failed sources are reported inline under 'Sources unavailable' rather than failing the call'), response format (markdown digest with a 'Research status:' line), per-source behaviors, and the distinction between domain-legitimate zero results and gaps. This level of transparency exceeds typical tool descriptions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very long but well-organized, front-loading the core purpose and then progressing into usage, error handling, and modes. Each paragraph serves a distinct function, yet some sections (e.g., the extended discussion of same-domain vs cross-domain checks) are arguably more detailed than necessary for an agent to invoke the tool. It is concise relative to its complexity but could be tightened without losing value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers every aspect an agent needs: what it returns (markdown digest, status line), how to interpret results (RETRY-RECOMMENDED, CONFIG, LOW-OVERLAP), error handling with specific retry guidance, per-source whiffing logic, mode differences, and query-shaping best practices. Even with an output schema (not shown), this description is entirely self-sufficient given the tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Despite 0% schema coverage, the description explains the `topic` (query shaping), `sources` (defaults, subset pass), and `mode` (grounded vs exploratory) parameters in depth. However, the `limit` parameter is never mentioned – its purpose and effect on output are not explained. Since the schema provides no help, this is a noticeable gap, though the description compensates well for the other three parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a precise verb and resource: 'Fetch prior art for a topic from arXiv + OpenAlex + Europe PMC (papers), Context7 (library docs), GitHub (repos), and HuggingFace (models), and return a markdown digest.' This clearly distinguishes the tool from its siblings (planning, brainstorming, validation, review, await) by focusing on external prior-art retrieval. No ambiguity about what the tool does.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit when-to-use instructions: 'Call it during the own-work window of a `q_brainstorm` or `q-skystorm` run to ground the synthesis.' It also provides query-shaping rules, mode interpretation ('grounded' vs 'exploratory'), error-handling strategies, and when not to trust results. Since there are no sibling research tools, it doesn't contrast with alternatives, but it provides rich contextual guidance on exactly when and how to use the tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• q_awaitA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations to rely on, the description carries the full burden and does so thoroughly. It discloses that the tool blocks until completion or error, that it is one-shot (a job_id can only be awaited once), and enumerates all error scenarios (job not found, TTL cancellation, server shutdown, underlying council errors). It also conveys the operational expectation that the host must not end its turn with an outstanding job. This goes far beyond a minimal disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is highly structured: the first sentence delivers the core purpose, the second paragraph provides usage context, and a bulleted list covers errors. Every sentence earns its place; there is no fluff. The information is front-loaded with the most critical fact (blocking behavior) stated immediately.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description fully covers what an agent needs to know to call this tool correctly: it explains the blocking nature, the one-shot constraint, the exact error outcomes, and the return value (rounds markdown). An output schema exists, so detailed return formatting is not required. There is no missing information that would prevent correct invocation or interpretation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The single parameter `job_id` has zero schema description, so the description must compensate. It does by explaining that the job_id identifies the background run, and the error section clarifies what happens if the id is invalid (expired, already retrieved, or invalid). It stops short of specifying a format (e.g., UUID), but since a valid id comes from a sibling start tool, this is sufficient for the agent to understand its role.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb ('Block'), a clear resource ('background council run identified by job_id'), and an outcome ('return its rounds markdown'). It distinguishes itself from the sibling start tools (q_plan_start, q_brainstorm_start, etc.) as the blocking completion counterpart, so an agent can immediately tell what this tool does and how it differs from its siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states that this is 'the blocking completion notification for every council start' and gives a clear directive: 'The orchestrating host must not end its turn with a live job outstanding; it calls q_await after its independent work.' It also warns that it is one-shot, leaving no ambiguity about when and how to use it. No alternative tools are mentioned, but the context of siblings being all start tools makes the usage obvious.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.