Server Quality Checklist

Tool Scores

• explain_actor_protocolA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already cover readOnly/idempotent/destructive hints, so the bar is lower; the description nonetheless adds genuine context: in-memory compilation of the whole directory, caching until a sibling changes, writes/executes nothing, and a stable error.code set with diagnostics in error.data. One defect: the error-code list omits `not_found`, which the input schema cites for unknown actor names, creating a small internal inconsistency. No contradiction with the annotations themselves.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Front-loads purpose, then usage routing, then return shape, then error behavior — a logical order with every sentence earning its place. It is dense rather than padded, though the return-shape enumeration is somewhat long; acceptable given the tool's complexity and the presence of an output schema.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given an output schema exists (so return values are structurally defined), the description goes beyond by summarizing the returned `actor` shape and error codes. Both required parameters are fully documented, alternatives are routed, and the read-only/caching behavior is disclosed. The only gap is the `not_found` inconsistency, slightly muddying the error-handling story.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% — both file and actor_name are thoroughly documented, including the path-resolution rules and the not_found failure with error.data.available_actors. The description adds no parameter-level meaning beyond this, so the high-coverage baseline of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific verb ('Describe one actor's protocol') tied to a precise resource, and enumerates exactly what the explanation covers: message type, constructors, state type, init params/effects, per-handler effects, and the declared effect summary. It also names sibling tools (emit_actor_effect_graph, lookup_definition) it is not, so an agent can disambiguate without opening any schema.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Gives explicit when-to-use ('learn how to talk to an actor or what each handler may do') and then names the two alternatives with the conditions that select them — emit_actor_effect_graph for transitive reachability, lookup_definition for location-only lookups. No inference required.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• emit_actor_effect_graphA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already provide readOnlyHint=true, idempotentHint=true, destructiveHint=false, and the description goes well beyond them: it discloses in-memory directory compilation with caching until a sibling changes, states it 'writes or executes nothing', and enumerates the stable error.code values plus the diagnostics structure on failures. No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long (roughly 150 words) but densely packed — scope, usage rationale, alternatives, return shape, read-only behavior, and error codes all in distinct clauses with no redundancy. Purpose is front-loaded. Slightly long, but every sentence earns its place given the tool's complexity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a graph-returning tool with compilation, caching, and multi-mode failures, the description covers scope definition, usage context, sibling routing, return shape (schema_version, module, root, actors/supervisors/tools), and the full error contract. An output schema exists, so return values need not be re-explained, and nothing an agent needs to call this correctly is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% — both file and actor_name are fully documented in the schema (path resolution, import behavior, not_found error with available_actors). The description adds little parameter-specific meaning beyond the schema, so the baseline of 3 applies. The only mild addition is the note that declarations the root does not reach are omitted, which concerns output rather than parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific verb and resource ('Emit the actor/effect graph rooted at one actor') and precisely enumerates the graph's contents (reachable actors via Send/Await/Spawn/Schedule, supervisors with child sets, named tools). Clearly differentiates itself from siblings by naming explain_actor_protocol and get_context_budget as the alternatives for narrower/sizing queries.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states the intended use case ('what does this actor transitively do or depend on'), explains why it is needed ('effect rows are per-process, so no single signature shows this'), and names two concrete alternatives with their selection conditions. This is exactly the when/when-not guidance the dimension calls for.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• lookup_definitionA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, destructiveHint=false. The description adds genuine value beyond that: the in-memory directory compilation with caching behavior ("cached until a sibling changes"), the explicit "writes or executes nothing" confirmation, the not_found contract with error.data.available, and the stable error.code taxonomy with coded diagnostics. No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Purpose is front-loaded in the first sentence, alternatives follow, then return format, then error behavior, then read-only confirmation — a logical progression where every sentence earns its place. It is long, but the length is justified by the error taxonomy and routing guidance that an agent genuinely needs. Only minor trimming would be possible.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a lookup tool with an output schema, nothing needed is missing: purpose, alternatives, return kinds, nullable fields, import behavior, not_found handling, error codes, and safety profile are all covered. An agent has everything required to call it correctly and interpret failures.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% — both file and name are fully documented in the input schema, so baseline 3 applies. The description adds minimal param-level detail; the only extra is the return-side note that file is the sibling module when imported, which touches the parameter but is primarily return semantics. Adequate, not additive.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a precise verb+resource — "Look up a top-level definition by name: defining file and line, kind, type, and doc comment." It enumerates the returned fields and explicitly names three siblings with the conditions that select them (get_context_for_symbol, explain_effect_row, render_ir_fragment), so an agent can distinguish it from every sibling without opening schemas.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Gives explicit routing: "Use it first to locate or identify a symbol" and then names alternatives with their differentiators — get_context_for_symbol for effects/callers/callees, explain_effect_row for interpreting effects, render_ir_fragment for the body. This is textbook when-to-use-vs-alternatives guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• render_ir_fragmentA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, destructiveHint=false, and idempotentHint=true. The description goes further by explaining the in-memory compilation and caching ('compiles the file's directory in memory (cached until a sibling changes) and writes or executes nothing') and by specifying error codes and diagnostics. No contradiction; it enriches beyond annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and well-structured, with purpose, usage, return format, and error handling presented in logical order. It is longer than minimal but each sentence earns its place; the only slight inefficiency is bundling multiple error codes into one run-on sentence.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (compiling a file directory, multiple error codes, return structure), the description covers everything an agent needs: purpose, when to use, return fields, error semantics, and read-only guarantees. The output schema exists but the description does not need to explain return values; it is complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, and both parameter descriptions in the schema are already detailed. The tool description adds no additional parameter-level information beyond what the schema provides, so baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb-resource pair ('Render the typed intermediate representation (IR) of one top-level definition as JSON') and explicitly contrasts with sibling tools (get_context_for_symbol, lookup_definition) by naming when the IR view is preferable. It distinguishes itself without ambiguity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit when-to-use criteria ('when the exact lowered structure matters: desugared bodies, the resolved type on every node, a tool's generated function') and directs the agent to alternatives for human-oriented views. This is textbook usage guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_context_for_symbolA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond annotations (readOnlyHint, idempotentHint, destructiveHint), the description adds substantial behavioral context: the in-memory compile with caching 'until a sibling changes', the claim that it 'writes or executes nothing', the exact failure taxonomy with stable error.code values (file_not_found, read_error, invalid_params, parse_error, check_error), the not_found behavior with available names, and the truncation/dropping semantics. This is far beyond what annotations alone convey, and it is consistent with them.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long (~180 words) but every sentence earns its place: purpose, sibling routing, return shape, truncation edge case, unknown-name behavior, safety/caching, and error codes. It is front-loaded with the core function and progressively covers edge cases, with no redundant or filler phrasing.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 3 required params, 7 siblings, an output schema, and multiple failure modes, the description covers every dimension an agent needs: what it does, when to choose it, what it returns, how budget behaves at the boundary, unknown-name handling, and all error codes with their payloads. Nothing an agent needs to invoke it correctly is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the schema documents all three parameters thoroughly; the description adds only marginal value on top, mostly reinforcing budget semantics (~4 chars/token, signature always present) and the not_found name resolution behavior. That is a small bonus over the baseline of 3, but the schema carries the heavy lifting for parameter meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb+resource: 'Summarize one symbol for an LLM prompt within an approximate token budget', then enumerates exactly what is included and in what order (signature, effect row, doc comment, callers, callees). It explicitly differentiates from siblings by naming lookup_definition, explain_effect_row, and render_ir_fragment and what each is for, so an agent can distinguish this tool without opening the sibling schemas.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Usage guidance is explicit and prescriptive: 'Use it as the default way to bring a symbol into context', followed by three concrete alternatives with the condition that selects each (just location, effect explanations, full body). This leaves nothing to inference and is the strongest possible usage guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• infer_typeA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint and idempotentHint, but the description goes further by disclosing that it compiles the file's directory in memory (cached until a sibling changes) and writes/executes nothing. It also details failure modes as isError results with stable error.code values and lists the codes, providing rich behavioral context beyond annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is dense but efficiently organized: purpose first, then usage, then returns, then read-only note, then error handling. Every sentence carries operational value; there is no fluff or repetition. For a tool with this complexity, the length is justified and the structure aids scanning.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With an output schema present, the description correctly focuses on behavior rather than return structure. It covers purpose, usage, return fields (token, type, effect_row), error codes, and the compilation/caching model. An agent has everything needed to invoke the tool correctly and interpret results.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% coverage, describing file, line, and column with details like coordinate bases, character vs byte columns, and token preference. The description adds no additional parameter meaning beyond what the schema already states—it only references return values and error handling. Baseline 3 is appropriate given full schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific verb (infer) and resource (type and effect row of an expression at a source location in a .hird file), and explicitly differentiates from siblings by naming lookup_definition and explain_effect_row as alternatives for different cases. The return values are also listed, making the tool's function unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit when-to-use guidance ('what is the type here' for local bindings, sub-expressions, use-list names) and when-not-to-use with named alternatives (lookup_definition for top-level defs, explain_effect_row for explained effects). This is direct routing with no inference required.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• explain_effect_rowA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already cover readOnlyHint=true, idempotentHint=true, and destructiveHint=false. The description adds value beyond those: it discloses in-memory directory compilation with caching until a sibling changes, states it writes/executes nothing, explains the open/pure return semantics, and enumerates stable error codes with the diagnostic payload location. No contradiction with annotations; substantial additive context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is dense (~150 words) but every sentence carries distinct information: purpose, usage, return shape, error handling, read-only behavior, and error codes. It is front-loaded with purpose and usage before details. Slightly long but justified given the tool's error-handling and return complexity; no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite an output schema being present, the description still explains the semantic meaning of return fields (open, pure, effects with explanations) and error contract. It covers purpose, when-to-use, alternatives, behavior, and failure modes exhaustively. Nothing an agent needs to call it correctly is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and both parameters (file path, fn_name) are well documented in the schema, so baseline is 3. The description adds error-handling semantics for fn_name beyond the schema: a non-function name yields not_a_function, an unknown name yields not_found with available names in error.data.available. This enriches parameter behavior meaningfully, earning a 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: 'Explain a function's effect row: the canonical row plus a one-sentence explanation of each effect.' It distinguishes itself from siblings by explicitly naming infer_type (arbitrary expression's row) and emit_actor_effect_graph (transitive actor effects), so an agent can tell them apart without opening schemas.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit when-to-use guidance: 'Use it for "what may this function do" or to interpret an unfamiliar effect.' It also states what it covers ('one named function') and names the alternatives with their selection criteria (infer_type for arbitrary expressions, emit_actor_effect_graph for transitive actor effects), leaving nothing to inference.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_context_budgetA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint, idempotentHint, and destructiveHint, but the description adds substantial context: it compiles the directory in memory (cached until a sibling changes), writes or executes nothing, and details failure modes with stable error codes and diagnostics. This goes well beyond the annotation profile.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is dense but well-structured, starting with the core purpose, then usage guidance, then return shape, then failure modes. Each sentence adds essential information with no filler. The use of lists for categories and error codes keeps it scannable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter read-only tool with a rich output schema (implied) and thorough annotations, the description covers everything an agent needs: purpose, usage, estimation method, caching behavior, failure modes, and error codes. There is no missing piece that would prevent correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema covers the 'file' parameter at 100% with a clear description. The tool description adds extra meaning by explaining that every .hird file in the directory is compiled as one program, which affects how imported names resolve and can make answers reference sibling files. This is value beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb ('Estimate the token cost') and a precise resource ('loading a file's declarations into an LLM context window'), then enumerates the exact categories returned. It explicitly distinguishes itself from the sibling get_context_for_symbol by stating it names no individual symbols, so an agent can immediately tell them apart.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives explicit usage scenarios: before pulling a module wholesale, to choose between get_context_for_symbol calls and a full read, or to pick a budget. It also names the alternative tool and implies the condition for choosing it, providing clear routing guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.