Server Quality Checklist

Tool Scores

• pi_modelsA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the behavioral burden and does so well: it states this is a read from the live catalog ('actually reach right now') and explicitly denies side effects. It could add error/availability behavior or auth expectations, but the core read-only, no-task traits are clear.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two tight sentences, front-loaded with the main action and output fields before the usage directive. Every sentence contributes either scope, use case, or a side-effect exclusion; no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple one-optional-parameter input and no output schema, the description is complete: it describes the returned model attributes, the purpose, and the non-mutating behavior. An agent can call it correctly without additional context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already documents the only parameter with 100% coverage, so the description need not compensate; it stays at the baseline. The description does not add extra semantic detail beyond the schema's 'Optional fuzzy filter' with examples.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource ('List the models pi can actually reach right now') and enumerates the return fields, making the tool's function unmistakable. It also ties itself to pi and pi_reply, which separates it from the session/task siblings without opening schemas.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly tells the agent when to use it: to pick `model` and `thinking` values for `pi` / `pi_reply`. The closing phrase 'Starts no session, runs no task' also signals when-not to use it, distinguishing it from pi_send, pi_running, and pi_sessions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• pi_sendA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden, and it delivers. It discloses the async behavior ('Returns immediately'), where the effect surfaces ('pi's reaction appears in the answer of the pi/pi_reply call still waiting on that turn'), and the transport/reachability constraint. This is exactly the kind of non-obvious behavioral context an agent needs beyond the schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Four dense sentences with zero waste. The core action is front-loaded, followed by constraints, behavioral timing, and a helpful pointer to pi_running. Every sentence earns its place and no information is duplicated from the schema.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having no annotations and no output schema, the description covers the essential context: transport constraint, timing semantics, where the result appears, sibling differentiation, and session discovery. The only minor gap is the exact return value of pi_send itself (an ack?) and behavior on an invalid or unknown session id, but these are small omissions for an otherwise complete definition.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3: the schema already documents each parameter's meaning, including the command enum behavior and message requirements. The description adds marginal value by framing session as a currently executing turn and pointing to pi_running, but it does not meaningfully augment the parameter docs.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Deliver a message into a pi turn that is executing right now.' It explicitly distinguishes the tool from pi_reply ('a session that already finished takes pi_reply, not pi_send'), so an agent can tell them apart instantly. The scope is precise — only live turns on 'rpc' transport.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  This is exemplary routing guidance. It states the exact precondition ('Works only on runs started with transport 'rpc''), a negative exclusion ('print' runs cannot be reached), and names the alternative tool for the finished-session case (pi_reply). It also points to pi_running for discovering reachable sessions. Nothing is left to inference.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• piA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full behavioral burden, and it does substantial work: it discloses that pi blocks until completion, returns only final answer plus stats, has no permission system, edits files and runs shell commands as the user inside cwd, and that a timeout-killed run is resumable. A small gap is not describing the exact stats format or failure behavior, but the most important behavioral traits are covered.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact yet dense: an opening behavioral summary, a clear 'Good for' section, and a Caution section. Every sentence earns its place. It front-loads the core behavior (starts a new task, blocks, returns session id) before routing and caution details, making it easy for an agent to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a complex 9-parameter tool with no output schema and no annotations, the description provides a complete operating picture: session lifecycle (via pi_reply), model discovery (pi_models), steering/aborting (pi_send), timeout recovery, safety caveats, and parameter defaults. The combination of description and 100% schema coverage leaves little ambiguity about how to invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all 9 parameters well. The description adds value beyond the schema by explaining which parameters to 'usually omit' (cwd, tools, transport, timeout_ms) and why, and by explaining the consequences of transport choices and timeout deadlines. This is helpful guidance an agent could not get from the schema alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('Start a NEW task') and resource ('local pi agent'), and clearly distinguishes this tool from its siblings by explaining pi_reply continues sessions, pi_models lists models, and pi_send steers running turns. The scope is explicit: it blocks until pi settles and returns only the final answer plus stats prefixed with [session: <id>].

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says when to use pi ('Good for: a second opinion from a different model, work kept out of this context, or parallel investigation') and offers clear caution about its lack of a permission system, advising to pass tools or no_tools for analysis-only work. It also references sibling tools for continued sessions and model listings, giving an agent actionable routing guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• pi_replyA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full behavioral burden and meets it well: it discloses session survival across timeout/cancellation, that pi retains prior turns but 'never this conversation', and that sessions survive server restarts. The schema's transport and timeout_ms descriptions add durability semantics (killed runs still return a session id and are resumable). Minor deduction for not covering what happens when a session id is invalid or expired, though the schema's reference to pi_sessions partially mitigates this.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Four sentences, zero filler, and the core purpose plus most critical edge case are front-loaded in the first sentence. Each subsequent sentence earns its place: context retention, persistence, and sibling routing. Nothing is redundant with the schema.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a stateful tool with 7 parameters, no annotations, and no output schema, the definition is complete. It covers how to obtain a session id (via schema references to pi_sessions and the [session: <id>] prefix), what the timeout behavior is (resumable, returns session id), when the call returns before the turn completes (transport semantics), and how to distinguish siblings. The essential return-value behavior is disclosed through the timeout_ms description.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3; the description adds genuine parameter-level value on top. It clarifies the session parameter's semantics (timed-out/cancelled sessions are still valid targets) and the prompt parameter's expected style ('so the follow-up can be short'). The schema parameter descriptions themselves are unusually rich, cross-referencing pi_models, pi_sessions, and pi_send, which justifies above-baseline scoring.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource — 'Send a new turn to an existing pi session' — with a precise scope condition ('that is not executing right now'). It actively distinguishes itself from siblings by naming both alternatives: 'resume it here instead of restarting with `pi`' and 'use pi_send instead.' An agent can tell exactly what this tool does without opening any sibling schema.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit when-to-use and when-not-to-use guidance is present. It covers valid edge cases ('including one that timed out or was cancelled'), gives the exclusion condition ('For a turn still running under "rpc", use pi_send instead'), and names the restart alternative (`pi`). The persistence guarantee ('Survives restarts of this server') further informs whether resuming is safe.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• pi_runningA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full behavioral burden. It discloses the live-only nature of the data, the rpc-transport limitation, the unavailability of 'print' runs during execution, and the fields returned. It does not spell out that the call is read-only, but 'List' strongly implies it, and the caveats add meaningful behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, no filler. The main action and scope are front-loaded, and each subsequent sentence adds a necessary caveat or alternative. The description is compact but information-dense.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter list tool with no output schema, the description is complete: it names the resource, the live scope, the transport restriction, the return fields, and the alternative for historical data. An agent has everything needed to select and invoke this tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has zero parameters, so there is nothing for the description to explain. The description adds value by itemizing the returned fields (session id, working directory, elapsed time, messages sent in), which helps the agent understand the tool's output even in the absence of an output schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('List') and a precise resource ('pi turns executing at this moment'), and clarifies the scope by saying 'the ones pi_send can reach.' It also distinguishes itself from pi_sessions explicitly, so an agent can tell them apart without reading schemas.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description states exactly when to use this tool: for currently executing pi turns over rpc-transport. It explicitly excludes 'print' runs as unreachable mid-run and directs the agent to pi_sessions for past sessions, giving clear when-to-use and when-not-to-use guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• pi_sessionsA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the behavioral burden. It clearly discloses listing order, the working directory field, and the inclusion of running, finished, and timed-out sessions. The only minor gap is that it doesn't state return format or error behavior, but for a listing tool the disclosed scope and status coverage are strong.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences with no filler. The first sentence delivers the core behavior and result fields, the second gives the primary use case, and the third handles sibling differentiation. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a parameterless list tool with no output schema, the description is complete enough: it states scope, ordering, covered statuses, a concrete usage purpose, and the related alternative for in-progress sessions. Nothing essential for correct invocation is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, which is the baseline case for a 4. The description correctly focuses on what the return value contains (all sessions, working directory, ordering) rather than parameter details, which are not applicable.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('List') with a clear resource ('pi sessions'), and adds distinguishing details: 'started through this server, newest first, with their working directory'. It also covers inclusion criteria (running, finished, timed out) and explicitly differentiates from the sibling pi_running, so an agent can select it correctly.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description states exactly when to use the tool: 'Use it to recover an id for pi_reply.' It also gives an explicit exclusion: 'For turns still executing (pi_send targets), use pi_running.' This directs the agent to the right sibling without ambiguity.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.