Server Quality Checklist

Tool Scores

• lens_catches_statsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of behavioral disclosure. It explains the threshold-based PROMOTE suggestion behavior, which is useful, but it does not disclose side effects: the schema's cwd description mentions catches.jsonl is written to, yet the main description does not clarify whether this tool mutates state or only returns analysis.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, each earning its place: the output concept, the threshold behavior with a clarifying explanation, and the intended decision use. The most important information is front-loaded and jargon is briefly unpacked.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter stats tool with no output schema, the description adequately conveys what output to expect and why it matters. It could be more complete by defining the threshold or explicitly stating side effects, but those are minor given the schema and the overall clarity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% for the single optional cwd parameter, and the schema already explains the working-directory semantics and the catches.jsonl read/write behavior. The tool description adds no parameter-specific meaning, but none is needed given the schema's completeness.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific resource (defect bank) and a specific computation (per-pattern recurrence counts), and explains the output's purpose: suggesting PROMOTE-to-deterministic-check for recurring patterns. It is clear even though it lacks an explicit verb like 'get' or 'list'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says to use this tool to decide what to harden next, which gives the agent a clear decision context. It does not name alternative tools or state when not to use it, so it stops short of full routing guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• validate_contentA
  4.2/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and does so thoroughly. It discloses the deterministic score semantics (10 = clean, floors at 0, computed by the API tier not the LLM), the precise wire contract (passed===false as the violation signal, never !passed), and halt semantics for Rights and consciousScan. This is far more transparent than a generic 'validates content' statement.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a dense paragraph covering purpose, outputs, score computation, authentication, domains, wire contract, and halt semantics — every sentence carries essential information and the core action is front-loaded. However, the contract and halt rules are packed into long sentences without visual structure like bullets, making it slightly harder to parse than necessary.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  This tool has no output schema and no annotations, so the description must explain return values and edge cases; it does. It covers pass/fail, per-lens results, violations, the 0-10 score, API key requirement, supported domains, the status/passed wire contract, and halt semantics. Nothing an agent needs for correct invocation and result interpretation is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description's domain list mirrors the schema enum exactly and adds no new parameter meaning. The 'context' parameter semantics come from the schema description (Relevance lens, Lens 9), not from the tool description. Thus the description adds no value beyond the schema for parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb and resource: 'Run the 10-lens validation gate on AI-generated content.' It clearly states what the tool does and lists its outputs (pass/fail, per-lens results, violation details, 0-10 score). The '10-lens' framing distinguishes it from sibling single-lens tools like lens_catches_relevant, even though no sibling is explicitly named.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives prerequisites (requires SOULFIELD_API_KEY) and supported domains, which helps an agent know when it can be used. However, it never explicitly tells an agent when to choose this aggregate validation gate over a sibling single-lens tool, nor does it state exclusions. Usage must be inferred from the '10-lens validation gate' framing rather than being directly guided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• lens_catches_relevantA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and discloses meaningful traits: it 'Runs the local lens-kit CLI, no LLM call' (execution mechanism and cost), returns 'most-recurrent patterns first' (ordering behavior), and explains the [PROMOTE] marker semantics. Minor gap: the schema's cwd description says catches.jsonl is both read from and written to, while the description frames this as purely a read, leaving the write side-effect ambiguous.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The purpose is front-loaded in the first sentence, and the remaining sentences each add distinct information: history rationale, PROMOTE semantics, execution mechanism, and the parameter workaround for the no-history case. The institutional-memory sentence is slightly rhetorical but reinforces why recurrence ordering matters, so no sentence is wasted.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema and no annotations, the description must carry the load, and it covers the core what (named defects), ordering, promotion marking, execution mode, and the special all=true case. The format parameter's schema descriptions ('block = paste-ready surface; json = raw records') partially compensate for the missing output schema, though the exact record shape is left unspecified.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3, and the description adds value on top: it explains the artifact_type/all interaction ('Omit artifact_type and pass all=true when the type has no history yet') and gives threshold meaning beyond the schema's terse wording by tying recurrence to deterministic checks. This pushes it clearly above baseline.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The opening sentence names a specific verb and resource — 'Read the defect bank' — plus the scope ('prior named defects for an artifact type') and ordering ('most-recurrent patterns first'). This clearly differentiates it from siblings like lens_catches_add (write), lens_catches_stats (aggregate), and validate_content (validate), and the workflow position 'BEFORE validating' pins down its role.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear context for when to use the tool — 'Read the defect bank BEFORE validating' — and an explicit scenario: 'Omit artifact_type and pass all=true when the type has no history yet.' It stops short of naming sibling alternatives or stating when not to use it, so it has clear context without formal exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• lens_catches_addA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the transparency burden. It discloses that the operation appends to a defect bank, that routine passes are rejected, that self_catch has a specific meaning around validator discipline, and that repeated patterns can promote to deterministic checks. It does not state output/return behavior or error handling, but for an append operation the core side effect is clearly conveyed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three dense sentences with no filler. The core action is front-loaded, policy guidance follows immediately, and the final sentence provides useful context about the improvement loop. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers purpose, policy, self-catch semantics, and the improvement-loop context. It is missing only an explicit pointer to the read-half siblings and any description of what the agent should expect after a successful append, but given 100% schema coverage and the simplicity of the operation, the remaining gaps are minor.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description adds real semantic value beyond the schema by explaining that self_catch=true means a failure of the validator's own discipline and by connecting catch, pattern, and rule to the intended defect-recording workflow. It does not add much for artifact_type or domain, but those are already self-explanatory in the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Append a NAMED DEFECT to the defect bank so it is caught next time.' It clearly differentiates itself as the write half of the improvement loop, which distinguishes it from read-oriented siblings like lens_catches_relevant and lens_catches_stats.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear usage context: use this when an actual defect is found, and explicitly warns not to log routine passes ('doctrine REJECTS routine passes'). It frames the tool as the write counterpart to the loop, but it does not explicitly name read alternatives or spell out conditions for when to use sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• lens_consistency_markersA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full behavioral burden and meets it: deterministic, no LLM, exit 6 on violation, case-sensitive matching, and two false-positive traps (case mismatch and deliberate subset rendering). The concrete TRIPWIRE example makes the matching behavior unambiguous.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Core purpose is front-loaded and the paragraph stays dense with useful caveats. The 'TRIPWIRE:' sentence is cryptic and slightly interrupts the flow, preventing a 5.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 5-parameter tool with no annotations or output schema, it covers the main behavior, exit status, matching mode, and practical verification steps. It does not state the success exit condition or the full return/result shape, and it leaves profile/marker interplay to the schema, but the description is otherwise complete enough to invoke correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema covers all 5 parameters, so baseline is 3. The description adds value beyond the schema by instructing users to choose markers with stable casing and by warning that a deliberate subset render legitimately under-counts, which directly informs how markers, source, and rendered should be used.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The first sentence names a specific verb and resource: check that source artifact markers survive into every rendered output. The title and examples further establish it as a parity/consistency verifier, and the explicit case-sensitivity contrast with lens_consistency_leaks sets it apart from siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives clear operational context: use markers with stable casing, and verify doubtful hits case-insensitively before reporting. It names lens_consistency_leaks as the case-insensitive alternative, though it never explicitly says 'use this tool when...' or lists exclusion conditions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• lens_consistency_numbersA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full behavioral burden, and it delivers: it discloses deterministic matching, no LLM usage, exit code 6 on violation, literal-only matching with no semantic or derived arithmetic, and the caveat that legitimately computed totals will flag. This gives the agent an unusually complete picture of the tool's behavior and limitations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded, starting with the core check, then adding determinism, exit behavior, purpose, limitation, and a review caution. Every sentence adds distinct information, and there is no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is complete for a tool with no annotations and no output schema: it covers what is checked, how it behaves, what exit code signals a violation, what its limitations are, and how the agent should handle results. The schema covers parameter details and the catches.jsonl side effect, so nothing critical is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents the summary, body, and cwd parameters well. The tool description reinforces the relationship between summary and body numbers but does not add parameter-level detail beyond what the schema provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific verb and resource: check that every numeric literal in a summary appears in the body it summarizes. It clearly distinguishes the tool from siblings by focusing on numeric literal parity with deterministic behavior and exit 6 on violation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear operational context: it is deterministic, uses no LLM, and is a tripwire rather than an oracle. It explains when the result may be a false positive and tells the agent to review hits, though it does not explicitly compare against sibling tools or state when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• lens_healthA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  There are no annotations, so the description fully carries the burden of disclosure. It states that no auth is required and that it returns status and version, and the verb 'check' clearly implies a non-mutating read-only operation. It could additionally mention timeouts or error behavior, but for a simple health check the provided transparency is solid.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three short sentences, each carrying distinct value: what the tool does, what it returns, and its authentication requirement. No extraneous words or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter health-check tool with no output schema, the description covers all essentials: purpose, return content (status and version), and auth. There are no missing details that would prevent an agent from invoking it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and the schema is trivially 100% covered. With no parameters to document, the description does not need to provide additional semantics, so the baseline score of 4 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific verb ('Check if') and a specific resource ('the Soulfield Lens API service'), and tells what it returns. This distinguishes it sharply from all sibling tools, which deal with content validation, PII scrubbing, or lens consistency.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The tool's purpose is self-evident as a health check, so the context for use is clear: verify that the Soulfield Lens API is up and responsive. It does not explicitly discuss alternatives or exclusions, but there are no similar sibling tools that could be confused with this one.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• scrub_piiA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden and succeeds: it discloses that replacement markers are generic [REDACTED], that finding types are only in the findings list, that names survive, that the scan is pattern-based server-side with no LLM call, that coverage is best-effort, and that SOULFIELD_API_KEY is required. This is unusually transparent behavioral disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is dense and front-loaded, with the core scan behavior stated first and critical caveats (names survive, best-effort, API key) following. There is minor redundancy between 'scrubs structured identifiers only' and 'targets structured identifiers', which prevents a perfect score.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a one-parameter tool with no output schema, the description is nearly complete: it covers input, output behavior, limitations, auth requirement, and execution model. It lacks precise structure for the findings list, but that does not hinder selection or invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description adds meaning beyond the schema's generic 'The text to scan for PII' by specifying which structured categories the text parameter is scanned for and clarifying that names/free-form PII will not be matched. This modestly exceeds the baseline.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('Scan'), a specific resource ('text' via 'hosted Lens API'), and the exact class of objects detected (structured PII and secrets). It clearly differentiates itself from the validation/consistency-focused sibling tools by emphasizing scrubbed output with [REDACTED] markers and a findings list.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear context for when the tool is appropriate: structured identifiers and secrets, not personal names or free-form PII. It explicitly excludes use cases where names must be removed and warns coverage is best-effort. It does not name alternative tools, but no direct scrubbing alternative appears among the siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• lens_consistency_leaksA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full behavioral burden and does so exceptionally: it discloses determinism, no LLM involvement, exit code 6 on hit, literal case-insensitive matching, and the important negation-blind limitation requiring adjudication before acting. This goes far beyond the schema and gives an agent realistic expectations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose and key matching semantics, then adds usage context, a critical limitation, and actionable guidance. Every sentence carries meaningful information, and the structure makes the important caveat prominent rather than burying it.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having no annotations and no output schema, the description provides enough context for an agent to call this tool appropriately: when to run it, what it catches, how it behaves, how to interpret results, and how it differs from related tools. The only minor omission is the explicit success-code behavior, but 'Exit 6 on a hit' is sufficient to signal contract expectations.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description adds useful semantic context beyond the schema by explaining what kinds of deny terms are appropriate ('strings that are wrong in EVERY context'), what files should be scanned (customer-facing), and how a hit should be interpreted. This exceeds the schema-only baseline.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Scan customer-facing files for deny-list terms (CASE-INSENSITIVE literal match...).' It also explicitly differentiates itself from the sibling lens_consistency_markers by contrasting case-sensitivity, so an agent can distinguish this tool without opening schemas.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives an explicit invocation rule: 'Run this on EVERY customer-facing file before any irreversible publish.' It also states what alternatives do not cover ('A credential scanner will not find these') and provides negative guidance by explaining that claims-you-do-not-make belong in a lens prompt rather than in the deny list.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.