Server Quality Checklist

Tool Scores

• refinery_cache_statsB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false. The description adds that the tool is free and returns specific fields, but it does not disclose much additional behavior such as rate limits, authentication, or pagination. No contradiction exists.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded: a clear action, a concise return summary, and a useful 'Free endpoint' note. Every sentence adds value and there is no redundant filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple read-only endpoint with one optional parameter, an output schema, and rich annotations, the description is nearly complete. The only minor gap is that it does not mention the response_format parameter or clearly state output behavior, though the schema covers that.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The single response_format parameter is documented in the schema with a default and description, but the reported schema description coverage is 0% and the description itself adds no parameter guidance. Given the low coverage signal, the description should compensate but does not mention response_format at all.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action ('Get cache statistics') and the target resource ('Refinery data store'), and it lists the returned data. It does not explicitly contrast with sibling tools like refinery_health or refinery_credits, so it is clear but not fully differentiated.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool by describing what it returns, and 'Free endpoint' gives a mild cost/accessibility hint. However, it provides no when-to-use/when-not-to-use guidance or alternatives, so usage context is implied rather than explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• refinery_creditsA
  3.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already establish read-only, idempotent, non-destructive behavior. The description adds useful context by noting it is a 'Free endpoint' that queries 'upstream ZillAPI', meaning the check likely does not consume credits and reflects an external provider's balance. No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two short sentences, front-loaded with the primary action. The second sentence repeats 'credit balance' but adds the free/upstream context, so the redundancy is minor.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple read-only balance check with an output schema and full safety annotations, the description is largely sufficient. It could mention output format or usage context, but the missing pieces are small given the tool's simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With schema description coverage reported as 0%, the description carries the burden for parameter meaning, but it never mentions the response_format parameter or its values. The schema itself documents 'markdown' or 'json', so the gap is partially mitigated, but the tool description adds no parameter-level value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: 'Check remaining ZillAPI credit balance.' It is clear about what the tool does and is distinguishable from siblings like refinery_payment_status, though it does not explicitly name a sibling to exclude.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Usage is implied by the purpose: call it when you need the remaining credit balance. It gives no explicit when-to-use versus alternatives or prerequisites, though none of the sibling tools obviously overlaps with this functionality.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• refinery_get_propertyA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already convey read-only and non-destructive behavior. The description adds valuable behavioral context beyond annotations: this is a paid lookup ($0.35 via x402), conditional on REFINERY_ENABLE_X402=false. It also names the included enrichment categories, which helps set expectations for the response.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded: purpose first, then cost/behavior, then included data. Every sentence contributes, though the final sentence partially restates the 'full scored details' idea from the opening.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-required-parameter lookup with an output schema and useful annotations, the description covers the essential purpose, the paid-lookup caveat, and the included contents. It lacks explicit sibling-tool routing, but that is not critical for a focused property detail lookup.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description mentions ZPID as the key input, but that is also present in the schema. It does not explain the response_format parameter or add any validation/format nuance beyond the schema. With schema description coverage reported as 0%, the description does not compensate for that gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Get') with a clear resource ('full scored details for a single property by ZPID'), immediately identifying the tool's purpose and distinguishing it from list/search/history sibling tools. 'By ZPID' also clarifies the scope of the lookup.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool: when you have a single ZPID and need full enriched property details. However, it does not explicitly contrast this with sibling tools like search_properties, list_properties, or geolice_history, and it does not state when not to use this tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• refinery_get_tax_historyA
  3.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the readOnlyHint/openWorldHint annotations, the description discloses a concrete paid-lookup behavior with a $0.25 x402 fee and an environment toggle (REFINERY_ENABLE_X402=false) that changes how payment applies. This is exactly the kind of payment/auth/rate context annotations do not capture, and it does not contradict the idempotentHint=false annotation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two short sentences, with the core purpose front-loaded and the payment caveat following immediately. Every sentence earns its place and there is no redundant filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-required-parameter tool with an output schema and annotations, the description plus schema cover the essential calling requirements. The only minor gap is that the conditional payment line is slightly ambiguous about what happens when REFINERY_ENABLE_X402=false, but that does not block correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Context signals indicate 0% schema description coverage, so the description must compensate for parameter documentation. It mentions ZPID in prose but does not explain response_format, its default, or the fact that zpid is required; the schema still carries most of the parameter burden.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a concrete verb and object: 'Get the tax/assessment history for a property by ZPID,' which clearly identifies the resource and lookup key. It does not explicitly contrast with sibling tools like refinery_get_price_history or refinery_get_schools, so it falls just short of full differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The first sentence implies the intended use: call this when you need tax/assessment history for a given ZPID. The payment note adds a practical condition, but the description does not state when not to use it or name alternative tools, leaving selection among tax/price/school siblings to inference.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• refinery_healthA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already cover read-only, idempotent, and non-destructive behavior. The description adds valuable behavioral context: it returns status, cached property count, rate limit info, and explicitly says no x402 payment is required. No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise: three short sentences, front-loaded with the main purpose, then return details and cost note. No unnecessary wording.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple read-only health endpoint with one optional parameter, an output schema, and strong annotations, the description is complete. It tells the agent what the endpoint does, what it returns, and clarifies that payment is not needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool description does not explain the response_format parameter, but the schema itself provides a description, default, and enum values. With one well-documented user-facing parameter, the description does not significantly add parameter meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the resource ('RE Data Refinery API health') and states what it returns. It is distinctive and actionable, but it does not explicitly differentiate itself from the sibling refinery_cache_stats, which may overlap on cache status.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use for checking API health and cache status, and usefully notes that the endpoint is free and does not require payment. However, it does not name alternatives or state when not to use this tool versus siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• refinery_payment_statusA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so the safety profile is fully covered. The description adds context about the specific status information (payment configuration and base URL), but does not disclose additional behavioral traits such as authentication needs or response characteristics.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence containing exactly the necessary information: what is checked and what is returned. There is no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple, parameterless read-only status tool with an output schema present, the description is complete. It tells the agent exactly what state it reports on, and no further detail about return values is needed because the output schema presumably covers that.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and the schema already reflects this with an empty properties object. With no parameters to document, the description has no obligation to explain parameter semantics; the baseline of 4 for zero-parameter tools applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Show') and a clear resource ('x402 automatic payments are configured' and 'active API base URL'). This clearly identifies the tool's purpose and differentiates it from siblings like refinery_health or refinery_credits, none of which mention payment configuration or base URL.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not provide any guidance on when to use this tool versus alternatives, nor does it mention exclusions or prerequisites. It only states what the tool shows, leaving the agent to infer the appropriate context from the tool name and siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• refinery_get_schoolsA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The readOnlyHint and destructiveHint annotations already establish that this is a safe read operation. The description adds valuable behavioral context beyond annotations by disclosing that this is a paid lookup ($0.25 via x402) with a configuration flag to disable payment.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two short sentences with no filler. The core action and lookup key are front-loaded, and the cost note is placed separately for clarity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-required-parameter read-only tool with an output schema, the description is largely complete: it states what is retrieved, how it is retrieved, and the cost. It does not offer explicit usage routing to sibling tools, but that is not a critical gap for this simple operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description names ZPID as the lookup key, which aligns with the required zpid parameter. However, it does not mention the optional response_format parameter or the params wrapper structure. The schema provides some description for these fields, but the schema coverage signal is low, so the description only partially compensates.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb and resource: 'Get the school ratings near a property by ZPID.' It clearly distinguishes this from sibling tools like price history and tax history by naming the school-ratings domain and the property-lookup key.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the intended use case: retrieve school ratings when you have a ZPID. However, it does not explicitly state when to use this tool instead of refinery_get_property, refinery_get_price_history, or refinery_get_tax_history, nor does it mention exclusions or alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• refinery_list_propertiesA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds meaningful behavioral context beyond the readOnlyHint and destructiveHint annotations by explaining the paid live-data path ($0.35 via x402) and the free cached-data path with no payment. It also clarifies the default behavior (live=true), which is important for cost-sensitive agents. It does not contradict the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three short sentences with zero filler. The primary purpose is front-loaded, followed immediately by the most important operational distinction (live vs cache). Every sentence contributes necessary information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a list tool with read-only annotations and an output schema, the essential behavioral and cost information is covered. Missing details like explicit limit/response_format guidance are available in the nested schema, and the only notable gap is lack of guidance on when to choose this tool over sibling search tools.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description names the key filtering dimensions (city, price) and provides important detail for the live parameter, including default and cost. However, it does not mention limit, response_format, or min/max price specifics, so it relies on the nested schema to fully document those. The cost context is a genuine addition beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb and resource ('List scored properties') with clear scoping dimensions ('by city' and 'optional price filtering'). It conveys a distinct purpose relative to siblings like refinery_search_properties, though it does not explicitly name an alternative or contrast itself.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear, actionable guidance on the critical live vs cache decision, including cost, payment requirement, and defaults. It does not explicitly discuss when to prefer this tool over refinery_search_properties, but it provides enough context for the main branching decision.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• refinery_get_price_historyA
  4.3/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds a critical behavioral trait not present in annotations: this is a paid lookup costing $0.25 via x402, with an environment-variable escape hatch. This goes well beyond the readOnlyHint and destructiveHint annotations and tells the agent what operational consequence to expect.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no filler: the main purpose is front-loaded, and the second sentence delivers the important payment condition. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a read-only single-resource lookup with an available output schema, the description, combined with annotations, is complete enough. An agent knows what the tool does, what input it needs, that it is safe, and that it has a cost implication. No critical call-blocking detail is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description names the central required input, 'ZPID', which is helpful, but it does not explain the params wrapper or the optional response_format, and it adds little meaning beyond what the input schema already provides for zpid and response_format. The schema coverage signal is low, so the description only partially compensates.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action ('Get'), a specific resource ('price-history timeline'), and the key input ('by ZPID'). This clearly distinguishes it from sibling tools like refinery_get_tax_history and refinery_get_schools, which cover different property subresources.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The usage context is implied rather than explicit: an agent can infer this tool is for price-history lookups, but the description does not explicitly name alternatives or state when not to use it. The paid-lookup caveat is a cost consideration, not guidance about choosing between sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• refinery_search_propertiesA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the annotations, the description discloses two important behavioral traits: this is a paid lookup costing $0.50 via x402 unless REFINERY_ENABLE_X402=false, and results are written through to the Refinery cache. These are valuable operational details that affect invocation decisions and are not present in the structured fields.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three concise sentences with no filler. The primary action is front-loaded, followed by the cost condition and cache side effect, making it easy for an agent to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers purpose, cost, payment bypass condition, and cache behavior, and the output schema handles return-value details. It does not explicitly guide an agent toward sibling alternatives or define what 'scored' means, but for a search tool with this schema, the essential context is present.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds some parameter meaning by specifying that the search uses natural language terms, which directly informs the query parameter. However, it does not describe limit or response_format; the nested input schema provides those details, so the description is sufficient but not rich.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('Search'), a specific resource ('scored Columbus, OH properties'), and a distinctive method ('using natural language terms'). This separates it from sibling tools like list_properties or get_property, which imply different operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly positions this as the natural-language search tool for scored Columbus, OH properties, providing enough context for an agent to select it over a plain list/get tool. It does not explicitly name when-not-to-use alternatives, but the core usage context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.