Server Quality Checklist

Tool Scores

• list_oval_streamsB
  3/5.0
  Behavior1/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are supplied, and the description carries the full burden for behavioral disclosure. It only states that this lists streams, optionally filtered; it does not disclose security requirements, read-only nature, pagination, return format, or whether it accesses stream metadata. Minimal behavioral context beyond the action itself.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single concise 20-word sentence that front-loads the primary action and filters. No redundancy or descriptive fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Adequate for a simple listing tool with 2 well-described params. It lacks a cross-reference to the sibling grabbing tool 'for fetching single streams', and it does not mention what information is returned for each stream (possibly no such announcement is ok because no output schema is defined. Completeness is constrained by the absence of alternative-use and output info.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% (both 'after' and 'label' have descriptions). The description adds semantic context by explaining 'after' as a filter for modification date in ISO 8601 format and 'label' as a product version label (e.g., jbossalem-eap-8, rhel-9). This is a moderate baseline because the schema already provides complete param info.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description uses specific verb 'List' and identifies the resource 'available Red Hat OVAL v2 streams' with two explicit filter options (product label, modification date). It does not explicitly differentiate from the sibling tool 'get_oval_stream' for retrieving a specific stream, but 'List' vs 'get' plus the optional filters conveys an adequate listing vs. retrieval distinction.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this vs. alternative tools. It does not mention that 'get_oval_stream' should be used to fetch a single stream's details, nor does it state any prerequisites. The optional filters are described but no context is provided for typical usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_oval_streamA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It only says 'Retrieve OVAL stream data' and does not disclose return format, read-only guarantees, error behavior, or whether the data is raw XML, JSON, or something else.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single clear sentence with no filler. It front-loads the key verb and resource, making the tool's purpose immediately understandable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple with one fully documented parameter, but the description omits relation to list_oval_streams and does not clarify what the returned 'OVAL stream data' looks like. Without an output schema or annotations, this is minimally adequate but leaves meaningful gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already describes the only parameter 'base' with the same example values (RHEL7, RHEL9), so schema description coverage is 100%. The description adds no additional semantic meaning beyond what the schema provides, so the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Retrieve'), a clear resource ('OVAL stream data'), and a selection criterion ('specific Red Hat product by base name'). This clearly distinguishes it from siblings like list_oval_streams, which presumably enumerates available streams, and from the CVE/CSAF search tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when a specific base name such as RHEL7 or RHEL9 is known, but it does not explicitly state when to use this tool versus list_oval_streams or other alternatives. There is no when-not-to-use guidance or mention of prerequisite steps.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_csaf_detailsA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full transparency burden. It accurately implies a read-only retrieval operation and communicates the principal behavior, but it does not disclose potential error cases, response format details, or whether the document is returned whole or truncated.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  One sentence, front-loaded with the action and object, and zero filler. Every word contributes directly to understanding the tool's purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple single-parameter retrieval tool, the description plus schema are mostly sufficient. The lack of an output schema and annotations means some detail about return structure is absent, but 'full CSAF document' adequately communicates the expected result for a get-by-ID operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the input schema already fully documents the single parameter including an example format. The description adds little beyond 'by its ID,' which is redundant with the schema, so the baseline score of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description uses a specific verb ('Retrieve') and clearly identifies the resource ('full CSAF document for a specific Red Hat Security Advisory'). It also distinguishes itself from sibling search tools by emphasizing retrieval by a specific RHSA ID rather than searching.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'for a specific Red Hat Security Advisory (RHSA) by its ID' clearly indicates this tool is for fetching a known advisory's full document. It implies the user must already have the exact RHSA ID, though it does not explicitly state when to use an alternative like search_csaf.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_cve_detailsA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It indicates a read-only operation ('Retrieve') and lists the return contents, but it does not disclose error handling, availability, authentication, or behavior for invalid CVE IDs. This is adequate but lacks depth for full transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that front-loads the action ('Retrieve full details') and efficiently lists key return items without unnecessary words or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with one parameter and no output schema, the description adequately explains what the tool does and what information it retrieves. It misses edge-case behavior like not-found or invalid ID handling, but the tool's simplicity means this is nearly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description already fully covers the single parameter cve_id with an example format ('CVE-2024-1234'). The tool description adds no new semantic information beyond what the schema already provides, so with 100% schema coverage, the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Retrieve full details for a specific CVE' and enumerates specific data elements (severity, CVSS scores, affected releases, package state, mitigations, references). This clearly distinguishes it from sibling tools like search_cves (searching) and get_csaf_details/OVAL tools (different data domains).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool ('for a specific CVE') and what it returns, but it does not explicitly mention alternatives or when not to use it. The context is clear, but there is no explicit when-not guidance or reference to sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_csafA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of disclosure. It adds value by stating results are paginated summaries, but omits behavioral details like how multiple filters are combined (AND/OR), default sort order, rate limits, or maximum page size. It's not misleading, just incomplete.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two efficiently structured sentences front-load the verb and resource, enumerate the filter dimensions, and state the return type. Every clause earns its place with zero redundancy. This is an exemplary model of concise tool documentation.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that all 10 parameters are thoroughly documented in the schema and the description covers the key runtime behavior (paginated summaries), the combination is quite complete for a search tool. Minor omissions like default result ordering or behavior with no filters prevent a perfect score, but the description is strong enough for an agent to use it effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents all 10 parameters. The description adds context about the paginated, summary output but no new parameter semantics beyond what the schema already provides. Per the rubric, high schema coverage yields a baseline of 3, which is appropriate here.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Search') with a well-defined resource ('Red Hat CSAF advisory documents') and enumerates concrete filter dimensions (date, severity, CVE, package, Bugzilla ID, RHSA ID). It distinguishes itself from siblings like 'search_cves' and 'get_csaf_details' by focusing on CSAF documents and returning 'a paginated list of advisory summaries.'

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly positions this as the search/filter entry point for CSAF advisories and indicates the output is summaries, implicitly distinguishing it from detail-fetching siblings like 'get_csaf_details.' However, it does not explicitly name alternative tools or provide when-not-to-use guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_cvesA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses that the tool returns a paginated list of CVE summaries and mentions the filter dimensions. For a read-oriented search tool, this is adequate behavioral disclosure, though it omits details like error handling, pagination defaults, or whether results are sorted. Still, it is a clear and honest overview.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core purpose, and directly states the return type. Every word is necessary and there is no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (15 parameters, 0 required) and that the schema already documents all parameters, the description adequately covers the tool's purpose and output (paginated CVE summaries). There is no output schema, but the description indicates the output is a list of summaries, which is sufficient for a search tool. It could be slightly more complete by stating pagination defaults or result field, but it is not a significant gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description adds a high-level grouping of filter types (date, severity, etc.) which can help the agent understand available query capabilities, but it does not add extra meaning beyond what each parameter's description in the schema already provides. No compensation needed since coverage is complete.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool searches Red Hat CVEs by multiple specific criteria (date, severity, CVSSv3 score, CWE, package, product, advisory, Bugzilla ID) and returns a paginated list of CVE summaries. The verb 'Search' and resource 'Red Hat CVEs' are specific, and it distinguishes from siblings like get_cve_details by explicitly targeting a list result.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for searching/filtering CVEs but does not provide explicit when-to-use vs. when-not-to-use guidance or mention alternatives like search_csaf or get_cve_details. It is clear for a search tool, but without exclusions or explicit differentiation, it leaves some inference to the agent.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.