AI Crawler Index

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint, and destructiveHint, so the safety profile is covered. The description adds genuinely useful behavior beyond annotations: the cursor advances only on a real change, data refreshes every 6h, and the first call requires no parameters. It does not describe pagination behavior or failure modes, but the added context is meaningful.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is dense but efficient, front-loading the critical first-call behavior before the cursor mechanics. The inline whoami JSON is slightly cluttering but serves a concrete purpose by showing the exact first-call payload; almost every element earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a low-complexity tool with two optional params and no output schema, the description covers the key operational facts: invocation pattern, cursor semantics, freshness cadence, and the categories of changes returned. The main gap is that no output schema exists and the description does not sketch the response shape, but the explanation is otherwise sufficient for correct invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so both `since` and `limit` are already documented, giving a baseline of 3. The description adds value on top by illustrating the special value since='0' and clarifying that the cursor comes from the previous result and advances only on real changes, which enriches the schema's terse 'Cursor from the last result'.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool returns what changed since the last call and enumerates the change categories (prefixes added/removed, upstreams failed/recovered, records edited), which is a specific resource scope. However, the sibling tool `whats_changed` has a nearly identical purpose and is never referenced, so sibling differentiation is absent.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly walks through the correct usage pattern: the FIRST CALL needs nothing and suggests the whoami call, then instructs the agent to send back the cursor from the previous result. It explains the cursor's advance semantics, but it does not state when to prefer this over the similar `whats_changed` sibling or give exclusion conditions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already establish the tool as read-only, idempotent, and non-destructive. The description adds useful behavioral context: it returns multiple classification facets, flags that a user-agent is not proof, and discloses an equivalent direct HTTP endpoint. No contradiction with annotations is present.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Most of the description is tight and useful, with a concrete example and a clear HTTP alternative. However, the leading 'FIRST CALL, needs nothing: {"name":"whoami","arguments":{}}' is irrelevant, confusing, and poorly structured. It should be removed or clarified, as it detracts from the otherwise efficient text.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a single-parameter, safe, idempotent tool with no output schema, the description provides sufficient context: output facets, a concrete example, a verification pointer, and a non-MCP invocation method. Minor gaps are the ambiguous whoami prefix and lack of differentiation from lookup_crawler, but the core information is complete enough to invoke the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, with the parameter already described as 'Raw User-Agent header value.' The description reinforces this and provides a concrete example ('GPTBot/1.2'), but it does not add meaningful semantic detail beyond what the schema already provides.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states what the tool does: identify the crawler behind a raw User-Agent header and return operator, category, robots.txt stance, verification method, and blocking cost. It also distinguishes itself from is_verified_crawler_ip by noting that a user-agent is a claim requiring confirmation. However, the confusing 'FIRST CALL, needs nothing: whoami' prefix muddies the purpose and no distinction from lookup_crawler is given.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly advises confirming the result with is_verified_crawler_ip, providing clear context for when additional verification is needed. It implies this tool is the initial classification step. It lacks an explicit when-not-to-use statement or guidance on alternatives like lookup_crawler, but the verification pointer is strong.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations, it adds concrete behavioral details: deterministic, read-only, touches no third party, requires no API key or account, uses only the host's own canned data, and returns a real structuredContent rather than a mock. This strongly supports safe agent invocation.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is front-loaded but highly repetitive: 'no third party' and 'server's own data' are stated multiple times, and several sentences restate the same idea. It could be reduced to two or three crisp sentences without losing information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a zero-argument, read-only example tool with an output schema, the description covers everything needed: what it does, why to call it, what it returns, and that it is safe and deterministic. Nothing essential is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has zero parameters, and the description explicitly states 'Takes no arguments' and shows arguments={}. There is no parameter ambiguity, and the schema already covers this completely.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool runs this server's own worked example by invoking classify_user_agent on canned input and returning the real structuredContent. It distinguishes itself from sibling tools by being a safe, no-argument demo rather than a live classification or lookup call.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly tells the agent to use it to see the shape of an answer before sending real data. It does not enumerate formal exclusions or alternatives, but the intended usage context is clear and actionable.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint, and non-destructive behavior, so the safety profile is covered. The description adds useful behavioral detail: the output is a ready-to-paste block, includes reasoning and crawler names, and behaves polymorphically when the stance argument is omitted. No contradictions with annotations were found.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is short but not cleanly structured. The opening phrase plus full whoami JSON snippet is tangential and confusing for a tool whose actual invocation uses a stance argument, and it competes with the real description. The later sentences are concise, but the misplaced first-call instruction adds noise rather than earning its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple, single-parameter, read-only tool, the description covers the key context: what the output contains, how to get a single stance, and how to get all stances. There is no output schema, so the description appropriately describes the return value's main elements. The confusing whoami reference is the main gap, but it does not leave the agent without enough information to invoke the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema already fully documents the stance parameter, including all allowed values, so the baseline is 3. The description adds meaningful semantic value by explaining that the parameter is optional and that omitting it produces all eight stances, plus a concrete example value. This goes beyond what the schema states about invocation behavior.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The core sentence clearly states the deliverable: a ready-to-paste robots.txt block for a stance, including reasoning and named crawlers. It also notes that omitting the argument returns all eight stances, which sharpens the scope. However, the opening 'FIRST CALL, needs nothing: {"name":"whoami","arguments":{}}' is distracting and could be misread as the actual invocation for this tool, slightly undermining clarity.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives some usage context: it suggests this can be an early call, says it needs nothing, and explains that omitting the argument returns all stances. But it does not explicitly tell the agent when to choose this over sibling tools like list_crawlers or classify_user_agent, nor does it state any when-not-to-use conditions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already carry the safety profile (readOnlyHint=true, idempotentHint=true, destructiveHint=false), so the bar is lower; the description still adds valuable behavior beyond them: the six-hour remirror cadence (data freshness), the return shape (matching prefix, source, crawlers), and the 'a miss is reported as a miss, not as a forgery' guarantee that makes negative results trustworthy. This is consistent with openWorldHint=false. No contradiction.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The core action, freshness, output, and example are compact and mostly front-loaded, and the HTTP fallback earns its place. However, the embedded whoami JSON ('FIRST CALL, needs nothing: {"name":"whoami","arguments":{}}') is workflow noise — an instruction to invoke a different tool smuggled into this tool's description — and the 'FIRST CALL' phrasing is ambiguous.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a one-parameter, read-only, idempotent tool with no output schema, the description is nearly complete: it covers input format, matching behavior, data freshness, output contents, miss semantics, and a non-MCP access path. Because no output schema exists, the description correctly takes on the return-value explanation. Only minor gaps remain (malformed-input errors, rate limits), which are not critical for a simple lookup.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% — the sole parameter ip is already described as 'IPv4 or IPv6 address to check' — so the baseline is 3. The description adds a concrete example value (ip='20.171.206.5') and confirms the address format, but it contributes no additional syntax, constraints, or edge-case detail beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a concrete action — 'Test an IPv4 or IPv6 address against every crawler prefix the operators themselves publish' — with a specific verb (test) and resource (an IP against crawler ranges). The title reinforces the same function, and no sibling (list_crawlers, lookup_crawler, classify_user_agent) overlaps with IP-range verification, so an agent can distinguish it without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives some deployment context — 'FIRST CALL, needs nothing' and the non-MCP HTTP fallback — but it never names sibling alternatives or states when to prefer this over lookup_crawler or list_crawlers. The whoami preamble is ambiguous about whether this tool or whoami should be called first, so routing guidance is implied rather than explicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already establish the tool as read-only, idempotent, and non-destructive. The description adds useful behavioral detail by saying each row reveals what the crawler is and what blocking it costs, and that the reply lists every valid filter value. No contradiction with the annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact and frontline the key 'FIRST CALL, needs nothing' guidance. The embedded whoami JSON is slightly noisy and potentially confusing inside a list_crawlers description, but the overall structure is efficient and scannable.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a zero-required-parameter, read-only list tool with rich schema coverage and no output schema, the description sufficiently explains input behavior, row meaning, and dynamic filter value discovery. It lacks explicit sibling differentiation, but the core usage context is complete enough for an agent to proceed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so each parameter already has a description. The description adds value by giving a concrete example and, more importantly, by stating that the reply lists every valid filter value, which helps an agent discover acceptable values dynamically rather than guessing.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The title and description clearly say this tool lists and filters crawlers, and the description adds that it is 'the index' with each row describing the crawler and its blocking cost. This is specific enough to understand the core function, though it does not explicitly distinguish itself from sibling lookup_crawler.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives a concrete usage example (category='ai-training', limit=20) and indicates this is a 'FIRST CALL' tool that 'needs nothing'. However, it does not explicitly state when to use list_crawlers versus alternatives like lookup_crawler, classify_user_agent, or changes_since, leaving some routing to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare the operation read-only, idempotent, and non-destructive. The description adds valuable behavioral context: the full record contents, near-match behavior for unknown slugs, and a required first call to whoami. No contradiction with annotations exists.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The core description is dense and efficient, listing return fields without padding. However, the opening 'FIRST CALL, needs nothing: {"name":"whoami","arguments":{}}' is a bulky, confusing preamble about a different tool, and it occupies the front-loaded position. Each sentence is informative, but the structure sacrifices clarity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With one required parameter, no output schema, and read-only annotations, the description adequately covers the tool: it names the return fields, explains unknown-slug behavior, and gives the input format. It does not detail the output structure or fully explain the whoami prerequisite flow, but the tool itself is sufficiently specified for correct invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the slug parameter is already documented. The description adds a concrete example and notes that unknown slugs return near matches, which is useful but not essential. Baseline 3 is appropriate because the schema carries the semantic weight.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool looks up one crawler and enumerates the returned fields: operator, category, robots.txt stance, user-agent substring, verification method, IP-range endpoint, and blocking cost. The word 'one' helps differentiate it from list_crawlers, but the opening 'FIRST CALL' instruction injects unrelated context that slightly muddles the primary purpose.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no explicit guidance about when to use this tool versus siblings like list_crawlers, classify_user_agent, or is_verified_crawler_ip. The 'FIRST CALL' note is a sequencing instruction about whoami, not a selection criterion. The example slug implies usage but does not clarify conditions or exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already mark the call read-only, idempotent, and non-destructive; the description adds valuable context about what information is returned, such as build time, source freshness, changes, failures, and the changelog. It also signals that the since parameter is optional. There is no contradiction with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The definition packs a lot of useful information into a few sentences, but the opening 'FIRST CALL, needs nothing: {"name":"whoami","arguments":{}}' is structurally confusing because it references a different tool and could distract an agent. The rest is compact and mostly front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple read-only tool with one optional parameter and annotations covering the safety profile, the description adequately covers what results to expect and when to call it. It does not need to document a return schema because none exists, though clarifying the relationship to changes_since would make it fully complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema already fully explains since as an ISO-8601 date or timestamp that can be omitted for current state. The description adds only a short example, not additional semantic meaning, so it meets the baseline without exceeding it.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly names the tool's output: freshness and change state of the index and operator sources, including when sources were last fetched, which changed, which are failing, and the dated changelog. This is specific enough to know what the tool does, though it does not explicitly distinguish it from the sibling changes_since.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives concrete use context: this is presented as the first call and the cron check before regenerating a config from the data, with an example since parameter. It does not name alternatives or exclusion conditions, and the embedded whoami snippet is somewhat confusing, but the main when-to-use guidance is clear.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already provide readOnlyHint and idempotentHint, and the description goes well beyond them by disclosing that the tool touches no third party, fetches nothing, is deterministic, requires no API key/account/consent, and derives every fact from request headers or an already-published file. There is no contradiction with the annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is far longer than needed and repeatedly restates the same output fields ('user-agent, address, class, seen before') and the same reassurances ('free, read-only, no API key, no account, no consent prompt'). It opens with a full JSON-RPC body instead of a concise summary, making it harder to scan.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a zero-argument introspection tool, nothing is missing: it explains what the tool returns, where the data comes from, that no side effects occur, and how it relates to the server's classifiers. An output schema exists, and the description is consistent with it.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With zero parameters and 100% schema coverage, there is almost nothing to explain. The description still adds useful confirmation by saying 'Takes no arguments,' 'no argument exists,' and giving an explicit arguments={} example, so the agent will not invent parameters.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly identifies the tool as a no-argument, read-only introspection call: it 'describes YOU, the caller' and returns the caller's user-agent, address, classification, and seen-before status. It stops short of explicitly distinguishing itself from sibling tools classify_user_agent and is_verified_crawler_ip, though it references them as internal building blocks.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The appropriate context is implied: use this when you want information about the current caller rather than classifying a supplied user-agent or IP, and the description stresses 'Takes no arguments' and 'nothing for you to invent.' However, it never explicitly says when to prefer this over the sibling tools or when not to use it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.