Skip to main content
Glama
BitingSnakes

silkworm-mcp

by BitingSnakes

silkworm-mcp

This is a full-featured MCP server for building scrapers with:

  • silkworm-rs: async crawling, fetching, follow links, and spider execution

  • scraper-rs: fast Rust-backed HTML parsing with CSS and XPath selectors

It is designed for LLM-assisted scraper development, so the server exposes both low-level page inspection tools and higher-level workflow helpers for validating selector plans and generating starter spider code.

An example: https://github.com/BitingSnakes/silkworm-example

Features

  • Fetch pages through silkworm's regular HTTP client or CDP renderer.

  • Query selectors directly against a CDP-rendered DOM snapshot.

  • Analyze inline and linked CSS with tinycss2, then optionally map selectors back onto HTML.

  • Extract structured records from live rendered pages before committing to a full crawl.

  • Cache HTML in a local document store and reuse it via document_handle.

  • Bound the document cache with max-document, max-bytes, and idle-TTL controls.

  • Inspect pages with summaries, parsed DOM trees, prettified HTML, CSS/XPath queries, selector comparisons, and link extraction.

  • Run ad hoc crawls from a structured CrawlBlueprint.

  • Generate reusable silkworm spider templates from the same blueprint and statically validate them, including pattern-specific variants for list-only, list+detail, sitemap/XML, and CDP-heavy crawls.

  • Expose MCP diagnostics plus HTTP /healthz and /readyz routes for production monitoring.

  • Publish MCP resources and prompts so clients can discover workflows, Silkworm idioms, and blueprint schemas.

Related MCP server: Selenium MCP Server

Tools

  • store_html_document

  • list_documents

  • delete_document

  • clear_documents

  • server_status

  • inspect_document

  • parse_html_document

  • parse_html_fragment

  • prettify_document

  • query_selector

  • analyze_css_selectors

  • find_selectors_by_text

  • compare_selectors

  • extract_links

  • silkworm_fetch

  • silkworm_fetch_cdp

  • query_selector_cdp

  • extract_structured_data_cdp

  • run_crawl_blueprint

  • generate_spider_template

  • validate_spider_code

Run

Install dependencies:

uv sync

Run over stdio for a desktop MCP client:

uv run python mcp_server.py --transport stdio

Run over HTTP:

uv run python mcp_server.py --transport http --host 127.0.0.1 --port 8000

HTTP deployments also expose:

  • GET /healthz: process liveness

  • GET /readyz: readiness, optionally including a CDP browser probe

The project also exposes a console entrypoint:

uv run silkworm-mcp --transport stdio

Docker

Build the image:

docker build -t silkworm-mcp .

Run the container over HTTP on port 8000:

docker run --rm -it -p 8000:8000 silkworm-mcp

The container entrypoint starts two processes by default:

  • the MCP server over HTTP on 0.0.0.0:8000

  • a bundled Lightpanda browser on 127.0.0.1:9222 for CDP-backed tools such as silkworm_fetch_cdp, query_selector_cdp, and extract_structured_data_cdp

Useful container environment variables:

  • MCP_TRANSPORT (default: http)

  • MCP_HOST (default: 0.0.0.0)

  • MCP_PORT (default: 8000)

  • MCP_PATH

  • LIGHTPANDA_ENABLED (default: 1)

  • LIGHTPANDA_HOST (default: 127.0.0.1)

  • LIGHTPANDA_PORT (default: 9222)

  • LIGHTPANDA_ADVERTISE_HOST (default: unset, falls back to LIGHTPANDA_HOST)

  • LIGHTPANDA_LOG_FORMAT (default: pretty)

  • LIGHTPANDA_LOG_LEVEL (default: info)

When Lightpanda binds to 0.0.0.0 inside a container, set LIGHTPANDA_ADVERTISE_HOST to a reachable hostname such as the container DNS name. Otherwise /json/version can advertise ws://0.0.0.0:9222/, which remote CDP clients cannot use.

Example with custom document-cache limits:

docker run --rm -it \
  -p 8000:8000 \
  -e SILKWORM_MCP_DOCUMENT_MAX_COUNT=256 \
  -e SILKWORM_MCP_DOCUMENT_MAX_TOTAL_BYTES=64000000 \
  -e SILKWORM_MCP_DOCUMENT_TTL_SECONDS=7200 \
  silkworm-mcp

For local development, compose.yml provides the same setup with health checks and restart policy:

docker compose up --build

Then verify the container is ready:

curl http://127.0.0.1:8000/readyz

Key runtime environment variables:

  • SILKWORM_MCP_DOCUMENT_MAX_COUNT

  • SILKWORM_MCP_DOCUMENT_MAX_TOTAL_BYTES

  • SILKWORM_MCP_DOCUMENT_TTL_SECONDS

  • SILKWORM_MCP_DOCUMENT_STORE_PATH

  • SILKWORM_MCP_LOG_LEVEL

  • SILKWORM_MCP_READINESS_REQUIRE_CDP

  • SILKWORM_MCP_READINESS_CDP_WS_ENDPOINT

Example Workflow

  1. Call silkworm_fetch for the target page.

  2. Use the returned document_handle with inspect_document.

  3. Use parse_html_document or parse_html_fragment when you need exact parser structure, node types, or parser errors.

  4. Use find_selectors_by_text to derive candidates from visible text, then iterate on query_selector, compare_selectors, and analyze_css_selectors when stylesheet structure or hidden elements matter.

  5. For JS-heavy pages, use query_selector_cdp or extract_structured_data_cdp against the rendered DOM.

  6. Use extract_links to verify pagination or detail pages.

  7. Feed the stable plan into run_crawl_blueprint.

  8. Convert the same blueprint into code with generate_spider_template, then check it with validate_spider_code.

Useful built-in MCP references:

  • silkworm://reference/overview

  • silkworm://reference/silkworm-cheatsheet

  • silkworm://reference/silkworm-playbook

  • silkworm://reference/template-variants

  • silkworm://reference/scraper-rs-cheatsheet

  • silkworm://reference/crawl-blueprint-schema

Use transport: "cdp" when pages require JavaScript rendering. run_crawl_blueprint will connect to the configured CDP endpoint, and generate_spider_template will emit a starter spider that runs through CDPClient instead of the default HTTP client.

Both run_crawl_blueprint and generate_spider_template accept a variant override. When omitted, they infer a crawl style from the blueprint:

  • list_only: listing pages emit items directly, with optional pagination

  • list_detail: listing pages schedule detail requests and a separate parse_detail

  • sitemap_xml: sitemap/XML entrypoints are fetched with meta={"allow_non_html": True} and parsed before scheduling page requests

  • cdp_heavy: rendered-page crawls keep the CDP execution path and a general-purpose parse/follow flow

run_crawl_blueprint returns the resolved execution_variant, and generate_spider_template returns the resolved template_variant, so clients can see which crawl shape was actually used.

Testing

Run the automated test suite with:

just test

Acknowledgement

This project builds on the excellent work behind FastMCP, silkworm-rs, and scraper-rs. Together they provide the MCP server framework, crawling runtime, and HTML parsing foundations that make this project possible.

Available Tools

22 tools
analyze_css_selectorsC

Parse inline, linked, or raw CSS with tinycss2 and optionally match selectors back onto HTML.

ParametersJSON Schema
NameRequiredDescriptionDefault
cssNo
htmlNo
limitNo
html_charsNo
match_htmlNo
source_urlNo
text_charsNo
match_limitNo
max_size_bytesNo
document_handleNo
timeout_secondsNo
truncate_on_limitNo
include_match_htmlNo
include_inline_stylesNo
only_hiding_selectorsNo
fetch_linked_stylesheetsNo
include_linked_stylesheetsNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
warningsNo
selectorsNo
source_urlNo
stylesheetsNo
document_handleNo
total_selectorsYes
omitted_selectorsYes
total_stylesheetsYes
returned_selectorsYes
hidden_selector_countYes
linked_stylesheet_urlsNo

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden for disclosing behavior. It only states the core function (parse/match) and does not mention limits, truncation, error handling, read-only nature, or any side effects. The many limit-related parameters suggest important behaviors that are left undisclosed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, concise sentence with no redundant information. It is front-loaded with the core purpose and uses efficient language.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 17 parameters, no annotations, and no parameter descriptions, this description is far too minimal. While an output schema exists and covers return values, the complex behaviors around selector matching, limits, linked stylesheet fetching, and truncation are entirely absent, leaving the agent without critical usage context.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, yet the description does not compensate by explaining parameter meanings. It loosely mentions 'inline, linked, or raw CSS' but does not map that to specific parameters like css, source_url, or document_handle. None of the 17 parameters (e.g., match_html, limit, fetch_linked_stylesheets) are explained in the description.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states what the tool does: 'Parse inline, linked, or raw CSS with tinycss2 and optionally match selectors back onto HTML.' The verb 'parse' plus the resources 'CSS' and 'HTML' make the purpose specific and distinguish it from sibling tools like query_selector or parse_html_document.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives. It does not mention any prerequisites, exclusions, or scenarios where a sibling tool would be more appropriate. The only implied use is having CSS to analyze, but no explicit comparison to other tools is given.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

clear_documentsA

Clear every cached document from the in-memory store.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

Output Schema

ParametersJSON Schema
NameRequiredDescription
deleted_countYes

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of disclosure. It clearly states that the operation clears all cached documents from the in-memory store, implying a bulk, possibly irreversible action. It does not mention return values or side effects, but for a simple clear-all operation this is sufficient.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single concise sentence that is front-loaded with the verb 'Clear' and immediately specifies the scope. Every word contributes meaning, with no redundancy or filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the low complexity (zero parameters) and the existence of an output schema, the description is largely complete. It specifies the target (cached documents) and storage (in-memory), which is valuable context. It could have added a note about irreversibility or distinction from delete_document, but these are minor gaps.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has zero parameters, so the description does not need to explain any parameter semantics. The baseline of 4 applies, and the description correctly avoids unnecessary parameter details.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Clear') and the target ('every cached document from the in-memory store'). It distinguishes itself from sibling tools like delete_document, which presumably targets a single document, and from list_documents/store_html_document.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for clearing the entire cache, but it does not explicitly state when to use it versus alternatives like delete_document, nor does it mention any exclusions or warnings. Guidance is minimal and must be inferred from the wording.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

compare_selectorsB

Compare multiple selectors against the same document to see which one is the most stable.

ParametersJSON Schema
NameRequiredDescriptionDefault
htmlNo
modeNocss
selectorsYes
html_charsNo
source_urlNo
text_charsNo
include_htmlNo
preview_limitNo
max_size_bytesNo
document_handleNo
truncate_on_limitNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
modeYes
source_urlNo
comparisonsNo
document_handleNo

TDQS

B3.1/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden. It implies a read-only comparison but omits return format, stability criteria, or side effects, leaving behavioral expectations unclear.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, front-loaded sentence with no redundancy. Every word contributes to stating the tool's purpose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite the presence of an output schema, the tool's complexity (11 parameters) and lack of usage/behavioral guidance make the one-sentence description insufficient for correct invocation. It covers only the core intent, not the operational context.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% and the description adds no meaning to any of the 11 parameters, including the required selectors array and key options like mode, html_chars, and truncate_on_limit. This creates a significant gap.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly states the action (compare multiple selectors against a document) and the outcome (determine which is most stable). It distinguishes itself from siblings like query_selector and analyze_css_selectors by its focus on comparative stability.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implicitly suggests use when evaluating selector stability but provides no explicit when-not-to-use or alternatives. Sibling tools such as analyze_css_selectors might be relevant, but no comparison is given.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

delete_documentC

Delete a cached document handle.

ParametersJSON Schema
NameRequiredDescriptionDefault
handleYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
handleYes
deletedYes

TDQS

C2.8/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden of behavioral disclosure. It only says 'delete' but does not state whether the deletion is irreversible, whether it affects the underlying document or just the cached handle, or what happens to associated data. No permissions or side effects are mentioned.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness2/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence and technically concise, but it is essentially a restatement of the tool name with no additional informative content. It under-specifies rather than earning its place with useful detail.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite being a simple tool, the description lacks important context such as the distinction between a cached handle and the actual document, potential errors, or any prerequisite conditions. The output schema exists but does not compensate for the missing behavioral context of a destructive operation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The sole parameter 'handle' has 0% schema description coverage, and the description only repeats the word 'handle' without clarifying its format, origin, or how it should be passed. This provides minimal added meaning beyond the schema's raw type declaration.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Delete') and the specific resource ('a cached document handle'). This distinguishes it from sibling tools like clear_documents, which likely deletes all documents. The singular 'a handle' aligns with the single required parameter.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided on when to use this tool versus alternatives such as clear_documents. The description does not mention prerequisites, comparisons, or any context for choosing this tool.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

extract_structured_data_cdpC

Fetch a rendered page through CDP and extract structured records from the rendered DOM.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
labelNo
fieldsYes
headersNo
item_limitNo
item_xpathNo
ws_endpointNows://127.0.0.1:9222
item_selectorNo
max_size_bytesNo
store_documentNo
timeout_secondsNo
truncate_on_limitNo
include_source_urlNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
urlYes
viaNo
itemsNo
statusNo
headersNo
summaryYes
final_urlYes
scope_modeNo
scope_queryNo
total_scopesYes
omitted_itemsYes
returned_itemsYes
document_handleNo

TDQS

C2.6/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description must carry full behavioral disclosure. It mentions CDP and DOM extraction, but omits significant behaviors such as the store_document parameter (which defaults to true and could persist the document), possible side effects, timeout behavior, or error conditions. The description is too thin to make the tool's side effects and operational details transparent.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is exactly one sentence with no filler or redundancy. It front-loads the main purpose (fetch and extract) and uses precise terminology (CDP, rendered page, structured records). Every word contributes meaning, and there is no unnecessary context.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness1/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's high complexity (13 parameters, no annotations, and an output schema not shown in the input), a one-line description is grossly inadequate. It fails to explain how the fields array works, what item selectors do, how store_document affects state, or what the structured output looks like. This leaves a knowledgeable agent without the context needed to invoke the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% for the 13 parameters, yet the description provides no insight into any of them. It does not mention url, fields, item_selector, item_xpath, store_document, or any of the many configuration options. The agent must rely solely on raw schema types, which is insufficient for a tool this complex.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description 'Fetch a rendered page through CDP and extract structured records from the rendered DOM' clearly identifies the core purpose: use CDP to fetch a rendered page and extract structured records. It uses specific verbs and a specific resource, which distinguishes it from other tools that just fetch or query selectors. However, it does not explicitly name or differentiate from siblings like silkworm_fetch_cdp or query_selector_cdp, so it lacks strong sibling differentiation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no guidance about when to use this tool versus alternatives. The description implies it is for rendered pages, but does not explicitly state prerequisites (e.g., a running CDP endpoint), when not to use it, or which sibling tools are more appropriate for simpler fetches or queries. This leaves the selection criteria unclear.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

find_selectors_by_textC

Find CSS/XPath selector candidates for the smallest matching elements by text.

ParametersJSON Schema
NameRequiredDescriptionDefault
htmlNo
limitNo
match_typeNoexact
source_urlNo
text_charsNo
text_queryYes
case_sensitiveNo
max_size_bytesNo
document_handleNo
truncate_on_limitNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
matchesNo
match_typeYes
source_urlNo
text_queryYes
total_matchesYes
case_sensitiveYes
document_handleNo
omitted_matchesYes
returned_matchesYes

TDQS

C2.8/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden of behavioral disclosure. It mentions that it finds "smallest matching elements," which hints at an algorithmic behavior, but it does not disclose how the tool obtains HTML (via html, source_url, or document_handle), potential side effects, or performance characteristics. The description is too thin for a tool with 10 parameters.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, front-loaded sentence with zero wasted words. It is concise and communicates the core purpose efficiently, but given the tool's complexity (10 parameters), it is arguably under-sized. Still, it does not contain redundant information, so it earns a high score, though not a perfect 5.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite having an output schema, the description is too brief to be complete for such a complex tool. It does not explain the tool's role among many selector-related siblings, nor does it provide context on input sources, matching behavior, or output structure. The agent is left with many unknowns for invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate, but it only mentions "text" which trivially maps to text_query. There is no explanation of match_type, case_sensitive, limit, truncate_on_limit, or the three mutually exclusive input paths (html, source_url, document_handle). The description adds no meaning beyond what is structurally obvious.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ("Find") with a clear resource ("CSS/XPath selector candidates") and a qualifier ("for the smallest matching elements by text"). This clearly distinguishes it from sibling tools like query_selector or analyze_css_selectors, which serve different purposes.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no explicit when-to-use guidance or mention of alternatives. The phrase "by text" implies a text-based selection scenario, but it does not state when to choose this tool over query_selector, compare_selectors, or other sibling tools. No exclusions or prerequisites are provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

generate_regexC

Generate a regular expression from sample strings using grex.

ParametersJSON Schema
NameRequiredDescriptionDefault
test_casesYes
verbose_modeNo
convert_wordsNo
convert_digitsNo
anchors_enabledNo
capturing_groupsNo
case_insensitiveNo
escape_non_asciiNo
convert_repetitionsNo
minimum_repetitionsNo
use_surrogate_pairsNo
minimum_substring_lengthNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
patternYes
test_casesNo
verbose_modeNo
convert_wordsNo
convert_digitsNo
anchors_enabledNo
capturing_groupsNo
case_insensitiveNo
escape_non_asciiNo
convert_repetitionsNo
minimum_repetitionsNo
use_surrogate_pairsNo
minimum_substring_lengthNo

TDQS

C2.8/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden of behavioral disclosure. It only mentions 'using grex,' which hints at an underlying library, but does not explain output behavior, limitations, error conditions, or side effects. The description fails to add meaningful behavioral context beyond the basic action.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, front-loaded sentence that efficiently states the core action. However, given the tool's parameter complexity (12 parameters), it is likely too terse to be fully useful, but structurally it is concise and earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has 12 parameters and an output schema, yet the description provides only the basic action. There is no usage context, parameter meaning, or expected behavior. While the output schema may clarify return values, the description is inadequate for an agent to select and configure the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, and the description names none of the 12 parameters. It only implies the 'test_cases' input via 'sample strings' but leaves all boolean options (e.g., convert_words, anchors_enabled) undefined. The description does not compensate for the schema's lack of documentation at all.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's function: 'Generate a regular expression from sample strings using grex.' It uses a specific verb ('Generate'), identifies the resource ('regular expression from sample strings'), and is distinct from sibling tools like HTML parsing and selector extraction, so there is no ambiguity about its purpose.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no guidance on when to use this tool versus alternatives. The description only states what the tool does, without any context about prerequisites, typical use cases, or exclusions. For a tool with many siblings, explicit usage guidance is missing.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

generate_spider_templateC

Generate a production starter spider that mirrors the crawl blueprint.

ParametersJSON Schema
NameRequiredDescriptionDefault
variantNoauto
blueprintYes
class_nameNoGeneratedSpider

Output Schema

ParametersJSON Schema
NameRequiredDescription
codeYes
class_nameYes
spider_nameYes
template_variantYes

TDQS

C2.7/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It only says 'generate' but does not explain what output is produced, whether files are written, if there are side effects, or any other behavioral details. This is a significant gap for a code-generation tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence of 11 words, with no fluff or repetition. It is front-loaded and efficient, though perhaps too terse for such a complex tool. It earns its place but could be expanded without losing conciseness.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the complexity of the input schema (large nested object, 3 params) and the presence of an output schema, the description is severely underspecified. It does not explain what a 'production starter spider' means, what the output format is, or how this relates to running a crawl blueprint. The description is inadequate for an agent to use this tool correctly without further context.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, and the description does not explain the top-level parameters 'variant' or 'class_name.' It only references 'the crawl blueprint,' which maps to the 'blueprint' parameter, but even that is not elaborated. Nested blueprint properties are well described in the schema, but the description adds minimal semantic value beyond the name.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a clear verb ('Generate') and resource ('a production starter spider that mirrors the crawl blueprint'). It indicates code generation from a blueprint, but does not explicitly differentiate from sibling tools like run_crawl_blueprint or validate_spider_code, though the word 'starter spider' hints at template creation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is given on when to use this tool versus alternatives such as run_crawl_blueprint or validate_spider_code. The description only implies usage through the phrase 'mirrors the crawl blueprint,' but it lacks explicit context, prerequisites, or exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

inspect_documentC

Build a high-level summary for stored HTML or an inline HTML snippet.

ParametersJSON Schema
NameRequiredDescriptionDefault
htmlNo
source_urlNo
max_size_bytesNo
document_handleNo
truncate_on_limitNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
labelNo
titleNo
handleNo
statusNo
form_countYes
html_charsYes
link_countYes
source_urlNo
text_charsYes
fetched_viaNo
image_countYes
text_previewYes
heading_previewNo

TDQS

C2.7/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It only states 'Build a high-level summary' without mentioning any side effects, error cases, handling of size limits (max_size_bytes), or truncation behavior. The absence of any caveats or return behavior details leaves the agent under-informed for a tool with multiple configurable parameters.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence that is front-loaded and easy to parse. It avoids unnecessary words and gets straight to the point. However, it may be too sparse given the tool's complexity, but as a concise statement it is effective.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has 5 optional parameters and an output schema, but the description provides minimal context. It does not explain what 'summary' entails, how the tool behaves when both html and document_handle are provided, what happens on size limits, or the relationship to sibling parse tools. This is incomplete for a tool that can operate in two different modes with configurable limits.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It maps 'stored HTML' to document_handle and 'inline HTML snippet' to html, adding some meaning for those two parameters. However, it ignores max_size_bytes and truncate_on_limit, which are completely unexplained. The description does not fully compensate for the schema's lack of parameter documentation.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses the specific verb 'Build' with a clear resource ('high-level summary') and scope ('stored HTML or an inline HTML snippet'). It distinguishes itself from sibling tools like parse_html_document or query_selector by focusing on summarization rather than extraction or parsing, though it doesn't explicitly name them.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for high-level summarization of HTML content but provides no explicit when-to-use or when-not-to-use guidance. It does not mention alternatives or edge cases (e.g., when to choose parse_html_document instead). It only offers a brief context of the two input modes, which is insufficient for tool selection.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_documentsA

List cached documents that can be reused by handle in later tool calls.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the burden. It discloses that documents are cached and reusable by handle, which is useful behavioral context. However, it does not state side effects (though listing implies read-only), return format details, or pagination/filtering behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, clear sentence that front-loads the verb and resource. Every word earns its place, with no redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the simplicity (no parameters, output schema exists), the description is mostly complete. It explains the core purpose and value proposition. It could mention that it lists all cached documents or how to use the handles, but the output schema likely covers return details.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has zero parameters, so there is no schema to explain. The description naturally adds no parameter-specific information, and the baseline of 4 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb ('List') and resource ('cached documents') with a clear purpose ('reused by handle in later tool calls'). It clearly distinguishes from sibling tools that create, delete, or inspect documents.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage (to see what documents are cached and get handles) but does not explicitly state when to use this tool versus alternatives like inspect_document or delete_document. There is no exclusions or alternative tool names mentioned.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

parse_html_documentC

Parse a full HTML document into a structured node tree using scraper_rs.parse_document.

ParametersJSON Schema
NameRequiredDescriptionDefault
htmlNo
source_urlNo
max_size_bytesNo
document_handleNo
truncate_on_limitNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
rootYes
parse_modeYes
source_urlNo
max_size_bytesNo
document_handleNo
truncate_on_limitYes

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It only states the internal implementation (scraper_rs.parse_document) without mentioning size limits, truncation behavior, error handling, or what the structured node tree looks like. The parameters suggest there are significant behavioral considerations (e.g., max_size_bytes, truncate_on_limit) that are entirely undisclosed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence that is concise and front-loaded, with no redundant wording. However, given the tool's 5 parameters and complexity, it could be argued that brevity overshadows usefulness, but that is more a completeness concern than a conciseness one.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness1/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has 5 parameters, an output schema, and several sibling tools, but the description only offers one sentence. It does not explain the input modes (inline html vs stored document handle), the meaning of the size limit and truncation flag, or how this relates to parse_html_fragment. This is severely incomplete for a tool of this complexity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, yet the description provides no information about any of the 5 parameters. It does not explain how to supply the HTML (html vs document_handle), the role of source_url, or the implications of max_size_bytes and truncate_on_limit. The description fails to compensate for the absent schema documentation.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly identifies the action ('Parse'), the resource ('a full HTML document'), and the outcome ('into a structured node tree'). It also distinguishes this from the sibling tool parse_html_fragment by specifying 'full' document, making the purpose unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage when you need to parse a complete HTML document, but it does not explicitly state when to use this tool instead of alternatives like parse_html_fragment or query_selector. No exclusions or prerequisites are mentioned, so guidance is only implicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

parse_html_fragmentC

Parse an HTML fragment into a structured node tree using scraper_rs.parse_fragment.

ParametersJSON Schema
NameRequiredDescriptionDefault
htmlNo
source_urlNo
max_size_bytesNo
document_handleNo
truncate_on_limitNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
rootYes
parse_modeYes
source_urlNo
max_size_bytesNo
document_handleNo
truncate_on_limitYes

TDQS

C2.6/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the burden of disclosing behavioral traits. It mentions 'parse' which implies read-only, but does not state side-effect freedom, performance implications, or error behavior. The added implementation detail 'using scraper_rs.parse_fragment' does not aid transparency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single concise sentence, which is structurally clear. However, it includes the unnecessary implementation detail 'using scraper_rs.parse_fragment', which adds no agent-relevant value and wastes words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness1/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given 5 undocumented parameters, no annotations, and an output schema, the description is far too sparse. It does not explain parameter interplay, size limits, truncation behavior, or how document_handle differs from direct html input, making it inadequate for reliable tool invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, and the description offers no explanation of the five parameters. It only hints at 'html' via 'HTML fragment', but source_url, max_size_bytes, document_handle, and truncate_on_limit remain completely unexplained, leaving the agent unable to infer their meaning or defaults.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific action—parsing an HTML fragment into a structured node tree—which clearly identifies the resource and operation. It does not explicitly differentiate from siblings like parse_html_document, but the term 'fragment' offers some distinction.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies the tool is for parsing HTML fragments but provides no explicit guidance on when to use this versus alternatives such as parse_html_document or query_selector. No exclusions or alternative references are given, leaving usage context only vaguely implied.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

prettify_documentC

Return prettified HTML for visual inspection.

ParametersJSON Schema
NameRequiredDescriptionDefault
htmlNo
max_size_bytesNo
document_handleNo
max_output_charsNo
truncate_on_limitNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
source_urlNo
output_charsYes
document_handleNo
prettified_htmlYes
truncated_outputYes

TDQS

C2.4/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description must disclose behavioral traits; it does not mention truncation behavior, size limits (max_size_bytes, max_output_chars), truncate_on_limit logic, or the dual-input mechanism (html or document_handle). The only behavioral hint is 'for visual inspection' implying formatting, which is minimal.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence with no filler, but it omits essential parameter and behavior context, making it under-specified rather than appropriately concise.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 5 parameters, size limits, and truncation options, the one-sentence description is incomplete. An output schema exists but does not compensate for missing guidance on how to choose input source or what limits apply.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, and the description provides no parameter explanations. All 5 optional parameters (html, max_size_bytes, document_handle, max_output_chars, truncate_on_limit) are left entirely undocumented, forcing the agent to guess their roles.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool returns prettified HTML for visual inspection, identifying its core function. It distinguishes from sibling parsing/inspecting tools by emphasizing 'prettified' formatting, though it doesn't explicitly mention input sources (html string or document_handle).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided on when to use this tool versus alternatives like inspect_document or parse_html_document, nor are there any exclusions or context about suitability. The description entirely omits usage context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

query_selectorC

Run a CSS or XPath query with scraper_rs and return structured match previews.

ParametersJSON Schema
NameRequiredDescriptionDefault
htmlNo
modeNocss
limitNo
queryYes
html_charsNo
source_urlNo
text_charsNo
include_htmlNo
max_size_bytesNo
document_handleNo
truncate_on_limitNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
modeYes
queryYes
matchesNo
source_urlNo
total_matchesYes
document_handleNo
omitted_matchesYes
returned_matchesYes

TDQS

C2.8/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden of behavioral disclosure. It mentions 'structured match previews' but does not reveal truncation behavior (html_chars, text_chars, limit), size limits, or how document_handle/source_url/html are prioritized. These are significant behavioral traits not disclosed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single concise sentence with no unnecessary words. However, it omits critical details, so while concise, it sacrifices informativeness. It is not bloated, but it is under-specified for the tool's complexity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool has 11 parameters and an output schema, the description is highly incomplete. It does not explain the relationship between input sources (html, source_url, document_handle), the meaning of preview limits, or how to choose between CSS and XPath modes. The output schema exists but the input semantics are entirely undocumented.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%; the description adds no meaning beyond the raw schema. Even the mention of 'CSS or XPath' weakly maps to the mode parameter, but none of the 11 parameters are explained, leaving the agent to guess semantics for html, document_handle, truncate_on_limit, max_size_bytes, etc.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's function: running a CSS or XPath query with scraper_rs and returning structured match previews. It specifies the verb, resource, and output type, and distinguishes it from CDP-related siblings like query_selector_cdp by noting the scraper_rs implementation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided on when to use this tool versus alternatives such as query_selector_cdp, compare_selectors, or analyze_css_selectors. The description does not mention use cases, exclusions, or prerequisites.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

query_selector_cdpA

Fetch a rendered page through CDP, then run a CSS or XPath query on the live DOM snapshot.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
modeNocss
labelNo
limitNo
queryYes
headersNo
html_charsNo
text_charsNo
ws_endpointNows://127.0.0.1:9222
include_htmlNo
max_size_bytesNo
store_documentNo
timeout_secondsNo
truncate_on_limitNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
modeYes
queryYes
matchesNo
source_urlNo
total_matchesYes
document_handleNo
omitted_matchesYes
returned_matchesYes

TDQS

A3.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It accurately describes the two main phases (fetch via CDP, then query the snapshot) and the 'live DOM snapshot' concept. However, it does not mention potential side effects like storing the document (store_document defaults to true), resource limits, or network behavior, leaving gaps.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single, well-structured sentence that front-loads the tool's purpose. Every word earns its place; no filler or redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite having an output schema, the tool has 14 parameters, no annotations, and a very terse description. It omits details about the fetch process, storage side effects, pagination, and parameter semantics. For such a complex tool, the description is too minimal to fully guide an agent.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, and the description only indirectly clarifies the 'query' and 'mode' parameters by mentioning 'CSS or XPath query'. It provides no guidance for the other 12 parameters, including required 'url' and important ones like 'limit', 'headers', or 'store_document'. The description fails to compensate for the lack of schema descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ('Fetch', 'run') and clearly identifies the resource ('a rendered page through CDP', 'live DOM snapshot'). It distinguishes the tool from siblings by highlighting the rendered/CDP aspect, which differentiates it from static query_selector and pure fetch tools like silkworm_fetch_cdp.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies the intended usage: use this tool when a page needs to be rendered (via CDP) before querying the live DOM. This provides clear context for when to choose it over alternatives, though it does not explicitly name sibling tools or provide exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

run_crawl_blueprintB

Run a configurable silkworm spider without writing code, useful for validating a scraping plan.

ParametersJSON Schema
NameRequiredDescriptionDefault
variantNoauto
blueprintYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
itemsNo
max_itemsYes
spider_nameYes
max_requestsYes
emitted_itemsYes
execution_variantYes
scheduled_requestsYes

TDQS

B3.3/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must disclose behavioral traits itself. It only states that it runs a spider, omitting potential side effects such as making many network requests, optionally writing JSONL output, following pagination/detail links, or resource usage. This is a significant gap for such a stateful and potentially long-running operation.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence that front-loads the main action ('Run a configurable silkworm spider') and tacks on a purpose clause. There is no wasted wording, and it is appropriately sized for a tool with a self-explanatory name.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Although a rich output schema exists (not shown here), the description is far too sparse for a tool with a complex nested blueprint configuration. It fails to mention what the tool returns (e.g., extracted items, status), whether it writes files (output_jsonl_path), or how it handles pagination/follow links. The description is minimal for a tool of this complexity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% for the top-level parameters (variant and blueprint), and the description adds no explanation of these parameters. While the nested blueprint schema includes internal field descriptions, the tool description itself provides zero parameter guidance, leaving the agent to infer usage entirely from the raw schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ('Run') and resource ('configurable silkworm spider'), and adds a clear use case ('validating a scraping plan'). It implicitly distinguishes from siblings like generate_spider_template by emphasizing 'without writing code' and from lower-level fetch/parse tools by focusing on the full spider execution.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The phrase 'useful for validating a scraping plan' provides a clear context for when to use this tool. It does not explicitly name alternatives or state exclusions, but the contrast with code generation is implied through 'without writing code'. This is strong but not fully explicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

server_statusB

Return runtime status, cache metrics, and optional CDP readiness information.

ParametersJSON Schema
NameRequiredDescriptionDefault
probe_cdpNo
include_documentsNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
healthNo
documentsNo
server_nameYes
configurationYes
document_storeYes
server_versionYes
uptime_secondsYes
process_started_atYes

TDQS

B3.4/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It indicates a read-only operation ('Return') but does not disclose potential side effects of the optional flags (e.g., probe_cdp might launch a CDP connection) or any resource implications.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is one concise sentence, front-loaded with the main action, and contains no redundant words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has a simple purpose and an output schema, so return values needn't be explained. However, the optional parameters are left underdocumented, and the overall description is minimal for a status tool that may have side effects.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, and the description only hints at probe_cdp via 'optional CDP readiness information.' The include_documents parameter is not mentioned at all, leaving its purpose ambiguous.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb 'Return' and names distinct resources: runtime status, cache metrics, and CDP readiness. This clearly differentiates it from sibling tools focused on document processing and CSS selectors.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No explicit when-to-use or alternative guidance is provided. However, the purpose as a status/health checker is implied clearly, and sibling context suggests it is for server-level checks rather than document operations.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

silkworm_fetchB

Fetch a page through silkworm's HttpClient and optionally cache the HTML for later selector work.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
labelNo
methodNoGET
paramsNo
headersNo
body_jsonNo
body_textNo
emulationNoFirefox139
keep_aliveNo
store_documentNo
timeout_secondsNo
body_preview_charsNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
urlYes
viaYes
methodYes
statusNo
headersNo
is_htmlYes
summaryNo
emulationNo
final_urlYes
body_charsYes
body_previewYes
document_handleNo

TDQS

B3.3/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions optional caching, but does not detail side effects, error handling, authentication requirements, or the nature of the stored HTML. This is insufficient for a network-fetching tool with potential storage implications.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, front-loaded sentence that wastes no words. It conveys the primary action and a key optional behavior without redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Although an output schema exists (so return values need not be explained), the tool has 12 parameters, no annotations, and no schema descriptions. The one-sentence description is far too sparse to make the tool safely and correctly invokable for non-trivial HTTP requests.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, and the description fails to compensate. It only alludes to caching (possibly linking to the store_document parameter) but does not explain the remaining 11 parameters such as method, headers, body, emulation, or timeout settings.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool fetches a page via silkworm's HttpClient and optionally caches HTML. This distinguishes it from silkworm_fetch_cdp and other sibling tools, giving the agent a clear understanding of the core function.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The phrase 'through silkworm's HttpClient' implies a lightweight HTTP fetch rather than a CDP-driven browser session, and 'for later selector work' hints at a use case. However, it does not explicitly state when to prefer this over the CDP variant or exclude JavaScript-heavy pages.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

silkworm_fetch_cdpB

Fetch rendered HTML through silkworm's CDP client for JavaScript-heavy pages.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
labelNo
headersNo
ws_endpointNows://127.0.0.1:9222
store_documentNo
timeout_secondsNo
body_preview_charsNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
urlYes
viaYes
methodYes
statusNo
headersNo
is_htmlYes
summaryNo
emulationNo
final_urlYes
body_charsYes
body_previewYes
document_handleNo

TDQS

B3.3/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must fully disclose behavior. It mentions CDP-based rendering but omits significant side effects such as the store_document parameter defaulting to true (which implies storing the fetched HTML), the reliance on ws_endpoint, and timeout behavior. This is a substantial gap for a tool that may persist data.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single 13-word sentence that is front-loaded with the action and key qualifier. Every word contributes to the core purpose, with no fluff or repetition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool has 7 parameters, no annotations, and a large sibling set, the description is too minimal. It does not address storage side effects, connection details, or usage nuance, leaving the agent to infer critical operational context from the schema alone.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, and the description provides no explanation for any of the 7 parameters (url, label, headers, ws_endpoint, store_document, timeout_seconds, body_preview_chars). The description does not compensate for the schema's lack of semantic detail.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ('Fetch') and resource ('rendered HTML through silkworm's CDP client'), clearly indicating it is for JavaScript-heavy pages. This distinguishes it from sibling tools like silkworm_fetch (likely plain fetch) and query_selector_cdp/extract_structured_data_cdp (which operate on CDP content but have different actions).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The phrase 'for JavaScript-heavy pages' provides clear context on when to use this tool over a simpler fetch. However, it does not explicitly name alternative tools or state when not to use it, stopping short of full guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

store_html_documentC

Store raw HTML in the server's in-memory document cache and return a scraper_rs summary.

ParametersJSON Schema
NameRequiredDescriptionDefault
htmlYes
labelNo
source_urlNo
max_size_bytesNo
truncate_on_limitNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
labelNo
titleNo
handleNo
statusNo
form_countYes
html_charsYes
link_countYes
source_urlNo
text_charsYes
fetched_viaNo
image_countYes
text_previewYes
heading_previewNo

TDQS

C2.7/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It discloses that storage is in-memory (volatile) and that a summary is returned, but omits crucial behaviors such as how size limits are enforced (max_size_bytes, truncate_on_limit), whether storing replaces existing documents, or what the summary actually contains.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, direct sentence with no fluff and front-loads the main action. It is efficient but omits important detail, which is a trade-off; however, for conciseness alone it scores well.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With 5 parameters, no annotations, and an output schema that is not explained, the description is incomplete. It does not mention how stored documents are referenced later, what the summary includes, or how the cache behaves under limits. Sibling tools imply a workflow, but this description does not connect to it.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It only implicitly references the 'html' parameter ('Store raw HTML') and says nothing about label, source_url, max_size_bytes, or truncate_on_limit. The description adds minimal value over the raw schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states the action (store) and resource (raw HTML) with a specific destination (server's in-memory document cache) and result (scraper_rs summary). It distinguishes itself from sibling tools like list_documents, delete_document, and parse_html_document, though the terminology 'scraper_rs summary' is somewhat opaque.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No explicit guidance on when to use this tool versus alternatives. The description implies a use case (storing HTML for later processing), but does not state exclusions, prerequisites, or when to prefer a sibling tool like parse_html_document or silkworm_fetch.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

validate_spider_codeB

Statically validate generated spider code for syntax and common silkworm/CDP wiring.

ParametersJSON Schema
NameRequiredDescriptionDefault
codeYes
expected_class_nameNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
issuesNo
syntax_okYes
spider_classesNo
uses_cdp_clientNo
uses_run_spiderNo
imports_silkwormNo
uses_run_spider_uvloopNo

TDQS

B3.1/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden. It discloses static validation (non-executing) and mentions specific validation checks, but does not state error handling, return format, or explicitly confirm it is read-only beyond what 'statically' implies.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single, focused sentence with no filler; every word adds meaning, and the description is front-loaded with the core action.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description is too sparse for a tool with undocumented parameters and lacking annotation context. It doesn't explain expected_class_name, desired outcomes, or how validation results are presented, despite having an output schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% and the description does not mention the 'code' or 'expected_class_name' parameters, leaving their semantics entirely to the raw schema field names. No compensation is made for the lack of schema descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb 'validate' with a clear resource 'generated spider code' and scopes validation to 'syntax and common silkworm/CDP wiring', making it distinct from sibling tools like generate_spider_template or run_crawl_blueprint.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No explicit when-to-use or alternative guidance is provided. The description implies it should be used on generated code but does not mention related tools like generate_spider_template or silkworm_fetch, leaving the agent to infer context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. 22 tool updatesv0.2.6
    • First observedanalyze_css_selectors
    • First observedclear_documents
    • First observedcompare_selectors
    • First observeddelete_document
    • First observedextract_links
    • First observedextract_structured_data_cdp
    • First observedfind_selectors_by_text
    • First observedgenerate_regex
    • First observedgenerate_spider_template
    • First observedinspect_document
    • First observedlist_documents
    • First observedparse_html_document
    • First observedparse_html_fragment
    • First observedprettify_document
    • First observedquery_selector
    • First observedquery_selector_cdp
    • First observedrun_crawl_blueprint
    • First observedserver_status
    • First observedsilkworm_fetch
    • First observedsilkworm_fetch_cdp
    • First observedstore_html_document
    • First observedvalidate_spider_code

TDQS

B3.1/5.0
Disambiguation4/5

Most tools have clear, distinct purposes, and the _cdp suffix separates rendered from static operations. However, the cluster of selector-related tools (query_selector, compare_selectors, analyze_css_selectors, find_selectors_by_text) and overlapping inspection tools (inspect_document, prettify_document, parse_html_document) could cause some misselection without careful reading.

Naming Consistency4/5

Tool names largely follow a verb_noun snake_case pattern (e.g., store_html_document, list_documents, parse_html_fragment). Minor inconsistencies like server_status (noun-first) and the silkworm_ prefix on some fetch tools break the strict pattern but remain readable and predictable.

Tool Count3/5

At 22 tools, this is on the heavy side but justified by the broad scope of a web scraping framework covering fetching, caching, parsing, selector analysis, and code generation. The count is manageable, but agents may need to scan many options before choosing.

Completeness4/5

The toolset covers the full scraping lifecycle from fetching (normal and CDP) to caching, parsing, selector analysis, structured extraction, and spider validation/generation. Minor gaps exist, such as no explicit export/save tool for extracted data and no update operation for cached documents, but these are workable.

Maintenance

ActivityInactive
ResponsivenessUnresponsive

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

  • A
    license
    A
    quality
    D
    maintenance
    A headless web-scraping MCP server built on Scrapy, providing tools for polite fetching, CSS/XPath extraction, link/table extraction, sitemap and robots.txt reading, and bounded asynchronous crawls.
    10
    MIT
  • F
    license
    Not graded
    quality
    D
    maintenance
    MCP server for crawling and extracting data from web pages using Selenium with CSS/XPath selectors. Supports 17 tools including smart extraction, pagination, infinite scroll, and screenshots.
    3
    -
  • A
    license
    Not graded
    quality
    B
    maintenance
    A comprehensive web scraping MCP server with 26 tools for fetching, parsing, extracting, and assisting with web content, returning Markdown-formatted results.
    MIT

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/BitingSnakes/silkworm-mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server