Skip to main content
Glama
eitan3
by eitan3

scrapy-mcp

A headless web-scraping MCP server built on Scrapy. It exposes Scrapy's scraping primitives — polite fetching, CSS/XPath extraction, link and table extraction, sitemap and robots.txt reading, and bounded asynchronous crawls — as MCP tools an agent can call over stdio.

  • Headless, no rendering. Pages are fetched and parsed as HTML; no browser, no JavaScript execution. This keeps the footprint tiny — it runs comfortably on weak machines.

  • Reactor-safe. Every operation runs in a short-lived Scrapy subprocess, so Twisted's reactor never lives inside the asyncio MCP server (no ReactorNotRestartable), and memory is reclaimed after each call.

  • Polite by default. Obeys robots.txt, throttles with AutoThrottle, and enforces hard page/depth caps so a crawl can't run away.

Install / run

Run straight from PyPI with uv — no install step:

uvx scrapy-mcp

Or install it:

uv pip install scrapy-mcp
scrapy-mcp

The server speaks MCP over stdio. Point any MCP client at it. For Claude Desktop, add to claude_desktop_config.json:

{
  "mcpServers": {
    "scrapy": {
      "command": "uvx",
      "args": ["scrapy-mcp"]
    }
  }
}

Related MCP server: silkworm-mcp

Tools

Tool

What it does

fetch_page(url, format, max_bytes, obey_robots)

Fetch one page as markdown (default), text, or html.

extract(url, selectors, obey_robots)

Pull structured fields with CSS/XPath selectors.

extract_tables(url, max_tables, obey_robots)

Extract every HTML <table> as {headers, rows}.

extract_links(url, same_domain, pattern, limit, obey_robots)

List de-duplicated links on a page.

get_sitemap(url, limit, obey_robots)

Read a sitemap (gzip + sitemap-index aware).

check_robots(url, user_agent)

Is a URL crawlable? Returns the crawl-delay and sitemaps.

start_crawl(start_url, allow_patterns, deny_patterns, max_pages, max_depth, same_domain, selectors, ...)

Start a bounded BFS crawl; returns a job_id.

crawl_status(job_id)

State + pages scraped for a crawl.

crawl_results(job_id, cursor, limit)

Page through a crawl's scraped items.

cancel_crawl(job_id)

Stop a running crawl; keep results so far.

Selector format (extract / start_crawl)

selectors maps an output field to a selector. Each value is either a CSS string (first match) or an object for more control:

{
  "title": "h1::text",
  "price": "span.price::text",
  "all_links": {"css": "a::attr(href)", "all": true},
  "first_heading": {"xpath": "//h1/text()"}
}

"all": true returns every match as a list; otherwise the first match is returned.

Crawls are asynchronous

start_crawl returns immediately with a job_id. The crawl runs as a detached worker that streams results to disk, so it survives a server restart. Poll crawl_status(job_id), then read items with crawl_results(job_id) (safe to call mid-crawl for partial results). Jobs are stored under the system temp dir and reclaimed after 7 days (configurable).

Configuration

All settings are optional environment variables (sensible, polite defaults tuned for a weak host). They're how you tune a uvx scrapy-mcp deployment.

Variable

Default

Meaning

SCRAPY_MCP_USER_AGENT

scrapy-mcp/<version> …

User-Agent header.

SCRAPY_MCP_OBEY_ROBOTS

true

Obey robots.txt.

SCRAPY_MCP_DOWNLOAD_DELAY

0.5

Seconds between requests to a host.

SCRAPY_MCP_CONCURRENT_REQUESTS

8

Global concurrency.

SCRAPY_MCP_CONCURRENT_REQUESTS_PER_DOMAIN

4

Per-host concurrency.

SCRAPY_MCP_DOWNLOAD_TIMEOUT

30

Per-request timeout (s).

SCRAPY_MCP_RETRY_TIMES

2

Retries on transient failures.

SCRAPY_MCP_AUTOTHROTTLE

true

Adapt delay to server latency.

SCRAPY_MCP_MAX_BYTES

50000

Max characters returned per page (then truncated).

SCRAPY_MCP_REQUEST_TIMEOUT

60

Wall-clock cap for a blocking single fetch (s).

SCRAPY_MCP_DEFAULT_MAX_PAGES / _MAX_PAGES_CAP

50 / 1000

Crawl page default / hard cap.

SCRAPY_MCP_DEFAULT_MAX_DEPTH / _MAX_DEPTH_CAP

2 / 10

Crawl depth default / hard cap.

SCRAPY_MCP_JOB_DIR

<tmp>/scrapy_mcp_jobs

Where crawl jobs are stored.

SCRAPY_MCP_JOB_TTL_DAYS

7

Delete crawl jobs older than this (0 disables).

SCRAPY_MCP_LOG_LEVEL

ERROR

Scrapy log level (to stderr).

Development

uv venv
uv pip install -e ".[dev]"
uv run pytest          # unit tests (no network)
uv build               # build wheel + sdist into dist/

License

MIT © Eitan Hadar

Available Tools

10 tools
cancel_crawlA

Stop a running crawl by killing its worker process. Results so far are kept.

ParametersJSON Schema
NameRequiredDescriptionDefault
job_idYes

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It discloses the destructive act of killing the worker process and the side effect that results are kept. This goes beyond a vague 'cancel' and gives the agent meaningful expectations about consequences.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two short sentences, front-loaded with the core action and immediately followed by the most important side effect. No filler or redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity (one parameter, no output schema, no annotations), the description covers the key aspects: what it stops, how it stops it, and what persists. It doesn't mention whether cancellation is reversible or what is returned, but those are not critical for this simple operation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has one required parameter (job_id) but zero description coverage. The description does not explicitly define job_id, though the tool name and context make it inferable as the crawl's ID. It would benefit from stating 'the ID of the crawl to cancel' or referencing start_crawl.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action (stop a running crawl), the mechanism (killing its worker process), and the outcome (results kept). This is a specific verb+resource+scope statement that distinguishes cancel_crawl from siblings like start_crawl and crawl_status.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies the use case: when a crawl is running and must be stopped. It does not explicitly contrast with alternatives, but the purpose is so singular that the context is clear. It would benefit from mentioning 'use crawl_status to check if a crawl is running', but the baseline is solid.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

check_robotsA

Check the site's robots.txt: is url crawlable, and what crawl-delay applies?

user_agent: the agent to evaluate rules for (defaults to the server's user agent). Returns {allowed, has_robots, crawl_delay, request_rate, sitemaps, robots_url}. Sites with no robots.txt are reported as allowed (has_robots=false).

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
user_agentNo

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description discloses the return fields and the special behavior for missing robots.txt (reported as allowed with has_robots=false). Since there are no annotations, this carries the full burden and effectively communicates the tool's read-only nature and output.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise: one line for purpose, one for parameter, and one for return values. It is front-loaded with the main purpose and contains no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple read-only tool with no output schema, the description fully covers: purpose, parameter semantics, return object fields, and an edge case (missing robots.txt). It is complete and self-contained.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description compensates by explaining user_agent's default and role. The url parameter is implicitly described as the site to check but lacks explicit format details; overall it adds meaning beyond the raw schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool checks a site's robots.txt to determine if a URL is crawlable and what crawl-delay applies. This is a specific verb+resource that distinguishes it from siblings like fetch_page or get_sitemap.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The tool's purpose implies it should be used before crawling to check compliance, but there is no explicit 'when to use' or 'when not to use' guidance. Siblings like start_crawl exist but no alternative is mentioned, leaving usage inferred rather than stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

crawl_resultsA

Page through a crawl's scraped items.

cursor: line offset to start from (use the returned next_cursor to continue). limit: items per page (1-1000). Safe to call while the crawl is still running -- you get whatever has been written so far.

ParametersJSON Schema
NameRequiredDescriptionDefault
limitNo
cursorNo
job_idYes

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden. It discloses key behavioral traits: cursor is a line offset, next_cursor is used to continue, limit range (1-1000), and that results reflect whatever has been written so far during an active crawl. This adds valuable context beyond the schema and clarifies the tool's eventual-consistency behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is brief and front-loads the main purpose. It then uses a clear list-style block for parameter guidance. Every sentence adds value, with no filler or repetition. The formatting is clean and readable.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple pagination tool with three parameters and no output schema, the description covers the primary purpose, parameter semantics, and a key behavioral nuance. It hints at the response shape via 'next_cursor' but does not explicitly describe the structure of items. Overall, this is complete enough for effective tool invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has 0% description coverage, so the description compensates well by explaining cursor and limit in detail. It does not explicitly describe job_id, but the tool name and 'a crawl's' context make it obvious that job_id identifies the crawl. This is sufficient for a required identifier parameter.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb "Page through" and clearly identifies the resource as "a crawl's scraped items." This distinguishes it from sibling tools like start_crawl or crawl_status, making its function unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context on when to use the tool (to retrieve crawl results in pages) and explains the pagination mechanism via cursor and limit. It also notes it can be safely used while the crawl is running, which implies appropriate usage during execution. It does not explicitly exclude alternatives, but the context is sufficient.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

crawl_statusA

Report a crawl's state and progress.

state is one of: starting, running, finished, failed, cancelled, interrupted. ('interrupted' means the worker process died without finishing -- e.g. the machine or server restarted; any results gathered so far are still readable.)

ParametersJSON Schema
NameRequiredDescriptionDefault
job_idYes

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It discloses possible state values and explains the 'interrupted' state in detail, including that results remain readable. This adds meaningful behavioral insight beyond a basic status check, though error behavior is not covered.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact and front-loaded with the core purpose, followed by a necessary list of state values and a clarifying note. No redundant wording.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool is simple, but without an output schema the description leaves the return format ambiguous ('state and progress' but no structure). It also does not address invalid job IDs, though the state enumeration is a helpful addition.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has one required parameter, job_id, with no description beyond the name. The tool description does not mention job_id or its provenance (e.g., returned by start_crawl), leaving parameter semantics under-specified despite 0% schema coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states 'Report a crawl's state and progress.' with a specific verb and resource. This distinguishes it from sibling tools like start_crawl and crawl_results, which focus on initiating and fetching results.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides context about the meaning of states, especially 'interrupted,' but does not explicitly state when to use this tool versus alternatives like crawl_results or cancel_crawl. Usage is implied but not explicitly articulated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

extractA

Fetch a page and pull structured fields out of it with CSS or XPath selectors.

selectors maps an output field name to a selector. Each value is either:

  • a CSS string, e.g. "h1::text" or "a.product::attr(href)" (returns the first match), or

  • an object: {"css": "...", "all": true} / {"xpath": "//h1/text()", "all": true} ("all": true returns every match as a list; default returns the first match).

Example: {"title": "h1::text", "prices": {"css": ".price::text", "all": true}} Returns {"data": {field: value | [values]}}.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
selectorsYes
obey_robotsNo

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the transparency burden. It explains selector syntax, first-match vs all behavior, and return format. However, it does not disclose the meaning of the obey_robots parameter or potential error handling, leaving some behavioral aspects untold.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-organized and front-loaded with the main purpose, followed by a compact specification of selector formats and an example. Every section adds necessary value, and formatting uses clear bullet points and code blocks.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description is largely complete for a complex extraction tool, including return structure and selector behavior. It could be more complete by addressing edge cases, obey_robots semantics, or error handling, but these are secondary to the core extraction logic.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It does so thoroughly for the selectors parameter, detailing CSS string and object forms with examples. The url parameter is implied by 'Fetch a page.' The obey_robots parameter is not explained, but its name and default null convey partial meaning.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's action: 'Fetch a page and pull structured fields out of it with CSS or XPath selectors.' It specifies the resource (page) and the mechanism (CSS/XPath), distinguishing it from siblings like extract_tables and extract_links which are more specialized.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context on how to use the tool by explaining the selectors mapping and examples. However, it does not explicitly mention when not to use it or alternatives such as extract_tables or fetch_page, so it falls short of full guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

extract_tablesA

Fetch a page and extract every HTML as {headers, rows}.

max_tables: cap on how many tables to return (1-100). Each table is {"headers": [...], "rows": [[cell, ...], ...], "n_rows": int, "n_cols": int}.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
max_tablesNo
obey_robotsNo

TDQS

A3.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the burden. It discloses the max_tables cap and specifies the return structure, but omits details like network side effects, robots.txt handling, error behavior, or whether JavaScript is executed. It is not misleading but lacks depth.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is three sentences, front-loaded with the main purpose, and each sentence adds value. The return format is compactly specified, with no wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no output schema, the description does provide the return format, which is helpful. However, it lacks guidance on usage context, exclude cases, and the third param (obey_robots), making it only partially complete for a moderate-complexity tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 0% schema description coverage, the description must compensate. It defines max_tables well (cap 1-100) and clarifies url implicitly via 'Fetch a page', but provides no meaning for obey_robots, which remains ambiguous. Partial compensation for a 3-param tool.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool fetches a page and extracts HTML <table> elements, with a specific output format. This distinguishes it from siblings like extract_links or fetch_page by focusing on tables as the resource.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for table extraction but does not explicitly mention when to use this tool over alternatives like fetch_page or extract. No exclusions or alternative tool names are provided, leaving usage guidance implicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

fetch_pageA

Fetch a single web page and return its content (no JavaScript rendering).

format: 'markdown' (default, compact and readable), 'text' (visible text only), or 'html' (raw HTML). max_bytes: cap on returned characters; defaults to the server's SCRAPY_MCP_MAX_BYTES. Oversized content is truncated and truncated is set to true. obey_robots: override the server's robots.txt policy for this call only.

An HTTP error status (404, 500, ...) is returned in status, not raised.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
formatNomarkdown
max_bytesNo
obey_robotsNo

TDQS

A4.3/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description fully carries the transparency burden. It clearly discloses the lack of JavaScript rendering, truncation behavior with the 'truncated' flag, robots.txt override behavior, and that HTTP errors are returned in 'status' rather than raised. This is comprehensive and goes beyond basic expectations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is front-loaded with the core purpose in the first sentence, followed by a concise bulleted parameter list and a final sentence on error handling. Every sentence adds value with no fluff or redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description covers the main behavior (content fetching, format options, truncation, error handling) and notes the 'status' and 'truncated' fields. However, it does not provide a full return structure or mention any response headers/URL that might be included, leaving some ambiguity about the exact response shape.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema description coverage is 0%, so the description must fully compensate. It explains 'format', 'max_bytes', and 'obey_robots' in detail, including defaults and effects. The 'url' parameter is implicitly clear from the tool's purpose. This exceeds the schema's bare definitions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource: 'Fetch a single web page and return its content'. It is clear and concise, but it does not explicitly differentiate from sibling tools like 'extract' or 'extract_links'. The 'no JavaScript rendering' note hints at scope but is not a direct alternative comparison.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for retrieving raw page content, but it does not explicitly state when to use this tool over siblings, nor does it provide exclusions (e.g., 'for structured data use extract'). The parameter explanations offer context but not direct comparative guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_sitemapA

Fetch a sitemap and return the URLs it lists.

Handles gzip-compressed sitemaps and recurses one level of into its child sitemaps. url should point at a sitemap (e.g. https://site.com/sitemap.xml). limit: maximum URLs to return (1-50000). Each entry is {"loc": ..., "lastmod": ..., "changefreq": ..., "priority": ...}.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
limitNo
obey_robotsNo

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It usefully explains that gzip-compressed sitemaps are handled, that `<sitemapindex>` is recursed one level, and the exact output entry structure. This adds non-obvious context beyond what the schema would imply.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is appropriately concise, with the purpose stated in the first sentence and additional details organized clearly. Every sentence adds value, and there is no redundant or fluff content.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple fetch tool with three parameters and no output schema, the description covers the main behaviors (gzip handling, index recursion, output format) and gives a working example. It omits details about `obey_robots` behavior and error cases, but overall it is substantially complete for common usage.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It does explain `url` and `limit` meaningfully (limit with a 1-50000 range), but `obey_robots` is entirely undocumented. Since one of three parameters is unexplained, the compensation is incomplete.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's function: 'Fetch a sitemap and return the URLs it lists.' The verb 'Fetch' and resource 'sitemap' are specific, and the output (URLs) is identified. It distinguishes itself from sibling tools by focusing exclusively on sitemaps rather than general pages or link extraction.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides a clear prerequisite: '`url` should point at a sitemap (e.g. https://site.com/sitemap.xml).' It also mentions the limit parameter. However, it does not explicitly compare to alternative tools like fetch_page or extract_links, nor does it state when not to use this tool.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

start_crawlA

Start a bounded breadth-first crawl. Returns a job_id immediately (non-blocking).

The crawl follows in-scope links from start_url up to max_pages / max_depth.

  • allow_patterns / deny_patterns: regexes a link URL must match / must not match.

  • same_domain (default true): restrict the crawl to start_url's host.

  • selectors: same format as the extract tool; when given, each crawled page yields the extracted data. Otherwise each page yields a short text excerpt.

  • obey_robots, download_delay: per-crawl overrides of the server defaults.

Poll progress with crawl_status(job_id) and read items with crawl_results(job_id).

ParametersJSON Schema
NameRequiredDescriptionDefault
max_depthNo
max_pagesNo
selectorsNo
start_urlYes
obey_robotsNo
same_domainNo
deny_patternsNo
allow_patternsNo
download_delayNo

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of disclosing behavior. It covers the non-blocking nature, bounded limits, same_domain default, robots/delay overrides, and selector semantics. It does not mention cancellation via cancel_crawl or potential rate limits, but the disclosed details are substantial and contextually useful.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise, front-loaded with the most important behavioral fact (non-blocking, returns job_id), and uses bullet-like separations for parameters. Every sentence adds value, and the overall length is appropriate for the tool's complexity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a complex crawl tool with no output schema, the description covers the initiation, parameter semantics, and follow-up tools, which is largely complete. It omits details about cancellation or error handling, but the presence of sibling tools like crawl_status and cancel_crawl partially fills that gap. Given the high complexity, a 4 is appropriate.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate for all 9 parameters. It explains allow/deny_patterns as regex constraints, same_domain as host restriction, selectors as same format as extract tool, and obey_robots/download_delay as per-crawl overrides. This fully covers the non-obvious parameters, leaving little ambiguity.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool starts a bounded breadth-first crawl and immediately returns a job_id, using a specific verb and resource. It distinguishes itself from sibling tools like crawl_status and crawl_results by focusing on initiation rather than status checking or retrieval.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context on how the crawl behaves and explicitly instructs to poll with crawl_status and read results with crawl_results. It does not explicitly contrast with alternatives like fetch_page or get_sitemap, but the follow-up workflow is clearly indicated, earning a 4 rather than a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. 10 tool updatesv0.1.0
    • First observedcancel_crawl
    • First observedcheck_robots
    • First observedcrawl_results
    • First observedcrawl_status
    • First observedextract
    • First observedextract_links
    • First observedextract_tables
    • First observedfetch_page
    • First observedget_sitemap
    • First observedstart_crawl

TDQS

A4.2/5.0
Disambiguation5/5

Each tool has a clearly distinct purpose: fetching a page, extracting tables, links, structured data, checking sitemap/robots, and managing crawls. There is no overlap between single-page extraction tools and crawl lifecycle tools.

Naming Consistency5/5

Tool names consistently follow a verb_noun pattern: extract_tables, extract_links, fetch_page, get_sitemap, check_robots, start_crawl, crawl_status, etc. The generic 'extract' is slightly less patterned but still readable and predictable.

Tool Count5/5

With 10 tools, the server is well-scoped for its purpose. The set balances single-page extraction utilities with essential crawl management operations without unnecessary bloat.

Completeness4/5

The domain of web scraping and crawling is well covered: page fetching, link/table/structured extraction, robots and sitemap handling, plus full crawl lifecycle (start, status, results, cancel). Minor gap is lack of a tool to list all crawls, but core workflows are complete.

Maintenance

ActivityInactive
ResponsivenessNo issues

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

  • F
    license
    B
    quality
    C
    maintenance
    A full-featured MCP server for building scrapers, with tools for page fetching, HTML parsing, CSS/XPath querying, and spider generation using silkworm-rs and scraper-rs.
    22
    10
    -
  • A
    license
    Not graded
    quality
    A
    maintenance
    Remote MCP server for web scraping with anti-bot evasion. Provides stealth HTTP fetching, headless browser with Cloudflare bypass, CSS selectors, YouTube transcripts, and Markdown conversion.
    1
    MIT
  • F
    license
    A
    quality
    C
    maintenance
    A portable MCP server that scrapes SPA/JS-rendered webpages and extracts structured API endpoint data. It uses headless Chromium to render JavaScript and provides tools for scraping, endpoint extraction, screenshots, and database seeding.
    4
    1
    -

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/eitan3/Scrapy_MCP_Scraper'

If you have feedback or need assistance with the MCP directory API, please join our Discord server