scrapy-mcp
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@scrapy-mcpScrape the main content of example.com as markdown"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
scrapy-mcp
A headless web-scraping MCP server built on Scrapy. It exposes Scrapy's scraping primitives — polite fetching, CSS/XPath extraction, link and table extraction, sitemap and robots.txt reading, and bounded asynchronous crawls — as MCP tools an agent can call over stdio.
Headless, no rendering. Pages are fetched and parsed as HTML; no browser, no JavaScript execution. This keeps the footprint tiny — it runs comfortably on weak machines.
Reactor-safe. Every operation runs in a short-lived Scrapy subprocess, so Twisted's reactor never lives inside the asyncio MCP server (no
ReactorNotRestartable), and memory is reclaimed after each call.Polite by default. Obeys
robots.txt, throttles with AutoThrottle, and enforces hard page/depth caps so a crawl can't run away.
Install / run
Run straight from PyPI with uv — no install step:
uvx scrapy-mcpOr install it:
uv pip install scrapy-mcp
scrapy-mcpThe server speaks MCP over stdio. Point any MCP client at it. For Claude Desktop, add to
claude_desktop_config.json:
{
"mcpServers": {
"scrapy": {
"command": "uvx",
"args": ["scrapy-mcp"]
}
}
}Related MCP server: silkworm-mcp
Tools
Tool | What it does |
| Fetch one page as |
| Pull structured fields with CSS/XPath selectors. |
| Extract every HTML |
| List de-duplicated links on a page. |
| Read a sitemap (gzip + sitemap-index aware). |
| Is a URL crawlable? Returns the crawl-delay and sitemaps. |
| Start a bounded BFS crawl; returns a |
| State + pages scraped for a crawl. |
| Page through a crawl's scraped items. |
| Stop a running crawl; keep results so far. |
Selector format (extract / start_crawl)
selectors maps an output field to a selector. Each value is either a CSS string (first
match) or an object for more control:
{
"title": "h1::text",
"price": "span.price::text",
"all_links": {"css": "a::attr(href)", "all": true},
"first_heading": {"xpath": "//h1/text()"}
}"all": true returns every match as a list; otherwise the first match is returned.
Crawls are asynchronous
start_crawl returns immediately with a job_id. The crawl runs as a detached worker that
streams results to disk, so it survives a server restart. Poll crawl_status(job_id), then
read items with crawl_results(job_id) (safe to call mid-crawl for partial results). Jobs are
stored under the system temp dir and reclaimed after 7 days (configurable).
Configuration
All settings are optional environment variables (sensible, polite defaults tuned for a weak
host). They're how you tune a uvx scrapy-mcp deployment.
Variable | Default | Meaning |
|
| User-Agent header. |
|
| Obey |
|
| Seconds between requests to a host. |
|
| Global concurrency. |
|
| Per-host concurrency. |
|
| Per-request timeout (s). |
|
| Retries on transient failures. |
|
| Adapt delay to server latency. |
|
| Max characters returned per page (then truncated). |
|
| Wall-clock cap for a blocking single fetch (s). |
|
| Crawl page default / hard cap. |
|
| Crawl depth default / hard cap. |
|
| Where crawl jobs are stored. |
|
| Delete crawl jobs older than this (0 disables). |
|
| Scrapy log level (to stderr). |
Development
uv venv
uv pip install -e ".[dev]"
uv run pytest # unit tests (no network)
uv build # build wheel + sdist into dist/License
MIT © Eitan Hadar
Available Tools
10 toolscancel_crawlA
Stop a running crawl by killing its worker process. Results so far are kept.
| Name | Required | Description | Default |
|---|---|---|---|
| job_id | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses the destructive act of killing the worker process and the side effect that results are kept. This goes beyond a vague 'cancel' and gives the agent meaningful expectations about consequences.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two short sentences, front-loaded with the core action and immediately followed by the most important side effect. No filler or redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (one parameter, no output schema, no annotations), the description covers the key aspects: what it stops, how it stops it, and what persists. It doesn't mention whether cancellation is reversible or what is returned, but those are not critical for this simple operation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has one required parameter (job_id) but zero description coverage. The description does not explicitly define job_id, though the tool name and context make it inferable as the crawl's ID. It would benefit from stating 'the ID of the crawl to cancel' or referencing start_crawl.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action (stop a running crawl), the mechanism (killing its worker process), and the outcome (results kept). This is a specific verb+resource+scope statement that distinguishes cancel_crawl from siblings like start_crawl and crawl_status.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies the use case: when a crawl is running and must be stopped. It does not explicitly contrast with alternatives, but the purpose is so singular that the context is clear. It would benefit from mentioning 'use crawl_status to check if a crawl is running', but the baseline is solid.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
check_robotsA
Check the site's robots.txt: is url crawlable, and what crawl-delay applies?
user_agent: the agent to evaluate rules for (defaults to the server's user agent).
Returns {allowed, has_robots, crawl_delay, request_rate, sitemaps, robots_url}.
Sites with no robots.txt are reported as allowed (has_robots=false).
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| user_agent | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description discloses the return fields and the special behavior for missing robots.txt (reported as allowed with has_robots=false). Since there are no annotations, this carries the full burden and effectively communicates the tool's read-only nature and output.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise: one line for purpose, one for parameter, and one for return values. It is front-loaded with the main purpose and contains no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple read-only tool with no output schema, the description fully covers: purpose, parameter semantics, return object fields, and an edge case (missing robots.txt). It is complete and self-contained.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description compensates by explaining user_agent's default and role. The url parameter is implicitly described as the site to check but lacks explicit format details; overall it adds meaning beyond the raw schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool checks a site's robots.txt to determine if a URL is crawlable and what crawl-delay applies. This is a specific verb+resource that distinguishes it from siblings like fetch_page or get_sitemap.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The tool's purpose implies it should be used before crawling to check compliance, but there is no explicit 'when to use' or 'when not to use' guidance. Siblings like start_crawl exist but no alternative is mentioned, leaving usage inferred rather than stated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crawl_resultsA
Page through a crawl's scraped items.
cursor: line offset to start from (use the returned next_cursor to continue).
limit: items per page (1-1000). Safe to call while the crawl is still running --
you get whatever has been written so far.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | ||
| cursor | No | ||
| job_id | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It discloses key behavioral traits: cursor is a line offset, next_cursor is used to continue, limit range (1-1000), and that results reflect whatever has been written so far during an active crawl. This adds valuable context beyond the schema and clarifies the tool's eventual-consistency behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is brief and front-loads the main purpose. It then uses a clear list-style block for parameter guidance. Every sentence adds value, with no filler or repetition. The formatting is clean and readable.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple pagination tool with three parameters and no output schema, the description covers the primary purpose, parameter semantics, and a key behavioral nuance. It hints at the response shape via 'next_cursor' but does not explicitly describe the structure of items. Overall, this is complete enough for effective tool invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has 0% description coverage, so the description compensates well by explaining cursor and limit in detail. It does not explicitly describe job_id, but the tool name and 'a crawl's' context make it obvious that job_id identifies the crawl. This is sufficient for a required identifier parameter.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb "Page through" and clearly identifies the resource as "a crawl's scraped items." This distinguishes it from sibling tools like start_crawl or crawl_status, making its function unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context on when to use the tool (to retrieve crawl results in pages) and explains the pagination mechanism via cursor and limit. It also notes it can be safely used while the crawl is running, which implies appropriate usage during execution. It does not explicitly exclude alternatives, but the context is sufficient.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crawl_statusA
Report a crawl's state and progress.
state is one of: starting, running, finished, failed, cancelled, interrupted.
('interrupted' means the worker process died without finishing -- e.g. the machine or
server restarted; any results gathered so far are still readable.)
| Name | Required | Description | Default |
|---|---|---|---|
| job_id | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses possible state values and explains the 'interrupted' state in detail, including that results remain readable. This adds meaningful behavioral insight beyond a basic status check, though error behavior is not covered.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and front-loaded with the core purpose, followed by a necessary list of state values and a clarifying note. No redundant wording.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is simple, but without an output schema the description leaves the return format ambiguous ('state and progress' but no structure). It also does not address invalid job IDs, though the state enumeration is a helpful addition.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has one required parameter, job_id, with no description beyond the name. The tool description does not mention job_id or its provenance (e.g., returned by start_crawl), leaving parameter semantics under-specified despite 0% schema coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Report a crawl's state and progress.' with a specific verb and resource. This distinguishes it from sibling tools like start_crawl and crawl_results, which focus on initiating and fetching results.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides context about the meaning of states, especially 'interrupted,' but does not explicitly state when to use this tool versus alternatives like crawl_results or cancel_crawl. Usage is implied but not explicitly articulated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extractA
Fetch a page and pull structured fields out of it with CSS or XPath selectors.
selectors maps an output field name to a selector. Each value is either:
a CSS string, e.g. "h1::text" or "a.product::attr(href)" (returns the first match), or
an object: {"css": "...", "all": true} / {"xpath": "//h1/text()", "all": true} ("all": true returns every match as a list; default returns the first match).
Example: {"title": "h1::text", "prices": {"css": ".price::text", "all": true}} Returns {"data": {field: value | [values]}}.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| selectors | Yes | ||
| obey_robots | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the transparency burden. It explains selector syntax, first-match vs all behavior, and return format. However, it does not disclose the meaning of the obey_robots parameter or potential error handling, leaving some behavioral aspects untold.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-organized and front-loaded with the main purpose, followed by a compact specification of selector formats and an example. Every section adds necessary value, and formatting uses clear bullet points and code blocks.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description is largely complete for a complex extraction tool, including return structure and selector behavior. It could be more complete by addressing edge cases, obey_robots semantics, or error handling, but these are secondary to the core extraction logic.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It does so thoroughly for the selectors parameter, detailing CSS string and object forms with examples. The url parameter is implied by 'Fetch a page.' The obey_robots parameter is not explained, but its name and default null convey partial meaning.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's action: 'Fetch a page and pull structured fields out of it with CSS or XPath selectors.' It specifies the resource (page) and the mechanism (CSS/XPath), distinguishing it from siblings like extract_tables and extract_links which are more specialized.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context on how to use the tool by explaining the selectors mapping and examples. However, it does not explicitly mention when not to use it or alternatives such as extract_tables or fetch_page, so it falls short of full guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extract_linksA
Fetch a page and return its links (de-duplicated, absolute URLs).
same_domain: keep only links on the same host as url.
pattern: keep only links whose URL matches this regular expression.
limit: maximum number of links to return (1-2000).
Each link is {"url": ..., "text": ..., "nofollow": bool}.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| limit | No | ||
| pattern | No | ||
| obey_robots | No | ||
| same_domain | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden of disclosure. It does disclose meaningful behaviors: deduplication, absolute URL conversion, and the output link object structure. However, it omits any explanation of the `obey_robots` parameter, which could significantly affect behavior (e.g., respecting robots.txt), and provides no details on error handling or edge cases like pages with no links.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is efficiently structured: a one-sentence core purpose followed by a compact parameter list and output format. Every line adds value with no redundancy, using a clear code block for parameters and return shape. It is front-loaded with the essential 'fetch and return links' message.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite no annotations or output schema, the description provides a solid overview of the tool's behavior and return format. It explains the main parameters and output object. The primary gap is the undocumented `obey_robots` parameter and lack of mention of potential errors or page fetch failures, preventing a perfect score.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It does explain `same_domain` (same host), `pattern` (regular expression), and `limit` (1-2000), adding value beyond the schema. However, `obey_robots` is entirely absent, and `pattern` lacks specifics about regex flavor or matching rules. The description covers 3 of 5 parameters meaningfully but leaves key gaps.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The first sentence clearly states the tool's function: 'Fetch a page and return its links (de-duplicated, absolute URLs).' It uses a specific verb and resource, and the output is clearly defined. This distinguishes it from sibling tools like fetch_page (returns page content) and extract (generic extraction) by focusing solely on link extraction with deduplication and URL normalization.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage contexts through the parameter explanations (e.g., same_domain, pattern) but provides no explicit guidance on when to choose this tool over alternatives such as extract_tables or get_sitemap. There are no 'use this when...' or 'for X use...' statements, leaving usage as inferred rather than explicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extract_tablesA
Fetch a page and extract every HTML as {headers, rows}.
max_tables: cap on how many tables to return (1-100). Each table is
{"headers": [...], "rows": [[cell, ...], ...], "n_rows": int, "n_cols": int}.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| max_tables | No | ||
| obey_robots | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the burden. It discloses the max_tables cap and specifies the return structure, but omits details like network side effects, robots.txt handling, error behavior, or whether JavaScript is executed. It is not misleading but lacks depth.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three sentences, front-loaded with the main purpose, and each sentence adds value. The return format is compactly specified, with no wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, the description does provide the return format, which is helpful. However, it lacks guidance on usage context, exclude cases, and the third param (obey_robots), making it only partially complete for a moderate-complexity tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With 0% schema description coverage, the description must compensate. It defines max_tables well (cap 1-100) and clarifies url implicitly via 'Fetch a page', but provides no meaning for obey_robots, which remains ambiguous. Partial compensation for a 3-param tool.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool fetches a page and extracts HTML <table> elements, with a specific output format. This distinguishes it from siblings like extract_links or fetch_page by focusing on tables as the resource.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for table extraction but does not explicitly mention when to use this tool over alternatives like fetch_page or extract. No exclusions or alternative tool names are provided, leaving usage guidance implicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
fetch_pageA
Fetch a single web page and return its content (no JavaScript rendering).
format: 'markdown' (default, compact and readable), 'text' (visible text only),
or 'html' (raw HTML).
max_bytes: cap on returned characters; defaults to the server's SCRAPY_MCP_MAX_BYTES.
Oversized content is truncated and truncated is set to true.
obey_robots: override the server's robots.txt policy for this call only.
An HTTP error status (404, 500, ...) is returned in status, not raised.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| format | No | markdown | |
| max_bytes | No | ||
| obey_robots | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description fully carries the transparency burden. It clearly discloses the lack of JavaScript rendering, truncation behavior with the 'truncated' flag, robots.txt override behavior, and that HTTP errors are returned in 'status' rather than raised. This is comprehensive and goes beyond basic expectations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the core purpose in the first sentence, followed by a concise bulleted parameter list and a final sentence on error handling. Every sentence adds value with no fluff or redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers the main behavior (content fetching, format options, truncation, error handling) and notes the 'status' and 'truncated' fields. However, it does not provide a full return structure or mention any response headers/URL that might be included, leaving some ambiguity about the exact response shape.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema description coverage is 0%, so the description must fully compensate. It explains 'format', 'max_bytes', and 'obey_robots' in detail, including defaults and effects. The 'url' parameter is implicitly clear from the tool's purpose. This exceeds the schema's bare definitions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource: 'Fetch a single web page and return its content'. It is clear and concise, but it does not explicitly differentiate from sibling tools like 'extract' or 'extract_links'. The 'no JavaScript rendering' note hints at scope but is not a direct alternative comparison.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for retrieving raw page content, but it does not explicitly state when to use this tool over siblings, nor does it provide exclusions (e.g., 'for structured data use extract'). The parameter explanations offer context but not direct comparative guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_sitemapA
Fetch a sitemap and return the URLs it lists.
Handles gzip-compressed sitemaps and recurses one level of into its
child sitemaps. url should point at a sitemap (e.g. https://site.com/sitemap.xml).
limit: maximum URLs to return (1-50000). Each entry is
{"loc": ..., "lastmod": ..., "changefreq": ..., "priority": ...}.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| limit | No | ||
| obey_robots | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It usefully explains that gzip-compressed sitemaps are handled, that `<sitemapindex>` is recursed one level, and the exact output entry structure. This adds non-obvious context beyond what the schema would imply.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is appropriately concise, with the purpose stated in the first sentence and additional details organized clearly. Every sentence adds value, and there is no redundant or fluff content.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple fetch tool with three parameters and no output schema, the description covers the main behaviors (gzip handling, index recursion, output format) and gives a working example. It omits details about `obey_robots` behavior and error cases, but overall it is substantially complete for common usage.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It does explain `url` and `limit` meaningfully (limit with a 1-50000 range), but `obey_robots` is entirely undocumented. Since one of three parameters is unexplained, the compensation is incomplete.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's function: 'Fetch a sitemap and return the URLs it lists.' The verb 'Fetch' and resource 'sitemap' are specific, and the output (URLs) is identified. It distinguishes itself from sibling tools by focusing exclusively on sitemaps rather than general pages or link extraction.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides a clear prerequisite: '`url` should point at a sitemap (e.g. https://site.com/sitemap.xml).' It also mentions the limit parameter. However, it does not explicitly compare to alternative tools like fetch_page or extract_links, nor does it state when not to use this tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
start_crawlA
Start a bounded breadth-first crawl. Returns a job_id immediately (non-blocking).
The crawl follows in-scope links from start_url up to max_pages / max_depth.
allow_patterns/deny_patterns: regexes a link URL must match / must not match.same_domain(default true): restrict the crawl tostart_url's host.selectors: same format as theextracttool; when given, each crawled page yields the extracteddata. Otherwise each page yields a short text excerpt.obey_robots,download_delay: per-crawl overrides of the server defaults.
Poll progress with crawl_status(job_id) and read items with crawl_results(job_id).
| Name | Required | Description | Default |
|---|---|---|---|
| max_depth | No | ||
| max_pages | No | ||
| selectors | No | ||
| start_url | Yes | ||
| obey_robots | No | ||
| same_domain | No | ||
| deny_patterns | No | ||
| allow_patterns | No | ||
| download_delay | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of disclosing behavior. It covers the non-blocking nature, bounded limits, same_domain default, robots/delay overrides, and selector semantics. It does not mention cancellation via cancel_crawl or potential rate limits, but the disclosed details are substantial and contextually useful.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise, front-loaded with the most important behavioral fact (non-blocking, returns job_id), and uses bullet-like separations for parameters. Every sentence adds value, and the overall length is appropriate for the tool's complexity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a complex crawl tool with no output schema, the description covers the initiation, parameter semantics, and follow-up tools, which is largely complete. It omits details about cancellation or error handling, but the presence of sibling tools like crawl_status and cancel_crawl partially fills that gap. Given the high complexity, a 4 is appropriate.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate for all 9 parameters. It explains allow/deny_patterns as regex constraints, same_domain as host restriction, selectors as same format as extract tool, and obey_robots/download_delay as per-crawl overrides. This fully covers the non-obvious parameters, leaving little ambiguity.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool starts a bounded breadth-first crawl and immediately returns a job_id, using a specific verb and resource. It distinguishes itself from sibling tools like crawl_status and crawl_results by focusing on initiation rather than status checking or retrieval.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context on how the crawl behaves and explicitly instructs to poll with crawl_status and read results with crawl_results. It does not explicitly contrast with alternatives like fetch_page or get_sitemap, but the follow-up workflow is clearly indicated, earning a 4 rather than a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
10 tool updates
v0.1.0- First observed
cancel_crawl - First observed
check_robots - First observed
crawl_results - First observed
crawl_status - First observed
extract - First observed
extract_links - First observed
extract_tables - First observed
fetch_page - First observed
get_sitemap - First observed
start_crawl
TDQS
Each tool has a clearly distinct purpose: fetching a page, extracting tables, links, structured data, checking sitemap/robots, and managing crawls. There is no overlap between single-page extraction tools and crawl lifecycle tools.
Tool names consistently follow a verb_noun pattern: extract_tables, extract_links, fetch_page, get_sitemap, check_robots, start_crawl, crawl_status, etc. The generic 'extract' is slightly less patterned but still readable and predictable.
With 10 tools, the server is well-scoped for its purpose. The set balances single-page extraction utilities with essential crawl management operations without unnecessary bloat.
The domain of web scraping and crawling is well covered: page fetching, link/table/structured extraction, robots and sitemap handling, plus full crawl lifecycle (start, status, results, cancel). Minor gap is lack of a tool to list all crawls, but core workflows are complete.
Maintenance
Resources
Unclaimed servers have limited discoverability.
Looking for Admin?
If you are the server author, to access and configure the admin panel.
Related MCP Connectors
One MCP for the Web. Easily search, crawl, navigate, and extract websites without getting blocked.…
One MCP server for 180+ live web-data APIs returning clean JSON from sites that block scrapers.
Free remote MCP server for fetching public web pages through a rotating proxy pool.
MCP server for web extraction and rendering via AceDataCloud WebExtrator
Related MCP Servers
- FlicenseBqualityDmaintenanceAn MCP Server for Web scraping and Crawling, built using Crawl4AI224-
- FlicenseBqualityCmaintenanceA full-featured MCP server for building scrapers, with tools for page fetching, HTML parsing, CSS/XPath querying, and spider generation using silkworm-rs and scraper-rs.2210-
- AlicenseNot gradedqualityAmaintenanceRemote MCP server for web scraping with anti-bot evasion. Provides stealth HTTP fetching, headless browser with Cloudflare bypass, CSS selectors, YouTube transcripts, and Markdown conversion.1MIT
- FlicenseAqualityCmaintenanceA portable MCP server that scrapes SPA/JS-rendered webpages and extracts structured API endpoint data. It uses headless Chromium to render JavaScript and provides tools for scraping, endpoint extraction, screenshots, and database seeding.41-
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/eitan3/Scrapy_MCP_Scraper'
If you have feedback or need assistance with the MCP directory API, please join our Discord server