spidra-mcp-server
OfficialThis server provides web scraping, crawling, and data extraction through Spidra infrastructure via Model Context Protocol.
Scrape pages: fetch 1–3 URLs and get clean markdown, text, tables, or structured JSON; supports AI extraction, browser actions, screenshots, cookies, and residential proxies.
Batch scrape: process 2–50 URLs in parallel, each returning independent per-URL results.
Crawl websites: start from one URL, follow plain-English link instructions, and extract data; supports path filtering, depth limits, subdomain traversal, and proxy routing.
Re-extract from crawls: apply new extraction instructions to completed crawls without re-fetching pages.
Job management: check statuses of scrapes, batches, and crawls; cancel pending or running jobs; retrieve full page content via signed download links.
Usage and logs: browse historical scrape logs with filters; check credit, token, and request usage.
Deployment: available hosted or locally via stdio/HTTP, with automatic retries and clear error messages.
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@spidra-mcp-serverscrape https://example.com/about for team member names"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Spidra MCP Server
The official Model Context Protocol (MCP) server for Spidra.
MCP is the standard that lets AI assistants use external tools. When you connect this server to an assistant like Claude Code, Claude Desktop, Cursor, Windsurf, VS Code, or Antigravity, the assistant gains the ability to scrape web pages, process lists of URLs, and crawl entire websites on its own.
All you have to do is describe what you want in plain language, and the assistant picks the right Spidra tool, runs it, and works with the extracted data directly in the conversation.
Features
Scrape any page and get clean markdown or structured JSON back
Extract exactly the fields you want using plain-language prompts or JSON schemas
Process up to 50 URLs in parallel with one request
Crawl whole sites by describing which links to follow in plain English
Run browser actions before scraping: click, type, scroll, or loop over elements
Route through residential proxies for geo-restricted or bot-protected sites
Hosted Streamable HTTP endpoint at
mcp.spidra.io— connect with one URL, nothing to installOAuth login for OAuth-capable clients, or a plain API key header for everything else
Also runs locally over stdio, or as your own self-hosted HTTP service
Built-in guidance that keeps the assistant from wasting your credits
Automatic retries for flaky network moments, with clear typed errors otherwise
Related MCP server: Thordata MCP Server
Before you start
You need two things:
A Spidra account. Sign up at app.spidra.io. Using the API-key method below instead of OAuth? Create one under Settings > API Keys — keys start with
spd_.An MCP-compatible client. Any of the assistants below works.
That's it if you use the hosted endpoint below. Running the server yourself additionally needs Node.js 20 or newer (check with node --version) — the npx command that runs it ships with Node.
Hosted (no install)
All you have to do is point your client at https://mcp.spidra.io/mcp.
Every request against the hosted endpoint is billed to your account exactly like a direct API call.
OAuth (recommended)
Log in with your Spidra account in the browser.
claude mcp add --transport http spidra https://mcp.spidra.io/mcpStart a new Claude Code session and run /mcp. It opens a browser to log in and approve access, then shows the connection as active.
Logging in via OAuth automatically creates (and reuses) a dedicated API key labeled MCP (OAuth) on your account — visible and revocable anytime under Settings > API Keys, exactly like any other key.
API key (alternative)
Prefer a static key for CI, scripting, or a client without OAuth support:
Claude Code
claude mcp add --transport http spidra https://mcp.spidra.io/mcp --header "Authorization: Bearer spd_YOUR_API_KEY"Start a new Claude Code session, then run /mcp to confirm the connection shows as active.
Claude Desktop
Open Settings > Developer > Edit Config, and add the spidra entry inside mcpServers:
{
"mcpServers": {
"spidra": {
"type": "http",
"url": "https://mcp.spidra.io/mcp",
"headers": {
"Authorization": "Bearer spd_YOUR_API_KEY"
}
}
}
}Quit and reopen Claude Desktop afterward.
Cursor
Open Cursor Settings
Go to Features > MCP Servers
Click + Add new global MCP server
Paste the following and replace the placeholder key:
{
"mcpServers": {
"spidra": {
"type": "http",
"url": "https://mcp.spidra.io/mcp",
"headers": {
"Authorization": "Bearer spd_YOUR_API_KEY"
}
}
}
}You can also put this in a .cursor/mcp.json file inside a single project if you only want Spidra available there.
VS Code
Press Ctrl + Shift + P (or Cmd + Shift + P on Mac), type Preferences: Open User Settings (JSON), and add:
{
"mcp": {
"servers": {
"spidra": {
"type": "http",
"url": "https://mcp.spidra.io/mcp",
"headers": {
"Authorization": "Bearer spd_YOUR_API_KEY"
}
}
}
}
}In a .vscode/mcp.json workspace file, drop the outer "mcp" wrapper — the root key there is servers directly, not mcpServers.
Windsurf
Add this to ~/.codeium/windsurf/mcp_config.json:
{
"mcpServers": {
"spidra": {
"type": "http",
"url": "https://mcp.spidra.io/mcp",
"headers": {
"Authorization": "Bearer spd_YOUR_API_KEY"
}
}
}
}Antigravity
Add this to ~/.gemini/config/mcp_config.json (global) or .agents/mcp_config.json (workspace-local). Note the field is serverUrl, not url/httpUrl — Antigravity rejects those field names for remote servers:
{
"mcpServers": {
"spidra": {
"serverUrl": "https://mcp.spidra.io/mcp",
"headers": {
"Authorization": "Bearer spd_YOUR_API_KEY"
}
}
}
}Local installation (stdio)
Prefer to run the server yourself? Every setup below does the same thing: it tells your assistant to run npx -y spidra-mcp and hands the server your API key through an environment variable.
This is also the option to use if you're calling a staging or self-hosted Spidra API instead of the public one — see Configuration.
Claude Code
Run this one command in your terminal, replacing the placeholder with your real key:
claude mcp add spidra -e SPIDRA_API_KEY=spd_YOUR_API_KEY -- npx -y spidra-mcpStart a new Claude Code session, then run /mcp to confirm the connection shows as active.
Cursor
Open Cursor Settings
Go to Features > MCP Servers
Click + Add new global MCP server
Paste the following and replace the placeholder key:
{
"mcpServers": {
"spidra": {
"command": "npx",
"args": ["-y", "spidra-mcp"],
"env": {
"SPIDRA_API_KEY": "spd_YOUR_API_KEY"
}
}
}
}You can also put this in a .cursor/mcp.json file inside a project if you only want Spidra available there.
Claude Desktop
Open Settings > Developer > Edit Config. This opens
claude_desktop_config.jsonAdd the
spidraentry insidemcpServers(create the object if the file is empty):
{
"mcpServers": {
"spidra": {
"command": "npx",
"args": ["-y", "spidra-mcp"],
"env": {
"SPIDRA_API_KEY": "spd_YOUR_API_KEY"
}
}
}
}Quit and reopen Claude Desktop. The tools appear under the tools icon in the chat input.
VS Code
Add this to your User Settings (JSON). Press Ctrl + Shift + P (or Cmd + Shift + P on Mac), type Preferences: Open User Settings (JSON), and add:
{
"mcp": {
"servers": {
"spidra": {
"command": "npx",
"args": ["-y", "spidra-mcp"],
"env": {
"SPIDRA_API_KEY": "spd_YOUR_API_KEY"
}
}
}
}
}To share the setup with your team instead, put the same servers block in a .vscode/mcp.json file in your repository and use a promptString input for the key so it never gets committed.
Windsurf
Add this to ~/.codeium/windsurf/mcp_config.json:
{
"mcpServers": {
"spidra": {
"command": "npx",
"args": ["-y", "spidra-mcp"],
"env": {
"SPIDRA_API_KEY": "spd_YOUR_API_KEY"
}
}
}
}Antigravity
Add this to ~/.gemini/config/mcp_config.json (global) or .agents/mcp_config.json (workspace-local):
{
"mcpServers": {
"spidra": {
"command": "npx",
"args": ["-y", "spidra-mcp"],
"env": {
"SPIDRA_API_KEY": "spd_YOUR_API_KEY"
}
}
}
}Running over HTTP instead of stdio
By default the server talks to your client over stdio, which is what all the configs above use and what you want on a single machine. If you need an HTTP endpoint instead (for example, a tool that connects to MCP servers over the network), start the server like this:
env HTTP_STREAMABLE_SERVER=true SPIDRA_API_KEY=spd_YOUR_API_KEY npx -y spidra-mcpThen connect to http://localhost:3000/mcp. On this transport the API key can also be sent per request using an Authorization: Bearer header, which is useful when one server instance serves more than one user.
This is exactly how the hosted https://mcp.spidra.io/mcp endpoint runs in production — one shared instance, keyed per request, so you don't need to set SPIDRA_API_KEY at all when using the hosted URL.
Try it
Once connected, just ask for web data in normal language. You never call the tools by name; the assistant does that for you. Some things to try:
"Scrape https://news.ycombinator.com and give me the top 5 stories with their points."
"Compare the pricing pages of stripe.com and paddle.com and tell me which is cheaper for a small SaaS."
"Here are 12 product URLs. Get me the name, price, and rating for each one as a table."
"Crawl the first 10 pages of docs.example.com and summarize what the product does."
If the assistant answers with real data from those pages, everything is working.
Configuration
These apply to the local/self-hosted server (npx, manual install, or your own HTTP instance). The hosted endpoint at mcp.spidra.io needs none of them — just your API key in a header.
Variable | Required | Description |
| Yes | Your Spidra API key, starting with |
| No | Override the API base URL, for staging or self-hosted setups |
| No | Set to |
| No | Bind address for the HTTP transport. Defaults are |
On the HTTP transport (self-hosted or hosted), the API key can also be sent per request via an Authorization: Bearer header, which is useful when one server instance serves more than one user. If a request carries both a header key and a logged-in OAuth session, the header key wins.
How to choose a tool
This section is written for humans, but the same guidance is embedded in the tool descriptions, so the assistant follows it on its own.
The deciding question is not how many URLs you have. It is what you want back:
You want one answer, and you know the URL (or 2 to 3 related URLs): use scrape. When you pass several URLs, their content is merged and the AI answers once across all of them. That makes it the right tool for comparing two pricing pages or summarizing three related articles into one answer.
You want separate data for each URL in a list: use batch scrape, even if the list only has 2 items. Every URL is processed independently and returns its own result. This is the tool for "extract the same fields from each of these product pages."
You do not know the page URLs yet: use crawl. You give it one starting URL and a plain-English instruction about which links to follow, and it discovers the pages itself.
Quick reference
Tool | Best for | Waits or polls? |
| One combined answer from 1 to 3 known URLs | Waits, returns the result directly |
| Re-checking a scrape that outlived its wait window | Instant lookup |
| Separate results for each of 2 to 50 known URLs | Returns a |
| Progress and per-URL results for a batch | Instant lookup |
| Stopping a batch you no longer need | Instant |
| Discovering and processing pages from one starting URL | Returns a |
| Progress, then full results, for a crawl | Instant lookup |
| Per-page results with raw HTML and markdown download links | Instant lookup |
| Asking a new question of an already-completed crawl | Returns a new |
| Stopping a crawl you no longer need | Instant |
| Looking up past jobs and their outputs | Instant lookup |
| Checking credit and request usage | Instant lookup |
A note on output format
When you need specific fields from a page, ask for JSON and describe the fields, or provide a JSON schema. The assistant gets back a small, focused payload instead of an entire page, which keeps the conversation fast and cheap. Ask for full markdown only when you genuinely need the whole page, such as summarizing a complete article.
If you use a schema, define every field you want extracted. An untyped object with no properties gives the AI nothing to fill in, so those fields come back empty.
Available tools
1. Scrape (spidra_scrape)
Scrapes 1 to 3 URLs and extracts their content with AI. This tool waits for the result, typically 10 to 60 seconds, and returns the extracted content directly. No polling needed.
The important behavior to understand: when you pass more than one URL, their content is combined and the AI produces one answer across all of them. The raw per-page content still comes back in the pages field, but the extraction itself is a single, merged result. Use multiple URLs here when you want the AI to compare or synthesize across pages. If you want the same extraction run separately on each URL, use spidra_batch_scrape instead, even for just 2 URLs.
Best for:
Getting content or specific data from a page you already know
One combined answer drawn from 2 or 3 related pages, like a pricing comparison
Not recommended for:
Separate results per URL (use
spidra_batch_scrape)Discovering pages on a site (use
spidra_crawl)
Common mistakes:
Passing several unrelated URLs expecting individual results for each. You will get one merged answer. Use batch scrape for per-URL results.
Omitting a prompt and a schema when you wanted structured data. With neither, you get the page back as raw markdown.
Prompt example:
"Get the product name, price, and description from https://example.com/product."
Usage example (structured extraction with a schema):
{
"name": "spidra_scrape",
"arguments": {
"urls": ["https://example.com/product"],
"prompt": "Extract the product information",
"output": "json",
"schema": {
"type": "object",
"properties": {
"name": { "type": "string" },
"price": { "type": "number" },
"description": { "type": "string" }
},
"required": ["name", "price"]
}
}
}Usage example (compare two pages in one answer):
{
"name": "spidra_scrape",
"arguments": {
"urls": ["https://competitor-a.com/pricing", "https://competitor-b.com/pricing"],
"prompt": "Compare the plans on these two pages and list the differences in price and features",
"output": "json"
}
}Usage example (raw markdown, no AI extraction):
{
"name": "spidra_scrape",
"arguments": {
"urls": ["https://example.com/blog/some-article"]
}
}Other options worth knowing:
actions: browser steps to run before extraction, in order. Supportsclick,type,check,uncheck,wait,scroll, andforEach(loop over every matching element, optionally with pagination). Use this to dismiss cookie banners, run a search, or expand hidden content before the scrape happens.cookies: a raw Cookie header string for pages behind a login, for example"session=abc123; token=xyz".useProxyandproxyCountry: route through a residential proxy, optionally pinned to a country like"us"or"de". Use for geo-restricted content or sites that block datacenter traffic.screenshot: capture a viewport screenshot. The result includes a URL to the image.extractContentOnly: strip navigation, ads, and boilerplate before the AI sees the page.scrapeMode:"fast"uses plain HTTP with no browser. Cheaper and quicker, but it cannot run actions or render JavaScript-heavy pages. The default mode uses a real browser.
Returns: the extracted content, the per-page raw data in pages, any screenshots, and stats with token counts and timing. If the wait window is ever exceeded, the job keeps running on the server and the error message hands the assistant the job ID to check with spidra_check_scrape_status. Nothing is lost.
2. Check scrape status (spidra_check_scrape_status)
Looks up a scrape job by ID. You only need this in one situation: a scrape took longer than the wait window (very slow or heavily protected sites). The timeout error includes the job ID, and the assistant uses this tool to fetch the result once the job finishes.
{
"name": "spidra_check_scrape_status",
"arguments": {
"jobId": "550e8400-e29b-41d4-a716-446655440000"
}
}Returns: the job status (waiting, active, completed, or failed) and the full result when completed.
3. Batch scrape (spidra_batch_scrape)
Submits 2 to 50 URLs that are all processed in parallel with the same prompt or schema. Each URL is handled independently and gets its own result. This is the opposite of multi-URL scrape, which merges everything into one answer.
This tool returns immediately with a batchId. It does not wait, because a 50-URL batch can take several minutes. The assistant then polls spidra_check_batch_status every 10 to 15 seconds until the batch reaches a terminal state. The tool's own response tells the assistant to do exactly that, so you do not have to manage any of it.
Best for:
Running the same extraction on each of many similar pages: product pages, listings, articles, profiles
Any case where you need a separate row of data per URL, even with only 2 URLs
Not recommended for:
One combined answer across pages (use
spidra_scrape)Pages you have not discovered yet (use
spidra_crawl)
Common mistakes:
Resubmitting the batch because results did not come back instantly. The batch is running; poll the status instead.
Prompt example:
"Here are 15 product URLs. Extract the name, price, and star rating from each one."
Usage example:
{
"name": "spidra_batch_scrape",
"arguments": {
"urls": [
"https://shop.example.com/product/1",
"https://shop.example.com/product/2",
"https://shop.example.com/product/3"
],
"prompt": "Extract the product name, price, and star rating",
"output": "json",
"schema": {
"type": "object",
"properties": {
"name": { "type": "string" },
"price": { "type": "string" },
"rating": { "type": "number" }
}
}
}
}Returns: { "batchId": "...", "total": 3 } plus instructions for the assistant to poll. URLs here are plain strings, not objects.
4. Check batch status (spidra_check_batch_status)
Fetches the current state of a batch: overall status, progress counters, and per-URL results for every item that has finished so far.
{
"name": "spidra_check_batch_status",
"arguments": {
"batchId": "550e8400-e29b-41d4-a716-446655440000"
}
}Returns: batch status (pending, running, completed, failed, or cancelled), completedCount, failedCount, and an items array where each entry carries its URL, status, extraction result, credits used, and timestamps.
One thing to know: a completed batch can still contain individual failed items. Check failedCount. Failed items can be retried without re-running the whole batch through the Spidra API or SDKs, which have a batch retry endpoint.
5. Cancel batch (spidra_cancel_batch)
Cancels a pending or running batch. Items that already finished keep their results, and credits for unprocessed items are refunded automatically.
{
"name": "spidra_cancel_batch",
"arguments": {
"batchId": "550e8400-e29b-41d4-a716-446655440000"
}
}Returns: the number of cancelled items and the credits refunded.
6. Crawl (spidra_crawl)
Crawls a website starting from one URL. Spidra discovers pages by following links according to a plain-English instruction you provide, and optionally extracts structured data from every page it visits. This is the tool for "get me something from every page in this section of a site" when you do not have the page URLs.
Like batch, this returns immediately with a jobId and the assistant polls spidra_check_crawl_status until it finishes.
Two instructions control a crawl, and keeping them straight matters:
crawlInstructioncontrols which links get followed. For example, "Follow blog post links only, skip tag and category pages."transformInstructioncontrols what gets extracted from each page. For example, "Extract the title, author, and publish date." If you leave it out (and pass no schema), each page comes back as raw markdown and no AI tokens are charged at all.
Best for:
Docs sites, blogs, product catalogs, or any section of a site where you want data from many pages you have not listed out
Building a structured dataset from a whole site section in one request
Not recommended for:
URLs you already know (scrape or batch scrape are faster and cheaper)
A single page (use
spidra_scrape)
Common mistakes:
Setting
maxPageshigher than needed. Every crawled page costs credits. Start small; you can always crawl again.Putting extraction wording into
crawlInstruction. Link-following and extraction are separate instructions.
Prompt example:
"Crawl example.com/blog, follow only the article links, and get me each post's title, author, and date. Cap it at 10 pages."
Usage example:
{
"name": "spidra_crawl",
"arguments": {
"baseUrl": "https://example.com/blog",
"crawlInstruction": "Follow blog post links only, skip tag and category pages",
"transformInstruction": "Extract the title, author, and publish date",
"maxPages": 10
}
}Scoping options: maxPages (default 5, maximum 50), maxDepth (0 means the base URL only), includePaths and excludePaths (path patterns like "/blog/*"), allowSubdomains, crawlEntireDomain, and ignoreQueryParams (treat URLs that differ only by query string as the same page). cookies, useProxy, and proxyCountry work the same as in scrape.
Returns: { "jobId": "..." } plus polling instructions for the assistant.
7. Check crawl status (spidra_check_crawl_status)
Fetches a crawl's progress while it runs, and its full results once it completes.
{
"name": "spidra_check_crawl_status",
"arguments": {
"jobId": "550e8400-e29b-41d4-a716-446655440000"
}
}Returns: while running, the status and a progress object with pagesCrawled out of maxPages. When completed, an array with every crawled page's URL, title, and extracted data. Terminal statuses are completed, failed, and cancelled.
8. Crawl pages (spidra_crawl_pages)
Fetches per-page results for a crawl, including signed download URLs for each page's raw HTML and markdown. Useful when you want the original page content rather than only the extracted data. The download links expire after 1 hour, so use them promptly.
This also works on cancelled crawls, returning whatever pages finished before the cancellation.
{
"name": "spidra_crawl_pages",
"arguments": {
"jobId": "550e8400-e29b-41d4-a716-446655440000"
}
}Returns: an array of pages, each with its URL, status, extracted data, and signed html and markdown download URLs.
9. Re-extract from a crawl (spidra_crawl_extract)
Runs a brand new extraction instruction over a crawl that already completed, without fetching any pages again. Spidra kept the page content, so only AI token credits are charged, no per-page scraping cost. This is the cheap way to ask a second question of the same site.
For example: you crawled a competitor's blog extracting titles and dates. Now you want the key topics of each post too. Re-extract instead of re-crawling.
{
"name": "spidra_crawl_extract",
"arguments": {
"jobId": "550e8400-e29b-41d4-a716-446655440000",
"transformInstruction": "List the main topics each post covers and its target audience"
}
}Returns: a new jobId. The assistant polls spidra_check_crawl_status with it, same as a normal crawl. The source crawl must have status completed.
10. Cancel crawl (spidra_cancel_crawl)
Cancels a queued or running crawl. Pages that were already processed are kept and remain retrievable through spidra_crawl_pages, and credits for unprocessed pages are refunded.
{
"name": "spidra_cancel_crawl",
"arguments": {
"jobId": "550e8400-e29b-41d4-a716-446655440000"
}
}Returns: confirmation with the cancelled job's ID.
11. Scrape logs (spidra_scrape_logs)
Browses past scrape jobs on the account: what ran, when, whether it succeeded, and how many credits it used. Pass a uuid to fetch one log entry with its complete AI output. Useful for finding the result of an earlier job, debugging a failure, or reviewing what a key has been used for.
{
"name": "spidra_scrape_logs",
"arguments": {
"status": "failed",
"searchTerm": "amazon.com",
"limit": 10
}
}Returns: a list of log entries with URLs, status, credits, tokens, and timing. With a uuid, the single entry including its full extraction result.
12. Usage (spidra_usage)
Reports the account's request, credit, and token usage broken down by day or week. Ask the assistant "how many credits have I used this week?" and this is the tool it reaches for. Also handy before kicking off a large batch or crawl.
{
"name": "spidra_usage",
"arguments": {
"range": "7d"
}
}Returns: rows of usage data. Accepted ranges are "7d", "30d", and "weekly".
Credits and how this server protects them
Every scraped URL costs credits: a base of 2 credits per URL, plus AI tokens when extraction runs, plus 10 credits per CAPTCHA solved. Agent loops can burn through credits quickly if the tools let them, so this server is deliberately built to prevent that:
The tool descriptions steer the assistant toward the cheapest tool that answers the question, and tell it to keep
maxPagessmall.Long-running jobs return a job ID with explicit polling instructions, so the assistant never resubmits a job that is still running.
Timeout errors say, in effect, "this job is still running, poll it, do not retry." Duplicate submissions are also deduplicated server-side within a short window.
Rate limit errors tell the assistant exactly how many seconds to wait. Validation errors list exactly what to fix and say not to retry unchanged. Permanent errors say not to retry at all. This prevents the expensive retry loops agents are prone to.
Cancelling unfinished work refunds the unprocessed portion, and the cancel tools say so in their descriptions.
Output size
Large pages and big crawls can produce more text than fits in a model's context window. The server truncates individual strings above 5,000 characters and caps any single tool response at 80,000 characters, marking every truncation clearly. If the assistant needs the complete raw content of crawled pages, spidra_crawl_pages provides download links to the full files.
Error handling
Errors come back as readable messages, not stack traces, and each one carries guidance the assistant can act on:
{
"content": [
{
"type": "text",
"text": "Rate limited (TOO_MANY_PENDING_JOBS): You have too many jobs queued. Wait 30 seconds before retrying."
}
],
"isError": true
}Transient network failures and 5xx responses are retried automatically with backoff before you ever see an error, courtesy of the underlying Spidra Node SDK.
Troubleshooting
The assistant does not see any Spidra tools. Restart your client after adding the config. Most clients only read MCP configuration at startup. In Claude Code, run
/mcpto check the connection status."No Spidra API key configured." For local/stdio setups, the
SPIDRA_API_KEYvariable is not reaching the server — make sure it is inside theenvblock of the server entry, not at the top level of the config file. For the hosted endpoint, check your header name and value (below).Hosted endpoint returns a 401. The header is missing, misnamed, or the key has been revoked. It must be exactly
Authorization: Bearer spd_..., and the key must still exist under Settings > API Keys in your dashboard.OAuth login doesn't open a browser, or the client falls back to asking for a key. Not every MCP client supports OAuth yet — use the API key method for that client instead. If a browser window did open but the flow failed partway through, retry; if it keeps failing, confirm you're logged into app.spidra.io in that browser.
OAuth login fails with "Invalid or expired transaction" or similar, then the assistant reports no credentials found. The login attempt was interrupted before it finished — usually from taking too long on the approve screen (the flow expires after 10 minutes) or retrying a stale login link. Remove and re-add the connector to start clean, and approve access promptly. Falls back to the API key method for that client if it keeps happening.
A scrape "timed out." The job is still running on the server and nothing is lost. The error includes the job ID, and the assistant will fetch the result with
spidra_check_scrape_status. Bot-protected sites can take a couple of minutes.Results come back empty when using a schema. Check the schema: every field you want must be defined with a type. An object with no properties gives the AI nothing to fill in.
npxcannot find the package. Make sure you are on Node 20 or newer and that your network allows access to the npm registry. This only applies to local/self-hosted setups — the hosted endpoint doesn't usenpx.
Development
git clone https://github.com/spidra-io/spidra-mcp-server.git
cd spidra-mcp-server
npm install
npm run build # bundles to dist/index.js
npm test # black-box smoke tests: spawns the built binary against a fake API
npm run typecheckTo run your local build against a local Spidra API, set SPIDRA_API_URL=http://localhost:4321/api.
Contributions are welcome. Fork the repository, create a feature branch, make sure npm test passes, and open a pull request.
License
MIT
Available Tools
12 toolsspidra_batch_scrapeARead-only
Scrape a list of 2-50 known URLs in parallel with the same extraction prompt/schema. Each URL is processed INDEPENDENTLY and gets its OWN result (unlike spidra_scrape, which merges multiple URLs into one combined answer). This tool returns IMMEDIATELY with a batchId — it does not wait.
Best for: running the same extraction on each of many similar pages (product pages, listings, articles) where you need separate data per URL — even for just 2 URLs. Workflow: call this, then poll spidra_check_batch_status with the batchId every 10-15 seconds until the batch reaches a terminal state. Do NOT resubmit while a batch is pending.
Costs: 2 credits per URL plus AI tokens. Failed items can be retried from the dashboard or cancelled with spidra_cancel_batch (credits for unprocessed items are refunded).
| Name | Required | Description | Default |
|---|---|---|---|
| urls | Yes | 2-50 URLs to scrape in parallel (plain strings) | |
| output | No | ||
| prompt | No | What to extract from each page. Omit for raw markdown. | |
| schema | No | JSON Schema enforcing the exact output shape. Define EVERY field you want extracted — an untyped object with no properties comes back empty. Missing fields return null instead of hallucinated values. | |
| cookies | No | ||
| useProxy | No | Route through a residential proxy (for blocked/geo-restricted sites) | |
| scrapeMode | No | ||
| proxyCountry | No | Two-letter country code for the proxy, e.g. "us", "de", "jp", or "eu"/"global" | |
| extractContentOnly | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true, openWorldHint=true, and destructiveHint=false. The description adds valuable behavioral context beyond those annotations: the tool returns immediately with a batchId instead of waiting, processes each URL independently, has a 10-15 second polling cadence, costs 2 credits per URL, and refunds unprocessed items on cancellation. This gives the agent important operational expectations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is about 120 words, front-loaded with the core behavior, followed by a 'Best for' section and a short workflow. Every sentence contributes actionable information—scope, async behavior, polling cadence, costs, and cancellation—so conciseness is excellent without sacrificing substance.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (9 parameters, async batch processing, no output schema), the description covers the full operational loop: how to invoke, immediate return, polling until terminal state, avoidance of resubmission, cost implications, retries/cancellation, and distinction from sibling tools. This is sufficient for an agent to use the tool correctly without additional documentation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 56%, so the schema carries a moderate amount of parameter meaning. The description reinforces URL count limits, the need for a unified prompt/schema per URL, and that omitting prompt yields raw markdown. However, it does not add clarity for undocumented parameters like output, cookies, scrapeMode, or extractContentOnly, leaving those gaps unresolved.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description states a specific verb ('Scrape'), a concrete resource ('a list of 2-50 known URLs'), and the processing model (each URL independently with its own result). It also explicitly distinguishes itself from spidra_scrape, making the tool's purpose unmistakable compared to siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit 'Best for' scenarios, contrasts with spidra_scrape, and gives a clear workflow: call the tool, get a batchId, poll spidra_check_batch_status every 10-15 seconds until terminal state, and do not resubmit while pending. It also mentions spidra_cancel_batch for failures, offering concrete alternative usage.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
spidra_cancel_batchADestructive
Cancel a pending or running batch scrape. Credits for unprocessed items are refunded; already-finished items keep their results.
| Name | Required | Description | Default |
|---|---|---|---|
| batchId | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description goes beyond the destructiveHint annotation by explaining the consequences: refunds for unprocessed items and preservation of finished results. This adds valuable behavioral context not available in structured data. It does not contradict annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise (two sentences) and front-loaded with the main action. Every sentence adds value, with no unnecessary words or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (one parameter, no output schema), the description covers the key behavioral aspects: what it cancels, refund behavior, and item preservation. It is sufficiently complete for an agent to invoke correctly, though it does not describe potential errors or return values.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has one parameter (batchId) with 0% description coverage, and the description does not explain what batchId refers to or where to obtain it. The parameter semantics are under-specified, although the name 'batchId' is self-explanatory, it lacks guidance on its provenance or validation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Cancel') and the resource ('a pending or running batch scrape'), distinguishing it from sibling tools like spidra_cancel_crawl by specifying scrape type. It also mentions outcome details (credits refunded, finished items kept), making the purpose unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context for when to use: canceling a batch scrape that is pending or running. It does not explicitly name alternatives or exclusions, but it is evident that this tool is for batch cancels, not crawls or individual scrapes. Lacks explicit when-not guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
spidra_cancel_crawlADestructive
Cancel a queued or running crawl job. Pages already processed are preserved and retrievable with spidra_crawl_pages; credits for unprocessed pages are refunded.
| Name | Required | Description | Default |
|---|---|---|---|
| jobId | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare this as destructive (destructiveHint=true). The description adds valuable behavioral context beyond that: partial preservation of processed pages (retrievable via spidra_crawl_pages) and credit refunds for unprocessed pages. This helps the agent understand the consequences. No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and front-loaded with the core action. It consists of two sentences, each contributing meaningful information (what is canceled and the consequences). No fluff or redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is simple—one parameter, no output schema. The description explains the immediate effect (cancellation), the state of processed pages, and the refund policy. It does not describe the response format, but that is not critical for a cancel action. It could mention edge cases (e.g., canceling a completed job), but that is a minor gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has one required parameter, jobId, with 0% description coverage. The description does not elaborate on the format or source of jobId beyond its name. However, the context of 'cancel a crawl job' makes it clear the parameter identifies the job. Since the description adds minimal meaning beyond the schema, a 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool cancels a crawl job, using a specific verb ('Cancel') and resource ('crawl job'), and qualifies the state ('queued or running'), distinguishing it from sibling tools like spidra_check_crawl_status or spidra_crawl.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly says when to use the tool: for queued or running crawl jobs. This implies it is not for canceled or completed jobs, and not for batch jobs (which have their own cancel tool). It does not explicitly name alternatives, but the context makes the usage clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
spidra_check_batch_statusARead-only
Check the status of a batch scrape by batchId. Returns per-URL statuses and results for finished items. Batch statuses: pending/running are in progress; completed, failed, and cancelled are terminal. A completed batch can still contain failed items — check failedCount.
| Name | Required | Description | Default |
|---|---|---|---|
| batchId | Yes | The batch id returned by spidra_batch_scrape |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and destructiveHint=false, so the safety profile is known. The description adds valuable behavioral detail: returns per-URL statuses, defines terminal vs in-progress states, and warns that completed batches can still contain failed items—key nuances for interpreting the result.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three sentences with no fluff. It front-loads the purpose, then efficiently explains status categories and an important exception. Every sentence earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple read-only tool with one required parameter and no output schema, the description fully covers what the agent needs: status meaning, return contents, and a critical edge case. It is self-contained and actionable without being verbose.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and the parameter description already explains batchId is returned by spidra_batch_scrape. The tool description adds no new parameter-level meaning beyond restating that connection, so it aligns with the baseline for well-covered schemas.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states 'Check the status of a batch scrape by batchId', identifying the specific verb-resource pair. It distinguishes from sibling spidra_check_scrape_status by explicitly saying 'batch' and detailing what it returns (per-URL statuses, results).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides clear usage context by tying batchId to spidra_batch_scrape and explaining status semantics (in-progress vs terminal). It does not explicitly name alternatives or exclusions, but the singular purpose and batch-specific language implicitly guide correct selection.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
spidra_check_crawl_statusARead-only
Check the status of a crawl job by jobId. While running, returns progress (pagesCrawled/maxPages). When completed, returns the extracted data for every page. Job statuses: waiting/active/running are in progress; completed, failed, and cancelled are terminal.
| Name | Required | Description | Default |
|---|---|---|---|
| jobId | Yes | The crawl job id returned by spidra_crawl |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the readOnlyHint annotation, the description adds valuable behavioral detail: it explains that running jobs return progress (pagesCrawled/maxPages) and completed jobs return extracted data. It also categorizes job statuses into in-progress and terminal, which is helpful context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three sentences long, with the main purpose in the first sentence and relevant details about return values and statuses in the following sentences. There is no redundant or extraneous content.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the simplicity of the tool (one parameter, read-only) and the absence of an output schema, the description adequately explains what the caller can expect: progress for running jobs, extracted data for completed jobs, and the meaning of each status. It is complete for its purpose.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already provides full coverage for the single parameter with the description 'The crawl job id returned by spidra_crawl.' The tool description only reiterates that it is checked by jobId, adding no new semantic information beyond what the schema states.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's function: 'Check the status of a crawl job by jobId.' It distinguishes itself from sibling tools by focusing specifically on crawl jobs and by describing the returned data based on job status.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description clearly indicates this is for checking crawl job status, and the parameter description specifies the jobId comes from spidra_crawl. It does not explicitly name alternatives like check_scrape_status, but the context is clear and unambiguous.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
spidra_check_scrape_statusARead-only
Check the status of a scrape job by jobId. Only needed when spidra_scrape reported that its wait window was exceeded. Job statuses: waiting/active/running are in progress; completed, failed, and cancelled are terminal.
| Name | Required | Description | Default |
|---|---|---|---|
| jobId | Yes | The scrape job id |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and destructiveHint=false, so the safety profile is covered. The description adds behavioral context by explaining what the job statuses mean (in-progress vs terminal), which goes beyond the annotations. No contradiction exists.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loaded with the core purpose, and every word adds value. It is concise without being under-specified.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple status-check tool with one parameter and no output schema, the description provides the trigger condition and lists status categories, which is sufficient for an agent to understand the tool's role and expected result. It could optionally describe the return format, but the status list implies it.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% for the single parameter jobId, with a description 'The scrape job id' in the schema itself. The tool description adds nothing semantically beyond repeating 'by jobId', so the baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool checks the status of a scrape job by jobId, using a specific verb and resource. It distinguishes itself from sibling tools like spidra_check_batch_status and spidra_check_crawl_status by explicitly targeting 'scrape job' status.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives an explicit condition: 'Only needed when spidra_scrape reported that its wait window was exceeded.' This provides clear when-to-use context, but it does not name alternatives like spidra_check_batch_status, so it falls short of a full 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
spidra_crawlARead-only
Crawl a website starting from one URL: Spidra discovers pages by following links according to your plain-language instruction, and optionally extracts structured data from every page. Returns IMMEDIATELY with a jobId — it does not wait.
Best for: extracting from many pages when you do NOT know their URLs upfront (docs sites, blogs, product catalogs). Not for: URLs you already know (use spidra_scrape or spidra_batch_scrape — cheaper and faster). Workflow: call this, then poll spidra_check_crawl_status with the jobId every 10-15 seconds until terminal. Do NOT resubmit while a crawl is pending. Cancel a mistake with spidra_cancel_crawl.
Behavior notes:
"crawlInstruction" controls which links are followed (e.g. "Follow blog post links only, skip tag pages").
"transformInstruction" controls what is extracted per page; omit it (and schema) for raw markdown with no AI token cost.
Keep "maxPages" small (default 5, max 50) — every page costs credits.
Usage example:
{
"name": "spidra_crawl",
"arguments": {
"baseUrl": "https://example.com/blog",
"crawlInstruction": "Follow blog post links only, skip tag and category pages",
"transformInstruction": "Extract the title, author, and publish date",
"maxPages": 10
}
}| Name | Required | Description | Default |
|---|---|---|---|
| schema | No | JSON Schema enforcing the exact output shape. Define EVERY field you want extracted — an untyped object with no properties comes back empty. Missing fields return null instead of hallucinated values. | |
| baseUrl | Yes | Starting URL for the crawl | |
| cookies | No | ||
| maxDepth | No | Max link depth from the base URL. 0 = base URL only. | |
| maxPages | No | Max pages to crawl (default 5). Keep small — each page costs credits. | |
| useProxy | No | Route through a residential proxy (for blocked/geo-restricted sites) | |
| excludePaths | No | URL path patterns to skip, e.g. ["/tag/*"] | |
| includePaths | No | URL path patterns to include, e.g. ["/blog/*"] | |
| proxyCountry | No | Two-letter country code for the proxy, e.g. "us", "de", "jp", or "eu"/"global" | |
| allowSubdomains | No | ||
| crawlInstruction | Yes | Which links to follow, in plain language | |
| crawlEntireDomain | No | ||
| ignoreQueryParams | No | ||
| transformInstruction | No | What to extract from each page. Omit for raw markdown (no AI cost). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the annotations (readOnlyHint, openWorldHint, destructiveHint), the description discloses that the tool returns immediately with a jobId and runs asynchronously. It warns not to resubmit while pending, explains the cost implications of maxPages, and details how crawlInstruction and transformInstruction affect link-following and extraction. These are valuable behavioral details not present in the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with a clear opening sentence, then 'Best for', 'Not for', workflow, behavior notes, and a usage example. Every section contributes useful information; there is no repetition of schema details or fluff. It is appropriately detailed for a complex async tool, and front-loads the most important information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers the tool's purpose, async behavior, workflow, cost implications, and key parameters, and provides an example. While it doesn't explain every optional parameter or the exact return format, the schema covers most parameters and the description gives enough context for an agent to select and invoke the tool correctly. The missing output schema is partially compensated by the workflow explanation, but some details about response structure are absent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
While the schema already documents 71% of parameters, the description adds important semantics for the two central instruction parameters (crawlInstruction, transformInstruction) and maxPages, clarifying defaults and cost. It also provides a concrete usage example that shows how to combine parameters. This goes beyond the schema's basic descriptions, though it doesn't explain every optional parameter in depth.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly defines the tool: it crawls a website from a starting URL, follows links based on a crawlInstruction, and optionally extracts structured data. It also distinguishes itself from related tools by stating what it is 'Not for' and naming alternatives like spidra_scrape and spidra_batch_scrape. This goes beyond a simple restatement, giving a specific verb and resource.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly provides 'Best for' and 'Not for' scenarios, naming spidra_scrape and spidra_batch_scrape as cheaper alternatives when URLs are known. It also outlines the required workflow, including polling with spidra_check_crawl_status and canceling with spidra_cancel_crawl. This is model guidance that clearly tells the agent when to use this tool and when to pick a sibling tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
spidra_crawl_extractARead-only
Run a NEW extraction prompt over an already-completed crawl without re-crawling any pages — much cheaper than crawling again (only AI token credits are charged). Returns a new jobId immediately; poll spidra_check_crawl_status with it. The source crawl must have status "completed".
| Name | Required | Description | Default |
|---|---|---|---|
| jobId | Yes | The completed source crawl job id | |
| transformInstruction | Yes | The new extraction instruction to apply to every crawled page |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the readOnlyHint and destructiveHint annotations, the description adds key context: no pages are re-crawled, only AI token credits are charged, a new jobId is returned immediately, and the source crawl must be completed. This gives the agent a clear model of the tool's behavior and cost implications.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three focused sentences with no filler. The main action is front-loaded, followed by cost-saving rationale and immediately actionable usage details (polling, precondition). Every sentence contributes value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple two-parameter job-creation tool, the description covers the use case, cost, return behavior, polling workflow, and a required precondition. With good annotations and full schema coverage, no significant information is missing for an agent to decide and invoke correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already provides 100% coverage with descriptions for both jobId and transformInstruction. The description does not add additional parameter-specific details, so it meets the baseline for schema-covered parameters but does not enhance understanding of parameter formats or constraints.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool runs a new extraction prompt over an already-completed crawl without re-crawling pages. It distinguishes itself from crawling tools by emphasizing the reuse of completed crawls, and from status tools by being an action that returns a jobId.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly conditions usage on the source crawl being 'completed' and positions it as a cheaper alternative to crawling again. It also directs the caller to poll spidra_check_crawl_status with the returned jobId, though it does not name alternative tools for other scenarios.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
spidra_crawl_pagesARead-only
Get per-page results for a crawl, including signed download URLs for each page's raw HTML and markdown (links expire after 1 hour). Works on completed crawls and on cancelled crawls (returns the pages processed before cancellation).
| Name | Required | Description | Default |
|---|---|---|---|
| jobId | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description adds valuable behavioral details beyond the readOnlyHint annotation: links expire after 1 hour, and cancelled crawls return partial results. This informs the user of time-sensitive URLs and the tool's behavior on incomplete crawls, which the annotations alone do not convey.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loaded with the primary purpose, and contains no filler. Every clause adds useful information: per-page results, signed URLs, expiry, and cancellation behavior.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (single parameter, no output schema), the description covers essential aspects: output type, URL expiry, and behavior on cancelled crawls. It is complete enough for an agent to select and invoke the tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has only one parameter, jobId, with no description (0% coverage). The description does not explicitly explain jobId, but the tool name and the phrase 'for a crawl' imply it is the crawl job identifier. While self-explanatory, the description adds no explicit parameter semantics beyond the parameter name.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's function: 'Get per-page results for a crawl' with specifics about signed download URLs for raw HTML and markdown. It also distinguishes itself from sibling tools like spidra_check_crawl_status by focusing on page-level output rather than status.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context for when the tool is applicable: 'Works on completed crawls and on cancelled crawls (returns the pages processed before cancellation).' It implicitly guides the user away from using it on active crawls, though it does not explicitly name alternative tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
spidra_scrapeARead-only
Scrape 1-3 known URLs and extract their content with AI. This tool WAITS for the result (typically 10-60 seconds) and returns the extracted content directly.
IMPORTANT: with multiple URLs, their content is COMBINED and the AI produces ONE answer across all of them (the per-URL raw pages are still returned in "pages"). Use several URLs here when you want to compare or synthesize across pages — e.g. "compare the pricing on these two pages". If instead you want the SAME extraction run separately on each URL (own result per URL), use spidra_batch_scrape even for just 2 URLs.
Best for: one URL, or one combined answer drawn from 2-3 related URLs. Not for: per-URL independent results (use spidra_batch_scrape) or discovering pages on a site (use spidra_crawl).
Behavior notes:
Omit "prompt" and "schema" to get the raw page content as markdown.
Pass "prompt" for free-form AI extraction, and add "schema" when you need a guaranteed JSON shape. Define every field in the schema — untyped objects come back empty.
Use "actions" to interact with the page first (dismiss cookie banners, type into search boxes, scroll, or loop over elements with forEach).
Use "useProxy" with "proxyCountry" for geo-restricted or bot-protected sites.
Costs: 2 credits per URL plus AI tokens; CAPTCHA solves cost 10 credits each.
Usage example:
{
"name": "spidra_scrape",
"arguments": {
"urls": ["https://example.com/pricing"],
"prompt": "Extract all pricing plans with name, price, and included features",
"output": "json"
}
}Returns: extracted content plus token/credit stats. If the wait window is exceeded, the job keeps running — poll spidra_check_scrape_status with the returned jobId.
| Name | Required | Description | Default |
|---|---|---|---|
| urls | Yes | 1-3 URLs to scrape in parallel | |
| output | No | Output format (default "markdown") | |
| prompt | No | What to extract, in plain English. Omit for raw markdown. | |
| schema | No | JSON Schema enforcing the exact output shape. Define EVERY field you want extracted — an untyped object with no properties comes back empty. Missing fields return null instead of hallucinated values. | |
| actions | No | Browser actions to run on each URL before extraction, in order | |
| cookies | No | Raw Cookie header string for pages behind a login, e.g. "session=abc" | |
| useProxy | No | Route through a residential proxy (for blocked/geo-restricted sites) | |
| scrapeMode | No | "fast" = HTTP only (no browser), cheaper but less capable | |
| screenshot | No | Capture a viewport screenshot (URL returned) | |
| proxyCountry | No | Two-letter country code for the proxy, e.g. "us", "de", "jp", or "eu"/"global" | |
| extractContentOnly | No | Strip navigation/ads/boilerplate before extraction |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the readOnly/destructive annotations, the description discloses the synchronous wait behavior, timeout fallback with spidra_check_scrape_status, credit costs, CAPTCHA costs, and the combined-answer behavior. No contradiction with annotations; it adds substantial context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is fairly long but well-structured with headers and bullets. A few points restate schema descriptions (e.g., untyped schema warning), but the example and prioritization earn their place. Slightly tighter would be perfect.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description explains returns (content + stats) and timeout behavior. It covers costs, alternatives, usage example, and edge cases like CAPTCHA and proxy. This is a complete guide for agent invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Although schema coverage is 100%, the description adds essential meaning: omitting prompt/schema yields raw markdown, untyped objects come back empty, and the relationship between prompt and schema is clarified. The usage example demonstrates correct argument structure.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb+resource: "Scrape 1-3 known URLs and extract their content with AI." It clearly distinguishes from siblings by explaining that multiple URLs produce a combined answer, while independent per-URL extraction belongs to spidra_batch_scrape.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicit "Best for" and "Not for" sections name alternatives (spidra_batch_scrape, spidra_crawl). It also gives conditional guidance for actions, proxy, and schema usage, telling the agent when each parameter is appropriate.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
spidra_scrape_logsARead-only
List past scrape jobs for this account with optional filters. Useful for finding a previous job's result, debugging failures, or checking what a key has been used for. Fetch a single log's full AI output by passing its uuid.
| Name | Required | Description | Default |
|---|---|---|---|
| page | No | ||
| uuid | No | Fetch one log entry (with full extraction output) instead of listing | |
| limit | No | Results per page (default 10 here) | |
| status | No | ||
| searchTerm | No | Filter by URL or prompt substring |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already disclose read-only and non-destructive behavior. The description adds meaningful context beyond annotations: the distinction between listing logs (summaries) versus fetching a single log's full AI output, plus the account scoping and key-usage insight. This enriches the behavioral model without contradicting annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three short, front-loaded sentences: the first states the primary action, the second gives use cases, and the third explains a key parameter mode. Every sentence adds value, with no redundancy or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (5 optional params, no output schema) and strong annotations, the description sufficiently covers purpose, usage, and the uuid behavior. It lacks a description of the returned log format, but overall the agent can infer basic behavior. The gap is minor, so it earns a 4 rather than a 5.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 60%, partially compensating for parameter details. The description explicitly explains the uuid parameter's special behavior (fetch full output) and mentions 'optional filters,' but leaves page and status without dedicated explanation. The schema and description together cover most semantics, but the generic 'filters' phrase does not fully eludicate all parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with 'List past scrape jobs for this account with optional filters,' which uses a specific verb ('list'), identifies the resource ('scrape jobs'), and scopes it to the account. It also distinguishes a second mode ('Fetch a single log's full AI output') and clearly separates this from siblings like status checkers and scraping tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit use cases: 'finding a previous job's result, debugging failures, or checking what a key has been used for.' This gives clear context for when to use the tool, though it does not explicitly name alternatives or state when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
spidra_usageARead-only
Get this account's request/credit/token usage broken down by day or week. Use it to answer "how many credits have I used" style questions or to check remaining headroom before a large batch/crawl.
| Name | Required | Description | Default |
|---|---|---|---|
| range | No | Time range (default "30d") |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The annotations already declare readOnlyHint=true and destructiveHint=false, so the agent knows this is a safe read operation. The description adds meaningful context beyond annotations by specifying the scope ('this account's'), the nature of usage (request/credit/token), and the breakdown granularity (day/week). It doesn't disclose return format or limits, but with annotations covering the safety profile, this is sufficient.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loaded with the core purpose and then a clear use-case statement. Every sentence earns its place; no filler or unnecessary detail. It is exceptionally concise while remaining informative.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple tool with one optional parameter, no output schema, and no nested objects, the description covers the what, when, and scope. It lacks an explicit description of return format, but without an output schema, the burden is moderate. The use-case framing ('remaining headroom') adds practical context, making this adequately complete for an agent to invoke correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, with the single 'range' parameter fully documented in the schema (enum and default). The description adds no additional parameter-specific meaning beyond repeating the day/week granularity already in the enum. Since the schema does the heavy lifting, a baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's function: 'Get this account's request/credit/token usage broken down by day or week.' The verb 'Get' and resource 'usage' are specific, and the breakdown by day/week adds clear scope. It also distinguishes from sibling tools, which are all scraping/crawling operations.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states when to use: 'Use it to answer "how many credits have I used" style questions or to check remaining headroom before a large batch/crawl.' This gives direct usage context and implicitly separates it from the scraping/crawling siblings, making the intended invocation clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
12 tool updates
v0.2.2- First observed
spidra_batch_scrape - First observed
spidra_cancel_batch - First observed
spidra_cancel_crawl - First observed
spidra_check_batch_status - First observed
spidra_check_crawl_status - First observed
spidra_check_scrape_status - First observed
spidra_crawl - First observed
spidra_crawl_extract - First observed
spidra_crawl_pages - First observed
spidra_scrape - First observed
spidra_scrape_logs - First observed
spidra_usage
TDQS
Each tool has a clearly distinct purpose: scrape (single combined), batch_scrape (parallel independent), crawl (discovery), with separate status/cancel tools per job type. The descriptions explicitly contrast spidra_scrape vs spidra_batch_scrape vs spidra_crawl, eliminating ambiguity.
All tools share the 'spidra_' prefix and mostly follow a verb_noun pattern (check_scrape_status, cancel_batch, crawl_pages). Minor deviations like 'spidra_usage' (pure noun) and 'spidra_crawl_extract' (slightly awkward compound) prevent a perfect score, but the overall convention is predictable.
12 tools is well-scoped for a scraping/crawling server. Each tool covers a specific action (start, poll, cancel, inspect) for each operation type (scrape, batch, crawl), plus logs and usage monitoring. No redundant or superfluous tools.
The toolset covers the full lifecycle for batch and crawl jobs (create, status check, cancel, retrieve results), plus logs and usage. Minor gaps: no explicit cancel for a single long-running scrape, and retry of failed items is only available via dashboard, not API. Core workflows are otherwise complete.
Maintenance
Resources
Unclaimed servers have limited discoverability.
Looking for Admin?
If you are the server author, to access and configure the admin panel.
Related MCP Connectors
Enable language models to perform advanced AI-powered web scraping with enterprise-grade reliabili…
Web scraping for AI agents. Extract text and metadata from any URL worldwide. $0.005/page.
AI-powered browser automation — navigate, click, fill forms, and extract data from any website.
Cloud scraping & crawling API for AI agents. Turn any URL into clean, LLM-ready markdown.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceProvides AI agents and coding assistants with advanced web crawling and RAG capabilities, allowing them to scrape websites and leverage that knowledge through various retrieval strategies.2MIT
- AlicenseNot gradedqualityNot gradedmaintenanceEnables AI models to scrape and extract structured data from any website globally using a 195+ country proxy network with JavaScript rendering, anti-bot bypass, and output in Markdown, HTML, or Links format.-
- FlicenseNot gradedqualityDmaintenanceEnables AI assistants to crawl websites, extract and store web content with semantic search capabilities using vector embeddings, and retrieve information through natural language queries with tag-based filtering and intelligent content cleaning.-
- AlicenseNot gradedqualityDmaintenanceProvides AI agents and assistants with advanced web crawling and RAG capabilities, enabling them to scrape websites and perform semantic search over crawled content.1MIT
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/spidra-io/spidra-mcp-server'
If you have feedback or need assistance with the MCP directory API, please join our Discord server