Crawlbase MCP
OfficialClick on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Crawlbase MCPCrawl Hacker News and return top stories as markdown."
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
What is Crawlbase MCP?
Crawlbase MCP is a Model Context Protocol (MCP) server that bridges AI agents and the live web. Instead of relying on outdated training data, your LLMs can now fetch fresh, structured, real-time content — powered by Crawlbase’s proven crawling infrastructure trusted by 70,000+ developers worldwide.
It handles the complexity of scraping for you:
JavaScript rendering for modern web apps
Proxy rotation & anti-bot evasion
Structured outputs (HTML, Markdown, screenshots)
How It Works
Get Free Crawlbase Tokens → Sign up at Crawlbase ↗️, get free Normal, and JavaScript tokens.
Set Up MCP Configuration → Configure the MCP server in your preferred client (Claude, Cursor, or Windsurf) by updating the MCP Servers settings.
Start Crawling → Use commands like crawl, crawl_markdown, or crawl_screenshot to bring live web data into your AI agent.
Related MCP server: Raccoon AI MCP Server
Setup & Integration
Claude Desktop
Open Claude Desktop → Settings → Developer → Edit Config
Add to
claude_desktop_config.json:Replace
your_token_hereandyour_js_token_herewith the tokens from your dashboard.
{
"mcpServers": {
"crawlbase": {
"type": "stdio",
"command": "npx",
"args": ["@crawlbase/mcp@latest"],
"env": {
"CRAWLBASE_TOKEN": "your_token_here",
"CRAWLBASE_JS_TOKEN": "your_js_token_here"
}
}
}
}Claude Code
Add to your claude.json configuration:
{
"mcpServers": {
"crawlbase": {
"type": "stdio",
"command": "npx",
"args": ["@crawlbase/mcp@latest"],
"env": {
"CRAWLBASE_TOKEN": "your_token_here",
"CRAWLBASE_JS_TOKEN": "your_js_token_here"
}
}
}
}Cursor IDE
Open Cursor IDE → File → Preferences → Cursor Settings → Tools and Integrations → Add Custom MCP
Add to
mcp.json:Replace
your_token_hereandyour_js_token_herewith the tokens from your dashboard.
{
"mcpServers": {
"crawlbase": {
"type": "stdio",
"command": "npx",
"args": ["@crawlbase/mcp@latest"],
"env": {
"CRAWLBASE_TOKEN": "your_token_here",
"CRAWLBASE_JS_TOKEN": "your_js_token_here"
}
}
}
}Windsurf IDE
Open WindSurf IDE → File → Preferences → WindSurf Settings → General → MCP Servers → Manage MCPs → View raw config
Add to
mcp_config.json:Replace
your_token_hereandyour_js_token_herewith the tokens from your dashboard.
{
"mcpServers": {
"crawlbase": {
"type": "stdio",
"command": "npx",
"args": ["@crawlbase/mcp@latest"],
"env": {
"CRAWLBASE_TOKEN": "your_token_here",
"CRAWLBASE_JS_TOKEN": "your_js_token_here"
}
}
}
}HTTP Transport Mode
For scenarios where you need a shared MCP server accessible over HTTP (e.g., multi-user environments, custom integrations), you can run the server in HTTP mode:
# Clone and install
git clone https://github.com/crawlbase/crawlbase-mcp.git
cd crawlbase-mcp
npm install
# Start HTTP server with tokens (default port: 3000)
CRAWLBASE_TOKEN=your_token CRAWLBASE_JS_TOKEN=your_js_token npm run start:http
# Or with custom port
CRAWLBASE_TOKEN=your_token CRAWLBASE_JS_TOKEN=your_js_token MCP_PORT=8080 npm run start:httpThe server exposes:
POST /mcp- MCP Streamable HTTP endpointGET /health- Health check endpoint
Per-Request Token Authentication
HTTP mode supports per-request tokens via headers, allowing multiple users to share a single server:
curl -X POST http://localhost:3000/mcp \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-H "X-Crawlbase-Token: your_token" \
-H "X-Crawlbase-JS-Token: your_js_token" \
-d '{"jsonrpc": "2.0", "method": "tools/list", "id": 1}'Headers:
X-Crawlbase-Token- Normal token for HTML requestsX-Crawlbase-JS-Token- JavaScript token for JS-rendered pages/screenshots
Headers override environment variables when provided, enabling multi-tenant deployments.
🔑 Get your free tokens at Crawlbase ↗️.
Usage
Once configured, use these commands inside Claude, Cursor, or Windsurf:
crawl → Fetch raw HTML
crawl_markdown → Extract clean Markdown
crawl_screenshot → Capture screenshots
Example prompts:
“Crawl Hacker News and return top stories in markdown.”
“Take a screenshot of TechCrunch homepage.”
“Fetch Tesla investor relations page as HTML.”
Async Crawling with Cloud Storage
For larger jobs, Crawlbase MCP can push crawl results to Crawlbase Cloud Storage instead of returning them immediately. Your AI agent can then come back later to read, list, or clean up those pages — useful when crawling many URLs at once, revisiting a dataset across sessions, or keeping heavy HTML out of the chat until you actually need it.
Example prompts:
“Crawl these 50 product pages and save them to my Crawlbase storage. Once they're saved, summarize each one.”
“Save the Hacker News front page to storage so I can analyze it later.”
“How many pages do I have stored in Crawlbase right now? Show me the most recent 20.”
“Pull back everything I saved yesterday from my Crawlbase storage and give me a report.”
“Delete all the pages I have in Crawlbase storage — I'm done with that project.”
Use Cases
Market research → Pull live data from competitors, news, and reports
E-commerce monitoring → Track products, reviews, and prices in real time
News & finance feeds → Keep AI agents up-to-date with live events
Autonomous AI agents → Give them vision to act on fresh web data
Resources & Next Steps
Looking to supercharge your AI agents with live web data? Get started here:
✍️ Learn More – See how MCP powers AI agents with real-time web data ↗️
🌐 Crawlbase Website – Get free tokens & start crawling today ↗️

Copyright 2026 Crawlbase
Available Tools
9 toolscrawlB
Crawl a URL and return HTML content. Pass store=true to push the result to Crawlbase Cloud Storage instead of returning the body — the response will then contain only the RID and metadata, which can later be retrieved via storage_get.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to crawl | |
| store | No | Push the result to Crawlbase Cloud Storage. When true, the response returns only RID + metadata (no body) so the body can be fetched later via storage_get. | |
| device | No | Device type for crawling | |
| country | No | Country code for geo-targeting | |
| ajax_wait | No | Wait time for AJAX requests in milliseconds | |
| page_wait | No | Wait time for page load in milliseconds | |
| screenshot | No | Take a screenshot of the page | |
| user_agent | No | Custom user agent string |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It explains the store=true behavior (no body returned, only RID and metadata) which is good. However, it omits other behavioral traits such as default return format, error handling, rate limits, or permissions. It does not contradict annotations (none exist).
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences long, with the first sentence stating the core purpose and the second explaining the critical optional behavior. No redundant information. It is appropriately front-loaded and every sentence adds value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With 8 parameters and no output schema or annotations, the description covers only the core function and store behavior. It fails to describe other parameters, return values, error conditions, or how it relates to sibling tools. This leaves significant gaps for an agent to use the tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description adds meaning to the store parameter by explaining the behavioral change. For other parameters (device, country, etc.), it adds no additional context beyond the schema. Overall, it provides some extra value but does not fully compensate for the low coverage that would otherwise be needed.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it crawls a URL and returns HTML content, which is a specific verb+resource. It also details the store parameter behavior. However, it does not explicitly distinguish from sibling tools like crawl_markdown or crawl_screenshot, leaving some ambiguity.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides guidance on using store=true for storage, but lacks any when-to-use or when-not-to-use guidance compared to alternative tools. There is no mention of prerequisites or context for choosing crawl over its siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crawl_markdownA
Crawl a URL and extract clean markdown content. Pass store=true to persist the original page in Cloud Storage and return only the RID + metadata.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to crawl | |
| store | No | Push the result to Crawlbase Cloud Storage. When true, returns only RID + metadata; retrieve later with storage_get (use as=markdown to convert). | |
| device | No | Device type for crawling | |
| user_agent | No | Custom user agent string |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden. It discloses main behavior (crawl → markdown) and the store option's alternative return. However, it omits error handling, rate limits, or response format details.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, direct and front-loaded. Every word is necessary; no fluff or redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has 4 parameters and no output schema. Description explains the two operational modes but does not specify the return structure for the default mode (e.g., the markdown content format) or error cases. It's adequate but not thorough.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema descriptions cover all 4 parameters (100% coverage), so baseline is 3. The description rephrases the store parameter's behavior but adds no new meaning beyond the schema for url, device, or user_agent.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it crawls a URL and extracts clean markdown content, specifying both verb and resource. It implies differentiation from siblings like crawl (likely raw) and crawl_screenshot by focusing on markdown output.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explains the store parameter's effect on behavior, but lacks explicit guidance on when to use this tool versus siblings (e.g., crawl for raw HTML). No disclaimers or alternatives are mentioned.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crawl_screenshotA
Take a screenshot of a webpage. Pass store=true to persist the underlying HTML page to Cloud Storage (the screenshot itself is not stored and is still returned as an ephemeral screenshot_url).
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to screenshot | |
| mode | No | Screenshot mode (default: fullpage) | |
| store | No | Persist the underlying HTML page to Cloud Storage. The screenshot itself is not stored; only RID + metadata + screenshot_url are returned (no image download). Storage is scoped to the JS token — use_js_token=true is required when retrieving it later. | |
| width | No | Maximum width in pixels (only with mode=viewport) | |
| device | No | Device type for screenshot | |
| height | No | Maximum height in pixels (only with mode=viewport) | |
| page_wait | No | Wait time before taking screenshot |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden. It discloses that the screenshot is ephemeral (returns an ephemeral screenshot_url), explains that store=true persists the HTML page (not the screenshot), and notes storage scoping to JS token. It does not mention rate limits or auth beyond token hint, but provides key behavioral context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with no waste. First sentence states main action, second sentence adds the critical nuance about store parameter. Information is front-loaded and every word adds value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has 7 parameters and no output schema, the description covers the main behavior but does not explain return format (beyond mentioning screenshot_url) or common use cases for parameters like mode, device, page_wait. It is adequate but could be more complete for an agent to invoke correctly without schema cross-referencing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description adds value by clarifying the store parameter behavior (persists HTML, not screenshot) and mentioning the use_js_token requirement, which is not in the schema description. This exceeds minimal schema explanation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool takes a screenshot of a webpage, distinguishing it from sibling tools like crawl and crawl_markdown which have different purposes (crawling and markdown extraction). The verb 'Take' with resource 'screenshot of a webpage' is specific and unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides guidance on using the store parameter (when to persist HTML) but does not explicitly state when to use this tool over siblings like crawl or crawl_markdown. It implies usage for visual capture but lacks explicit when-not or alternative comparisons.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
storage_bulk_deleteB
Bulk-delete up to 100 items from Crawlbase Cloud Storage by RID. Irreversible.
| Name | Required | Description | Default |
|---|---|---|---|
| rids | Yes | Up to 100 storage RIDs to delete. | |
| use_js_token | No | Operate on the JS token's storage instead of the normal token's. Set to true if the target RIDs came from a crawl whose response showed token_type=js. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided. The description only notes irreversibility, which is critical, but lacks other behavioral details like partial failures, idempotency, or required authentication. Minimal disclosure.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, zero waste. Front-loaded with purpose and key constraint, then immediate behavioral warning. Appropriate for a straightforward tool.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema and moderate complexity, the description covers purpose and irreversibility. However, lacks usage context, error handling notes, and batch semantics, making it adequate but not complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and both parameters have clear descriptions in the schema. The description adds no additional parameter-specific information beyond the schema, so baseline is 3.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Bulk-delete', the resource 'Crawlbase Cloud Storage', the constraint 'by RID', and the limit 'up to 100 items'. It distinguishes from siblings like storage_delete (single) and storage_bulk_get.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when to use this tool vs alternatives such as storage_delete for single deletes. No prerequisites or contexts mentioned.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
storage_bulk_getA
Bulk-fetch up to 100 stored items by RID. Bodies are decoded automatically. Default as=metadata_only returns just RID/URL/timestamps to keep context lean; use as=html or as=markdown to include bodies.
| Name | Required | Description | Default |
|---|---|---|---|
| as | No | metadata_only (default) returns RID/URL/timestamps only; json returns full records; html/markdown include bodies. | |
| rids | Yes | Up to 100 storage RIDs to retrieve. | |
| auto_delete | No | Delete each item from storage after it is retrieved. | |
| use_js_token | No | Operate on the JS token's storage instead of the normal token's. Set to true if the target RIDs came from a crawl whose response showed token_type=js. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Without annotations, description carries full burden. It mentions automatic decoding and default return format, but does not disclose side effects of auto_delete, authentication needs, or what happens on errors. Could be improved.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences, front-loaded with key information (limit, decoding, default behavior). No wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Adequate for a tool with 4 parameters and no output schema. Covers main use and important parameter. Could mention return format and clarify auto_delete as destructive action.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema covers all parameters with 100% description coverage. Description adds value by explaining default 'as' behavior and trade-offs, but does not elaborate on auto_delete or use_js_token beyond schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states the tool bulk-fetches up to 100 stored items by RID, with automatic decoding. It distinguishes from siblings like storage_get (single item) and storage_bulk_delete (delete operation).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Implies usage for fetching multiple items, but does not explicitly mention alternatives like storage_get for single items or storage_bulk_delete for deleting. Provides guidance on the 'as' parameter but lacks when-not-to-use.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
storage_countA
Get the total number of documents in Crawlbase Cloud Storage.
| Name | Required | Description | Default |
|---|---|---|---|
| use_js_token | No | Query the JS token's storage instead of the normal token's. Set to true if the prior crawl response showed token_type=js. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It clearly indicates a read operation (get count), which is transparent for this simple tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One concise sentence that front-loads the purpose, with no unnecessary words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema exists, so the description should hint at the return format. It doesn't, though for a simple count it's minimally adequate given sibling tools.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and describes the parameter well. The description adds no additional meaning beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it retrieves the total number of documents, distinguishing it from sibling tools that list, get, or delete specific documents.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when to use this tool versus alternatives like storage_list or storage_get. The context is implied but not stated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
storage_deleteA
Delete a single item from Crawlbase Cloud Storage by RID.
| Name | Required | Description | Default |
|---|---|---|---|
| rid | Yes | Storage request ID to delete. | |
| use_js_token | No | Operate on the JS token's storage instead of the normal token's. Set to true if the target RIDs came from a crawl whose response showed token_type=js. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, and the description does not disclose behavioral traits such as whether deletion is permanent, what happens if the RID does not exist, or any side effects. The description is too brief to add meaningful context beyond the tool's basic function.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, concise sentence with no wasted words. It is front-loaded and efficiently communicates the core purpose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (2 params, no output schema), the description is mostly adequate. It could be improved by noting that deletion is irreversible and by hinting at sibling tools for bulk operations, but it covers the essential functionality.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 100% coverage with descriptions for both parameters (rid and use_js_token). The description adds no extra meaning beyond the schema; it only mentions deletion by RID, ignoring the use_js_token parameter. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action (Delete), resource (a single item from Crawlbase Cloud Storage), and method (by RID). It effectively distinguishes from the sibling tool storage_bulk_delete, which handles bulk deletions.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for deleting a single item by RID but does not explicitly state when to use this tool versus alternatives like storage_bulk_delete or when not to use it. No prerequisites or context are provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
storage_getA
Retrieve a single stored page from Crawlbase Cloud Storage by RID or URL. Only works for pages previously crawled with store=true. Returns the raw JSON record by default; pass as=html to return just the body, or as=markdown to convert it.
| Name | Required | Description | Default |
|---|---|---|---|
| as | No | How to render the body. Default: json (full record). | |
| rid | No | Storage request ID. Either rid or url must be provided. | |
| url | No | Original crawled URL. Either rid or url must be provided. | |
| use_js_token | No | Query the JS token's storage instead of the normal token's. Storage is per-token. Set this to true if the prior crawl response showed token_type=js (i.e. screenshot or JS-rendered pages); otherwise the lookup will return 'Not found'. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Describes core behavior: retrieval method, precondition, and return format options (as=html returns body, as=markdown converts). No annotations exist, so the description carries the full burden, which it meets for a non-destructive retrieval tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences that efficiently convey purpose, precondition, and key parameter behavior. No redundant information; every word contributes to clarity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Fully covers what the tool does, when it applies, what it returns, and how to modify output format. No output schema exists, but description sufficiently explains return value. No gaps for a simple retrieval tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, but the description adds value beyond the schema by explaining the effect of the 'as' parameter ('return just the body' for html, 'convert it' for markdown). For rid and url, description reinforces schema hints. No extra for use_js_token, but schema already covers it.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Clearly states 'Retrieve a single stored page' with specific verb and resource, specifies identification by RID or URL, and includes precondition (previously crawled with store=true). Distinguishes from sibling tools like storage_list or storage_bulk_get.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states the precondition for use ('Only works for pages previously crawled with store=true'), providing clear context. Does not explicitly mention alternatives or when not to use, but the precondition effectively limits appropriate contexts.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
storage_listA
List RIDs of pages previously crawled with store=true in Crawlbase Cloud Storage. Supports scroll-based pagination (scroll session expires after 15s of inactivity). Returns at most 1000 per call.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Max RIDs to return (≤1000). | |
| scroll | No | Enable scroll pagination. | |
| scroll_id | No | Continuation token from a previous response. | |
| scroll_order | No | Sort direction (default desc). | |
| use_js_token | No | Query the JS token's storage instead of the normal token's. Set to true if the prior crawl response showed token_type=js. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description carries full burden. Discloses scroll session expiry (15s inactivity) and max 1000 per call. Implies read-only nature (listing RIDs). Could mention rate limits or error scenarios, but covers key behaviors.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences: first states purpose, second adds key behavioral constraints (scroll expiry, limit). No redundancy, every word earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema, so description must clarify response format. It mentions listing RIDs and scroll pagination but does not specify whether response includes a scroll_id or other fields. Incomplete for a tool with 5 parameters and no output schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description adds value by mentioning scroll session expiry and max return limit, which are not in parameter descriptions. Provides behavioral context over schema alone.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Clearly states it lists RIDs of stored crawled pages with store=true, using specific verbs and resource naming. Distinct from sibling storage_get (single) and storage_count (count). Includes scroll pagination and limit details.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Describes what the tool does but does not explicitly guide when to use it over alternatives like storage_get or storage_count. Agent must infer usage context from sibling names. Lacks 'when-not-to-use' or explicit differentiation.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
9 tool updates
v1.3.0- First observed
crawl - First observed
crawl_markdown - First observed
crawl_screenshot - First observed
storage_bulk_delete - First observed
storage_bulk_get - First observed
storage_count - First observed
storage_delete - First observed
storage_get - First observed
storage_list
TDQS
Each tool has a clearly distinct purpose: three crawl tools for different output formats (HTML, markdown, screenshot) and six storage tools for managing stored pages (get, list, delete, count, bulk operations). No overlap or ambiguity.
All tool names follow a consistent pattern: 'crawl_*' for crawling actions and 'storage_*' for storage operations. The verbs are clear and the prefix clearly indicates the domain.
9 tools is well-scoped for a web crawling and storage server. It covers crawling with three output formats and full CRUD for storage, without unnecessary bloat or missing essential operations.
The tool surface provides comprehensive coverage: crawling with HTML, markdown, and screenshot output; storage with create (via store=true), read (single, bulk, list), delete (single, bulk), and count. No obvious gaps for the intended purpose.
Maintenance
Resources
Unclaimed servers have limited discoverability.
Looking for Admin?
If you are the server author, to access and configure the admin panel.
Related MCP Connectors
Reliable web access for AI agents: smart HTTP, rotating proxies, and full-browser rendering.
A comprehensive Model Context Protocol (MCP) server that enables AI assistants to interact with yo…
A Model Context Protocol server for Wix AI tools
Hosted browser for AI agents: screenshots, post-JS DOM, console, WCAG. No install, no API key.
61
Related MCP Servers
- FlicenseNot gradedqualityDmaintenanceA server that enables AI systems to browse, retrieve content from, and interact with web pages through the Model Context Protocol.1-

Raccoon AI MCP Serverofficial
FlicenseNot gradedqualityDmaintenanceA Model Context Protocol server that enables web browsing, data extraction, and complex web tasks automation through Raccoon AI's LAM API.-- AlicenseBqualityCmaintenanceA Model Context Protocol server that enables LLMs to interact with web pages, take screenshots, generate test code, scrape web pages, and execute JavaScript in a real browser environment.296222MIT
- AlicenseBqualityCmaintenanceA Model Context Protocol server that enables AI assistants to control a real web browser with stealth capabilities, avoiding bot detection while performing tasks like clicking, filling forms, taking screenshots, and extracting data.115626MIT
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/crawlbase/crawlbase-mcp'
If you have feedback or need assistance with the MCP directory API, please join our Discord server