webclaw
The webclaw server lets you extract, crawl, and analyze web content for AI agents, RAG pipelines, and developer workflows. Here's what you can do:
Scrape – Extract content from a single URL as markdown, LLM-optimized text, plain text, HTML, or JSON; supports CSS selector filtering, main-content extraction, and auto-fallback to cloud API when bot protection or JS rendering is detected.
Crawl – Breadth-first crawl of a website from a seed URL, following links up to a configurable depth and page limit, with optional sitemap seeding and concurrent requests.
Batch – Scrape multiple URLs concurrently and return extracted content for all of them at once.
Map – Discover all URLs from a website's sitemaps (via
robots.txt+sitemap.xml) without fully extracting every page.Extract – Extract structured data from a web page using an LLM, guided by a JSON schema or a natural language prompt.
Summarize – Generate a concise LLM-powered summary of a web page, with configurable sentence count.
Diff – Compare a URL's current content against a previously saved extraction snapshot to highlight what has changed.
Brand – Extract brand identity assets (colors, fonts, logo, favicon) from a website's HTML and CSS.
Search – Search the web for a query and return structured results (requires API key).
Research – Run a deep, multi-source research investigation on a topic or question, with an optional deep mode for more thorough results (requires API key).
Integrates with local Ollama instances for private, AI-powered structured data extraction and content summarization.
Utilizes OpenAI's API to perform schema-enforced extraction and content summarization on scraped data.
Provides specialized extraction of structured metadata and content details from YouTube video pages.
Most web scraping tools give your agent one of two bad outputs:
a blocked page, login wall, or empty app shell
raw HTML full of nav, scripts, styling, ads, and duplicated boilerplate
webclaw.io is the hosted web extraction API for webclaw. This repo contains the open-source CLI, MCP server, extraction engine, and self-hostable server.
webclaw turns a URL into clean content your tools can actually use.
webclaw https://example.com --format markdown# Example Domain
This domain is for use in illustrative examples in documents.
You may use this domain in literature without prior coordination or asking for permission.Use it from the terminal, wire it into Claude/Cursor through MCP, call the hosted API from your app, or self-host the OSS server.
Install
Agent setup
The fastest way to connect webclaw to Claude Code, Claude Desktop, Cursor, Windsurf, OpenCode, Codex CLI, and other MCP-compatible tools:
npx create-webclawThe installer detects supported clients and configures the MCP server for you.
Homebrew
brew tap 0xMassi/webclaw
brew install webclawPrebuilt binaries
Download macOS, Linux, and Windows binaries from GitHub Releases.
Docker
docker run --rm ghcr.io/0xmassi/webclaw https://example.comCargo
cargo install --git https://github.com/0xMassi/webclaw.git webclaw-cli
cargo install --git https://github.com/0xMassi/webclaw.git webclaw-mcpIf building from source fails because native build tools are missing, install the platform prerequisites:
OS | Command |
Debian / Ubuntu |
|
Fedora / RHEL |
|
Arch |
|
macOS |
|
Related MCP server: read-website-fast
Quick Start
Scrape one page
webclaw https://stripe.com --format markdownReturn LLM-optimized text
webclaw https://docs.anthropic.com --format llmKeep only the main content
webclaw https://example.com/blog/post --only-main-contentInclude or exclude selectors
webclaw https://example.com \
--include "article, main, .content" \
--exclude "nav, footer, .sidebar, .ad"Crawl a documentation site
webclaw https://docs.rust-lang.org --crawl --depth 2 --max-pages 50Workflow examples
Extract brand assets
webclaw https://github.com --brandCompare a page over time
webclaw https://example.com/pricing --format json > pricing-old.json
webclaw https://example.com/pricing --diff-with pricing-old.jsonMCP Server
webclaw ships with an MCP server for AI agents.
Zero-install — point any MCP client at the npx launcher:
{
"mcpServers": {
"webclaw": {
"command": "npx",
"args": ["-y", "@webclaw/mcp"]
}
}
}Or run npx create-webclaw to auto-detect your AI tools and write their configs for you.
Then ask your agent things like:
Scrape these competitor pricing pages and summarize the differences.Crawl this documentation site and prepare clean context for a RAG index.Extract the brand colors, fonts, and logos from this company website.Use as an agent skill
Add webclaw to Claude Code, Cursor, Windsurf, and other MCP agents in one command:
npx skills add 0xMassi/webclaw-skillYour agent gets scrape, crawl, map, extract, summarize, diff, brand, and search
as native tools. Most sites extract locally with no API key. Set WEBCLAW_API_KEY
to handle bot-protected and JavaScript-rendered pages.
Find it on skills.sh.
Tools
Tool | What it does | Local |
| Extract one URL as markdown, text, JSON, LLM format, or HTML | Yes |
| Follow same-origin links and extract discovered pages | Yes |
| Discover URLs without extracting every page | Yes |
| Scrape multiple URLs in parallel | Yes |
| Convert page content into structured data | Yes, with local or configured LLM |
| Summarize a page | Yes, with local or configured LLM |
| Compare page content snapshots | Yes |
| Extract colors, fonts, logos, and metadata | Yes |
| Search the web and scrape results | Hosted API |
| Multi-source research workflow | Hosted API |
SDKs
npm install @webclaw/sdk
pip install webclaw
go get github.com/0xMassi/webclaw-goimport { Webclaw } from "@webclaw/sdk";
const client = new Webclaw({ apiKey: process.env.WEBCLAW_API_KEY! });
const page = await client.scrape({
url: "https://example.com",
formats: ["markdown"],
only_main_content: true,
});
console.log(page.markdown);from webclaw import Webclaw
client = Webclaw(api_key="wc_your_key")
page = client.scrape(
"https://example.com",
formats=["markdown"],
only_main_content=True,
)
print(page.markdown)curl -X POST https://api.webclaw.io/v1/scrape \
-H "Authorization: Bearer $WEBCLAW_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"formats": ["markdown"],
"only_main_content": true
}'Output Formats
Format | Use it when you need |
| Clean page content with structure preserved |
| Compact context for agents and RAG pipelines |
| Plain text with minimal formatting |
| Structured metadata, links, images, and extracted fields |
| Cleaned HTML for custom processing |
Local First, Hosted When Needed
The CLI and MCP server work locally without an account for the core extraction path.
Use the hosted API at webclaw.io when you need:
protected-site access without managing infrastructure
JavaScript rendering
async crawl and research jobs
web search
watches and production usage tracking
SDKs for application code
export WEBCLAW_API_KEY=wc_your_key
webclaw https://example.com --cloudWhat You Can Build
Use case | Example |
AI agent web access | Give Claude, Cursor, or another MCP client clean page context |
RAG ingestion | Crawl docs, help centers, blogs, and knowledge bases |
Competitor monitoring | Track pricing pages, changelogs, docs, and product pages |
Structured extraction | Turn messy pages into typed JSON for automations |
Research workflows | Search, scrape, summarize, and cite multiple sources |
Brand intelligence | Extract logos, colors, fonts, and social metadata |
Architecture
webclaw/
crates/
webclaw-core HTML to markdown, text, JSON, and LLM-ready output
webclaw-fetch Fetching, crawling, batching, and mapping
webclaw-llm Local and hosted LLM provider support
webclaw-pdf PDF text extraction
webclaw-mcp MCP server for AI agents
webclaw-cli Command-line interfacewebclaw-core is pure extraction logic: no network I/O, small surface area, and usable independently from the fetching layer.
Configuration
Variable | Description |
| Hosted API key |
| Ollama URL for local LLM features |
| OpenAI-compatible LLM provider key |
| OpenAI-compatible base URL |
| Anthropic-compatible LLM provider key |
| Anthropic-compatible base URL |
| OrcaRouter LLM provider key |
| OrcaRouter base URL (defaults to https://api.orcarouter.ai/v1) |
| Single proxy URL |
| Proxy pool file |
Contributing
The most useful contributions right now are practical and small:
add examples for real agent and RAG workflows
improve SDK snippets
report pages that extract poorly
add failing fixtures for messy HTML
improve docs for MCP clients and local setup
test the CLI on more Linux/macOS environments
Good first places to start:
If a page extracts badly, include:
URL:
Command or API request:
Expected output:
Actual output:
Format used: markdown / llm / text / json / html
CLI, MCP, SDK, or API:Please remove secrets, cookies, private tokens, and customer data from logs before posting.
Strategic Partner
Infrastructure Partner
Studio Partners
Community Plugins
Third-party plugins that integrate webclaw with AI agent platforms:
Plugin | Platform | What it does |
Native webclaw v1 API plugin with 9 tools: scrape, search, crawl, extract, summarize, diff, map, batch, brand | ||
Web search provider and 9 dedicated tools for the full v1 API surface. Install with |
Built a webclaw integration? Open a PR to add it here.
Contributors
Thanks to everyone improving webclaw through issues, examples, docs, bug reports, and pull requests.
Star History
License
Available Tools
10 toolsbatchB
Extract content from multiple URLs concurrently.
| Name | Required | Description | Default |
|---|---|---|---|
| concurrency | No | Number of concurrent requests (default: 5) | |
| format | No | Output format: "markdown" (default), "llm", "text" | |
| urls | Yes | List of URLs to extract content from |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must carry full behavioral disclosure. It successfully mentions 'concurrently' indicating parallel execution, but lacks critical safety information for a network tool: error handling behavior (what happens if one URL fails?), timeout behavior, rate limiting, or whether results preserve input ordering.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Extremely efficient at 6 words. Front-loaded with action verb ('Extract') and scope ('multiple URLs concurrently'). Zero redundancy or filler content. Every word earns its place without requiring editing.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With 100% input schema coverage, the description need not elaborate on parameters. However, lacking both annotations and output schema, the description should disclose error behavior or return structure for this concurrent network operation. It is minimal but adequate given the straightforward nature of the operation and good schema coverage.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% (concurrency, format, urls all documented), establishing baseline 3. The description implies the 'urls' parameter through 'multiple URLs' but adds no semantic detail beyond what the schema already provides for parameters (no syntax examples, no clarification of 'llm' format value, no default explanations).
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses specific verb 'Extract' and resource 'content from multiple URLs', and includes 'concurrently' which hints at the batch nature. However, it does not explicitly distinguish this from the sibling 'extract' tool (likely single-URL) or 'crawl' (which may also handle multiple URLs), leaving some ambiguity about when to choose this specific tool.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives like 'extract', 'scrape', or 'crawl'. It does not mention prerequisites, rate limiting considerations, or appropriate use cases for concurrent processing versus sequential operations.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
brandA
Extract brand identity (colors, fonts, logo, favicon) from a website's HTML and CSS. Automatically falls back to the webclaw cloud API when bot protection is detected.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to extract brand identity from |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full disclosure burden. It adds valuable behavioral context about the automatic fallback to webclaw cloud API when bot protection is detected, which is critical for dependency understanding. However, it omits other important behavioral details like rate limits, authentication requirements, or what happens when extraction partially fails.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two efficiently structured sentences with zero waste. The first front-loads the core purpose with specific deliverables, while the second adds critical operational context (fallback behavior). Every word earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a single-parameter tool with full schema coverage and no output schema, the description adequately covers the primary unusual behavior (external API fallback). It could be improved by briefly mentioning the expected return structure or noting that this is a read-only operation, but the essential functional and behavioral context is present.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Despite 100% schema coverage (baseline 3), the description adds meaningful semantic context by specifying extraction occurs from 'HTML and CSS', implying the URL should point to a styled webpage rather than arbitrary resources. This adds clarity beyond the schema's basic 'URL to extract brand identity from'.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the specific extraction target (colors, fonts, logo, favicon) using the verb 'Extract', making the scope well-defined. However, it lacks explicit differentiation from the sibling 'extract' tool, which could cause confusion about when to choose 'brand' over general extraction.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides implicit guidance by specifying the domain (brand identity extraction) and mentions automatic fallback behavior when bot protection is detected. However, it lacks explicit 'when to use' guidance contrasting with siblings like 'extract', 'scrape', or 'crawl'.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crawlB
Crawl a website starting from a seed URL, following links breadth-first up to a configurable depth and page limit.
| Name | Required | Description | Default |
|---|---|---|---|
| concurrency | No | Number of concurrent requests (default: 5) | |
| depth | No | Maximum link depth to follow (default: 2) | |
| format | No | Output format for each page: "markdown" (default), "llm", "text" | |
| max_pages | No | Maximum number of pages to crawl (default: 50) | |
| url | Yes | Seed URL to start crawling from | |
| use_sitemap | No | Seed the frontier from sitemap discovery before crawling |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It reveals the algorithm (breadth-first) and constraints (depth/page limits), but omits critical operational details: that it makes external HTTP requests, potential rate-limiting concerns, concurrency behavior implications, and what data structure it returns.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficient sentence with zero waste. It front-loads the operation ('Crawl a website') and logically flows from seed URL through traversal method to constraints.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (6 parameters, network I/O, potentially long-running) and the absence of annotations or output schema, the description is minimally adequate but omits important context. It fails to describe return values, error conditions, or external side effects that an agent would need to know.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, establishing a baseline of 3. The description mentions 'configurable depth and page limit' and 'seed URL', which reference parameters, but adds no additional semantic context, syntax guidance, or usage patterns beyond what the schema already documents.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description provides a clear specific verb ('Crawl') and resource ('website'), and explains the core mechanism ('following links breadth-first'). However, it does not explicitly differentiate from sibling tools like 'scrape' or 'map' that might overlap in functionality.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description offers no guidance on when to use 'crawl' versus siblings such as 'scrape', 'extract', or 'map'. There are no exclusion criteria or stated prerequisites for selection.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
diffA
Compare the current content of a URL against a previous extraction snapshot, showing what changed. Automatically falls back to the webclaw cloud API when bot protection is detected.
| Name | Required | Description | Default |
|---|---|---|---|
| previous_snapshot | Yes | Previous extraction snapshot as a JSON string (ExtractionResult) | |
| url | Yes | URL to fetch current content from |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure and successfully reveals the automatic fallback to 'webclaw cloud API when bot protection is detected.' However, it omits other critical behavioral traits such as whether the operation is read-only, idempotent, or what error conditions might occur beyond bot protection.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description consists of exactly two sentences with zero wasted words. The first sentence front-loads the core functionality, while the second provides relevant operational context regarding the fallback mechanism, maintaining tight focus throughout.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's moderate complexity (comparison logic with fallback) and the absence of an output schema, the description adequately covers the input requirements and operational model. However, it would benefit from explicitly stating the prerequisite relationship to the 'extract' sibling and describing the expected diff output format.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has 100% description coverage for both parameters ('url' and 'previous_snapshot'), establishing a baseline of 3. The description references these parameters in context ('current content of a URL,' 'previous extraction snapshot') but does not add additional semantic details beyond what the schema already provides, such as validation rules or the expected format of the ExtractionResult.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Compare') and clearly identifies both resources involved ('current content of a URL' vs 'previous extraction snapshot'), along with the output ('showing what changed'). It effectively distinguishes this from siblings like 'extract' or 'scrape' by specifying the comparison paradigm.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The mention of 'previous extraction snapshot' implicitly signals that this tool should be used after an extraction has been performed, providing context for the prerequisite workflow. However, it lacks explicit guidance on when to choose this over siblings like 'crawl' or 'scrape' for monitoring changes, or what conditions make the fallback behavior trigger.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extractA
Extract structured data from a web page using an LLM. Provide either a JSON schema or a natural language prompt. Automatically falls back to the webclaw cloud API when bot protection is detected.
| Name | Required | Description | Default |
|---|---|---|---|
| prompt | No | Natural language prompt describing what to extract | |
| schema | No | JSON schema describing the structure to extract | |
| url | Yes | URL to fetch and extract structured data from |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full transparency burden. It successfully discloses the automatic fallback to 'webclaw cloud API when bot protection is detected,' which is substantive behavioral context. However, it omits rate limits, auth requirements, and output format details.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three tightly constructed sentences: purpose declaration, input guidance, and fallback behavior. Every sentence earns its place with no redundancy or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With 9 sibling tools sharing similar domains ('scrape', 'crawl', 'summarize'), the description lacks explicit differentiation to aid tool selection. While the core functionality is covered, the absence of output schema disclosure and sibling comparisons leaves gaps for agent decision-making.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, establishing a baseline of 3. The description adds value by clarifying the relationship between 'schema' and 'prompt' parameters ('either... or'), indicating they are alternative specification methods—a semantic constraint not explicit in the schema alone.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Specific verb ('Extract') and resource ('structured data from a web page') are clear, plus method ('using an LLM'). However, it does not explicitly differentiate from siblings like 'scrape' (raw HTML) or 'crawl' (multiple pages), only implying the distinction via the LLM mention.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Implies usage by stating input options ('Provide either a JSON schema or a natural language prompt'), but lacks explicit when-to-use guidance versus alternatives like 'scrape' or 'summarize', and does not state prerequisites or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
mapA
Discover URLs from a website's sitemaps (robots.txt + sitemap.xml).
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Base URL to discover sitemaps from (e.g. `<https://example.com>`) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description carries full burden. It adds valuable behavioral context by specifying it checks both 'robots.txt' and 'sitemap.xml'. However, missing critical behavioral details: whether it follows sitemap index files, rate limiting, output format, or error handling when sitemaps are absent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Extremely efficient at 9 words. Front-loaded with the action 'Discover URLs' and immediately specifies the source mechanism. Every word earns its place with zero redundancy or boilerplate.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Appropriately complete for a single-parameter discovery tool. The description covers the core function and specific data sources. Minor gap: does not describe the return value (list of URLs), though this is somewhat implied by 'Discover URLs'.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Input schema has 100% description coverage with the 'url' parameter well-documented as 'Base URL to discover sitemaps from'. The description does not add additional semantics about the parameter (e.g., protocol requirements, trailing slashes), warranting the baseline score for high-coverage schemas.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Specific verb 'Discover' with clear resource 'URLs' and distinct mechanism 'sitemaps (robots.txt + sitemap.xml)'. This effectively distinguishes the tool from siblings like 'crawl' or 'scrape' by specifying it extracts URLs from sitemap files rather than page content.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Implies usage through the specific mechanism mentioned (sitemaps), suggesting use when architectural URL discovery is needed. However, lacks explicit guidance on when to prefer this over 'crawl' or 'search' siblings, or prerequisites like requiring valid sitemap files.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
researchA
Run a deep research investigation on a topic or question. Requires WEBCLAW_API_KEY. Starts an async research job on the webclaw cloud API, then polls until complete.
| Name | Required | Description | Default |
|---|---|---|---|
| deep | No | Enable deep research mode for more thorough investigation (default: false) | |
| query | Yes | Research query or question to investigate | |
| topic | No | Topic hint to guide research focus (e.g. "technology", "finance", "science") |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Discloses async execution model (starts job, polls) and authentication requirement, which is valuable given no annotations. However, missing critical behavioral details: no description of return format (text? JSON? report?), no mention of rate limits, cost implications, or error states for the cloud API operation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences with zero waste: purpose front-loaded, prerequisites stated, execution model explained. No redundancy with structured fields.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Adequate for basic invocation but gaps remain: no output schema exists yet description doesn't specify return format (critical for a research tool), and lacks guidance on expected duration or result structure given the async polling pattern described.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema has 100% description coverage, establishing baseline 3. Description reinforces parameters by mentioning 'deep research' (deep), 'topic' (topic), and 'question' (query), but adds no syntax details, examples, or constraints beyond what the schema already provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Clear verb+resource ('Run a deep research investigation') and distinguishes from siblings like 'search' by emphasizing 'deep' investigation and async execution ('polls until complete'). However, could more explicitly contrast with 'search' or 'summarize' siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides prerequisite ('Requires WEBCLAW_API_KEY') and implies usage context through 'deep' and async behavior, but lacks explicit when-to-use guidance versus alternatives like 'search' for quick lookups or 'scrape' for specific extraction.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scrapeA
Scrape a single URL and extract its content as markdown, LLM-optimized text, plain text, or full JSON. Automatically falls back to the webclaw cloud API when bot protection or JS rendering is detected.
| Name | Required | Description | Default |
|---|---|---|---|
| browser | No | Browser profile: "chrome" (default), "firefox", or "random" | |
| exclude_selectors | No | CSS selectors to exclude from output | |
| format | No | Output format: "markdown" (default), "llm", "text", or "json" | |
| include_selectors | No | CSS selectors to include (only extract matching elements) | |
| only_main_content | No | If true, extract only the main content (article/main element) | |
| url | Yes | URL to scrape |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description carries full disclosure burden. Adds valuable operational context about automatic fallback to webclaw cloud API for bot protection and JS rendering, but omits rate limits, authentication requirements, timeout behavior, and error handling.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with zero waste. First sentence establishes core functionality and output options; second provides critical implementation detail about fallback behavior. Well-structured and appropriately front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Adequate for a 6-parameter tool with complete schema coverage. Description compensates for missing output schema by detailing return format options, though could enhance with error behavior or rate limit documentation given the web scraping domain.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema has 100% description coverage (baseline 3). Description adds semantic value by clarifying that 'llm' format means 'LLM-optimized text' and elaborating on format intentions beyond the schema's terse descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Clear specific verb 'scrape' with scope 'single URL' that explicitly distinguishes from sibling tools like 'batch' and 'crawl'. Lists output formats (markdown, LLM-optimized text, plain text, JSON) to clarify extraction capabilities.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Implicitly differentiates from batch/crawl via 'single URL' phrasing, but lacks explicit when-to-use guidance versus siblings like 'extract' or 'map'. No mention of prerequisites such as URL accessibility or when to prefer local vs cloud fallback.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
searchB
Search the web for a query and return structured results. Requires WEBCLAW_API_KEY.
| Name | Required | Description | Default |
|---|---|---|---|
| num_results | No | Number of results to return (default: 10) | |
| query | Yes | Search query |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description carries full burden. It successfully discloses the WEBCLAW_API_KEY requirement and mentions 'structured results', but lacks safety profile disclosure (read-only vs destructive), rate limits, or specific result format details.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two efficient sentences with no waste. Front-loaded with action ('Search the web'), followed by output description and auth requirement. Every element earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Adequate for a simple 2-parameter tool with no output schema, covering purpose, auth, and return type. However, gaps remain in distinguishing from functionally similar siblings and detailing what 'structured results' specifically means.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% with clear descriptions for both 'query' and 'num_results'. Description adds minimal parameter-specific semantics beyond the schema, but baseline 3 is appropriate given comprehensive schema coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Clear verb ('Search') and resource ('the web') with output format specified ('structured results'). However, given siblings like 'crawl', 'scrape', and 'research', it could better differentiate this web search capability from those related tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides no guidance on when to use this tool versus siblings like 'research', 'crawl', or 'scrape', nor does it mention prerequisites beyond the API key requirement. No 'when-not-to-use' guidance present.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
summarizeA
Summarize the content of a web page using an LLM. Automatically falls back to the webclaw cloud API when bot protection is detected.
| Name | Required | Description | Default |
|---|---|---|---|
| max_sentences | No | Number of sentences in the summary (default: 3) | |
| url | Yes | URL to fetch and summarize |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description carries full burden. It effectively discloses two key behaviors: (1) it uses an LLM (not just extraction), and (2) it 'automatically falls back to the webclaw cloud API when bot protection is detected' - crucial resilience behavior. Missing: output format, error handling on failure, or rate limit warnings.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, both essential. First sentence front-loads the core purpose (LLM summarization). Second sentence provides critical operational detail (fallback mechanism). Zero waste, appropriately sized for the tool's complexity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 2-parameter tool with no output schema, the description adequately covers core function and resilience behavior. Minor gap: no mention of return value format (string vs object) or what happens if the URL is unreachable even after fallback.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema has 100% description coverage ('URL to fetch and summarize', 'Number of sentences...'), establishing a baseline of 3. Description reinforces the 'web page' concept aligning with the url parameter, but adds no syntax details, format constraints, or semantic clarification beyond the schema definitions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states the tool 'Summarize[s] the content of a web page using an LLM' - providing specific verb (summarize), resource (web page content), and method (LLM). This distinguishes it from siblings like 'scrape' (raw extraction), 'extract' (structured data), and 'crawl' (multi-page), which don't imply LLM-based condensation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Description implies usage by mentioning LLM-based summarization, but provides no explicit guidance on when to choose this over siblings like 'scrape' or 'extract'. The fallback behavior hints at resilience but doesn't state prerequisites or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
10 tool updates
v0.1.0- First observed
batch - First observed
brand - First observed
crawl - First observed
diff - First observed
extract - First observed
map - First observed
research - First observed
scrape - First observed
search - First observed
summarize
TDQS
Most tools have distinct purposes, but there is some potential overlap between 'extract' (structured data via LLM) and 'scrape' (content extraction in various formats), which could cause confusion. Other tools like 'brand', 'diff', 'map', and 'summarize' are clearly specialized, helping to minimize misselection.
All tool names follow a consistent pattern of single, descriptive verbs (e.g., batch, brand, crawl, diff, extract, map, research, scrape, search, summarize). There are no mixed conventions or deviations, making the set predictable and easy to navigate.
With 10 tools, the count is well-scoped for a web scraping and research server. Each tool appears to serve a specific function in the domain, from basic scraping to advanced research, without feeling excessive or insufficient for the intended purpose.
The toolset covers a broad range of web-related operations, including extraction, crawling, searching, and analysis. Minor gaps exist, such as the lack of explicit update or delete operations for stored data, but these are not critical for the server's primary focus on content retrieval and processing.
Maintenance
Resources
Unclaimed servers have limited discoverability.
Looking for Admin?
If you are the server author, to access and configure the admin panel.
Related MCP Connectors
Cloud scraping & crawling API for AI agents. Turn any URL into clean, LLM-ready markdown.
Web scraping for AI agents. Converts URLs to clean, LLM-ready Markdown with anti-bot bypass.
Clean Markdown and AI-readability scoring for any URL. Built for AI agents.
11Web scraping for AI agents. Extract text and metadata from any URL worldwide. $0.005/page.
Related MCP Servers
- AlicenseAqualityCmaintenanceA Model Context Protocol (MCP) server implementation that integrates with FireCrawl for advanced web scraping capabilities.2640,1397,395MIT
- AlicenseAqualityBmaintenanceFast, token-efficient web content extraction tool that converts websites to clean Markdown for AI agents, featuring smart caching, content extraction with Mozilla Readability, and polite crawling capabilities.1534161MIT
- AlicenseBqualityCmaintenance🚀 Active Fork of executeautomation/mcp-playwright This repository is an actively maintained continuation of the original MCP Playwright server: >👉 https://github.com/executeautomation/mcp-playwright A Model Context Protocol server that provides browser automation capabilities using Playwright.63218,1221MIT
- AlicenseNot gradedqualityBmaintenanceThe web data platform for AI agents. Fetch, search, crawl, extract, monitor, and screenshot any URL. 55+ domain extractors, 65-98% token savings. 7 MCP tools included.33212AGPL 3.0
Appeared in Searches
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/0xMassi/webclaw'
If you have feedback or need assistance with the MCP directory API, please join our Discord server