Skip to main content
Glama

Most web scraping tools give your agent one of two bad outputs:

  • a blocked page, login wall, or empty app shell

  • raw HTML full of nav, scripts, styling, ads, and duplicated boilerplate

webclaw.io is the hosted web extraction API for webclaw. This repo contains the open-source CLI, MCP server, extraction engine, and self-hostable server.

webclaw turns a URL into clean content your tools can actually use.

webclaw https://example.com --format markdown
# Example Domain

This domain is for use in illustrative examples in documents.

You may use this domain in literature without prior coordination or asking for permission.

Use it from the terminal, wire it into Claude/Cursor through MCP, call the hosted API from your app, or self-host the OSS server.


Install

Agent setup

The fastest way to connect webclaw to Claude Code, Claude Desktop, Cursor, Windsurf, OpenCode, Codex CLI, and other MCP-compatible tools:

npx create-webclaw

The installer detects supported clients and configures the MCP server for you.

Homebrew

brew tap 0xMassi/webclaw
brew install webclaw

Prebuilt binaries

Download macOS, Linux, and Windows binaries from GitHub Releases.

Docker

docker run --rm ghcr.io/0xmassi/webclaw https://example.com

Cargo

cargo install --git https://github.com/0xMassi/webclaw.git webclaw-cli
cargo install --git https://github.com/0xMassi/webclaw.git webclaw-mcp

If building from source fails because native build tools are missing, install the platform prerequisites:

OS

Command

Debian / Ubuntu

sudo apt install -y pkg-config libssl-dev cmake clang git build-essential

Fedora / RHEL

sudo dnf install -y pkg-config openssl-devel cmake clang git make gcc

Arch

sudo pacman -S pkg-config openssl cmake clang git base-devel

macOS

xcode-select --install


Related MCP server: read-website-fast

Quick Start

Scrape one page

webclaw https://stripe.com --format markdown

Return LLM-optimized text

webclaw https://docs.anthropic.com --format llm

Keep only the main content

webclaw https://example.com/blog/post --only-main-content

Include or exclude selectors

webclaw https://example.com \
  --include "article, main, .content" \
  --exclude "nav, footer, .sidebar, .ad"

Crawl a documentation site

webclaw https://docs.rust-lang.org --crawl --depth 2 --max-pages 50

Workflow examples

Extract brand assets

webclaw https://github.com --brand

Compare a page over time

webclaw https://example.com/pricing --format json > pricing-old.json
webclaw https://example.com/pricing --diff-with pricing-old.json

MCP Server

webclaw ships with an MCP server for AI agents.

Zero-install — point any MCP client at the npx launcher:

{
  "mcpServers": {
    "webclaw": {
      "command": "npx",
      "args": ["-y", "@webclaw/mcp"]
    }
  }
}

Or run npx create-webclaw to auto-detect your AI tools and write their configs for you.

Then ask your agent things like:

Scrape these competitor pricing pages and summarize the differences.
Crawl this documentation site and prepare clean context for a RAG index.
Extract the brand colors, fonts, and logos from this company website.

Use as an agent skill

Add webclaw to Claude Code, Cursor, Windsurf, and other MCP agents in one command:

npx skills add 0xMassi/webclaw-skill

Your agent gets scrape, crawl, map, extract, summarize, diff, brand, and search as native tools. Most sites extract locally with no API key. Set WEBCLAW_API_KEY to handle bot-protected and JavaScript-rendered pages.

Find it on skills.sh.


Tools

Tool

What it does

Local

scrape

Extract one URL as markdown, text, JSON, LLM format, or HTML

Yes

crawl

Follow same-origin links and extract discovered pages

Yes

map

Discover URLs without extracting every page

Yes

batch

Scrape multiple URLs in parallel

Yes

extract

Convert page content into structured data

Yes, with local or configured LLM

summarize

Summarize a page

Yes, with local or configured LLM

diff

Compare page content snapshots

Yes

brand

Extract colors, fonts, logos, and metadata

Yes

search

Search the web and scrape results

Hosted API

research

Multi-source research workflow

Hosted API


SDKs

npm install @webclaw/sdk
pip install webclaw
go get github.com/0xMassi/webclaw-go
import { Webclaw } from "@webclaw/sdk";

const client = new Webclaw({ apiKey: process.env.WEBCLAW_API_KEY! });

const page = await client.scrape({
  url: "https://example.com",
  formats: ["markdown"],
  only_main_content: true,
});

console.log(page.markdown);
from webclaw import Webclaw

client = Webclaw(api_key="wc_your_key")

page = client.scrape(
    "https://example.com",
    formats=["markdown"],
    only_main_content=True,
)

print(page.markdown)
curl -X POST https://api.webclaw.io/v1/scrape \
  -H "Authorization: Bearer $WEBCLAW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "formats": ["markdown"],
    "only_main_content": true
  }'

Output Formats

Format

Use it when you need

markdown

Clean page content with structure preserved

llm

Compact context for agents and RAG pipelines

text

Plain text with minimal formatting

json

Structured metadata, links, images, and extracted fields

html

Cleaned HTML for custom processing


Local First, Hosted When Needed

The CLI and MCP server work locally without an account for the core extraction path.

Use the hosted API at webclaw.io when you need:

  • protected-site access without managing infrastructure

  • JavaScript rendering

  • async crawl and research jobs

  • web search

  • watches and production usage tracking

  • SDKs for application code

export WEBCLAW_API_KEY=wc_your_key

webclaw https://example.com --cloud

What You Can Build

Use case

Example

AI agent web access

Give Claude, Cursor, or another MCP client clean page context

RAG ingestion

Crawl docs, help centers, blogs, and knowledge bases

Competitor monitoring

Track pricing pages, changelogs, docs, and product pages

Structured extraction

Turn messy pages into typed JSON for automations

Research workflows

Search, scrape, summarize, and cite multiple sources

Brand intelligence

Extract logos, colors, fonts, and social metadata

Architecture

webclaw/
  crates/
    webclaw-core     HTML to markdown, text, JSON, and LLM-ready output
    webclaw-fetch    Fetching, crawling, batching, and mapping
    webclaw-llm      Local and hosted LLM provider support
    webclaw-pdf      PDF text extraction
    webclaw-mcp      MCP server for AI agents
    webclaw-cli      Command-line interface

webclaw-core is pure extraction logic: no network I/O, small surface area, and usable independently from the fetching layer.


Configuration

Variable

Description

WEBCLAW_API_KEY

Hosted API key

OLLAMA_HOST

Ollama URL for local LLM features

OPENAI_API_KEY

OpenAI-compatible LLM provider key

OPENAI_BASE_URL

OpenAI-compatible base URL

ANTHROPIC_API_KEY

Anthropic-compatible LLM provider key

ANTHROPIC_BASE_URL

Anthropic-compatible base URL

ORCAROUTER_API_KEY

OrcaRouter LLM provider key

ORCAROUTER_BASE_URL

OrcaRouter base URL (defaults to https://api.orcarouter.ai/v1)

WEBCLAW_PROXY

Single proxy URL

WEBCLAW_PROXY_FILE

Proxy pool file


Contributing

The most useful contributions right now are practical and small:

  • add examples for real agent and RAG workflows

  • improve SDK snippets

  • report pages that extract poorly

  • add failing fixtures for messy HTML

  • improve docs for MCP clients and local setup

  • test the CLI on more Linux/macOS environments

Good first places to start:

If a page extracts badly, include:

URL:
Command or API request:
Expected output:
Actual output:
Format used: markdown / llm / text / json / html
CLI, MCP, SDK, or API:

Please remove secrets, cookies, private tokens, and customer data from logs before posting.


Strategic Partner


Infrastructure Partner


Studio Partners


Community Plugins

Third-party plugins that integrate webclaw with AI agent platforms:

Plugin

Platform

What it does

openclaw-webclaw

OpenClaw

Native webclaw v1 API plugin with 9 tools: scrape, search, crawl, extract, summarize, diff, map, batch, brand

hermes-webclaw

Hermes Agent

Web search provider and 9 dedicated tools for the full v1 API surface. Install with hermes plugins install jal-co/hermes-webclaw

Built a webclaw integration? Open a PR to add it here.


Contributors

Thanks to everyone improving webclaw through issues, examples, docs, bug reports, and pull requests.


Star History


License

AGPL-3.0

Available Tools

10 tools
batchB

Extract content from multiple URLs concurrently.

ParametersJSON Schema
NameRequiredDescriptionDefault
concurrencyNoNumber of concurrent requests (default: 5)
formatNoOutput format: "markdown" (default), "llm", "text"
urlsYesList of URLs to extract content from

TDQS

B3.3/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must carry full behavioral disclosure. It successfully mentions 'concurrently' indicating parallel execution, but lacks critical safety information for a network tool: error handling behavior (what happens if one URL fails?), timeout behavior, rate limiting, or whether results preserve input ordering.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Extremely efficient at 6 words. Front-loaded with action verb ('Extract') and scope ('multiple URLs concurrently'). Zero redundancy or filler content. Every word earns its place without requiring editing.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With 100% input schema coverage, the description need not elaborate on parameters. However, lacking both annotations and output schema, the description should disclose error behavior or return structure for this concurrent network operation. It is minimal but adequate given the straightforward nature of the operation and good schema coverage.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% (concurrency, format, urls all documented), establishing baseline 3. The description implies the 'urls' parameter through 'multiple URLs' but adds no semantic detail beyond what the schema already provides for parameters (no syntax examples, no clarification of 'llm' format value, no default explanations).

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses specific verb 'Extract' and resource 'content from multiple URLs', and includes 'concurrently' which hints at the batch nature. However, it does not explicitly distinguish this from the sibling 'extract' tool (likely single-URL) or 'crawl' (which may also handle multiple URLs), leaving some ambiguity about when to choose this specific tool.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives like 'extract', 'scrape', or 'crawl'. It does not mention prerequisites, rate limiting considerations, or appropriate use cases for concurrent processing versus sequential operations.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

brandA

Extract brand identity (colors, fonts, logo, favicon) from a website's HTML and CSS. Automatically falls back to the webclaw cloud API when bot protection is detected.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesURL to extract brand identity from

TDQS

A3.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full disclosure burden. It adds valuable behavioral context about the automatic fallback to webclaw cloud API when bot protection is detected, which is critical for dependency understanding. However, it omits other important behavioral details like rate limits, authentication requirements, or what happens when extraction partially fails.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two efficiently structured sentences with zero waste. The first front-loads the core purpose with specific deliverables, while the second adds critical operational context (fallback behavior). Every word earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a single-parameter tool with full schema coverage and no output schema, the description adequately covers the primary unusual behavior (external API fallback). It could be improved by briefly mentioning the expected return structure or noting that this is a read-only operation, but the essential functional and behavioral context is present.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Despite 100% schema coverage (baseline 3), the description adds meaningful semantic context by specifying extraction occurs from 'HTML and CSS', implying the URL should point to a styled webpage rather than arbitrary resources. This adds clarity beyond the schema's basic 'URL to extract brand identity from'.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the specific extraction target (colors, fonts, logo, favicon) using the verb 'Extract', making the scope well-defined. However, it lacks explicit differentiation from the sibling 'extract' tool, which could cause confusion about when to choose 'brand' over general extraction.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides implicit guidance by specifying the domain (brand identity extraction) and mentions automatic fallback behavior when bot protection is detected. However, it lacks explicit 'when to use' guidance contrasting with siblings like 'extract', 'scrape', or 'crawl'.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

crawlB

Crawl a website starting from a seed URL, following links breadth-first up to a configurable depth and page limit.

ParametersJSON Schema
NameRequiredDescriptionDefault
concurrencyNoNumber of concurrent requests (default: 5)
depthNoMaximum link depth to follow (default: 2)
formatNoOutput format for each page: "markdown" (default), "llm", "text"
max_pagesNoMaximum number of pages to crawl (default: 50)
urlYesSeed URL to start crawling from
use_sitemapNoSeed the frontier from sitemap discovery before crawling

TDQS

B3.3/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It reveals the algorithm (breadth-first) and constraints (depth/page limits), but omits critical operational details: that it makes external HTTP requests, potential rate-limiting concerns, concurrency behavior implications, and what data structure it returns.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence with zero waste. It front-loads the operation ('Crawl a website') and logically flows from seed URL through traversal method to constraints.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (6 parameters, network I/O, potentially long-running) and the absence of annotations or output schema, the description is minimally adequate but omits important context. It fails to describe return values, error conditions, or external side effects that an agent would need to know.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, establishing a baseline of 3. The description mentions 'configurable depth and page limit' and 'seed URL', which reference parameters, but adds no additional semantic context, syntax guidance, or usage patterns beyond what the schema already documents.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description provides a clear specific verb ('Crawl') and resource ('website'), and explains the core mechanism ('following links breadth-first'). However, it does not explicitly differentiate from sibling tools like 'scrape' or 'map' that might overlap in functionality.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description offers no guidance on when to use 'crawl' versus siblings such as 'scrape', 'extract', or 'map'. There are no exclusion criteria or stated prerequisites for selection.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

diffA

Compare the current content of a URL against a previous extraction snapshot, showing what changed. Automatically falls back to the webclaw cloud API when bot protection is detected.

ParametersJSON Schema
NameRequiredDescriptionDefault
previous_snapshotYesPrevious extraction snapshot as a JSON string (ExtractionResult)
urlYesURL to fetch current content from

TDQS

A3.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure and successfully reveals the automatic fallback to 'webclaw cloud API when bot protection is detected.' However, it omits other critical behavioral traits such as whether the operation is read-only, idempotent, or what error conditions might occur beyond bot protection.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description consists of exactly two sentences with zero wasted words. The first sentence front-loads the core functionality, while the second provides relevant operational context regarding the fallback mechanism, maintaining tight focus throughout.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's moderate complexity (comparison logic with fallback) and the absence of an output schema, the description adequately covers the input requirements and operational model. However, it would benefit from explicitly stating the prerequisite relationship to the 'extract' sibling and describing the expected diff output format.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has 100% description coverage for both parameters ('url' and 'previous_snapshot'), establishing a baseline of 3. The description references these parameters in context ('current content of a URL,' 'previous extraction snapshot') but does not add additional semantic details beyond what the schema already provides, such as validation rules or the expected format of the ExtractionResult.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ('Compare') and clearly identifies both resources involved ('current content of a URL' vs 'previous extraction snapshot'), along with the output ('showing what changed'). It effectively distinguishes this from siblings like 'extract' or 'scrape' by specifying the comparison paradigm.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The mention of 'previous extraction snapshot' implicitly signals that this tool should be used after an extraction has been performed, providing context for the prerequisite workflow. However, it lacks explicit guidance on when to choose this over siblings like 'crawl' or 'scrape' for monitoring changes, or what conditions make the fallback behavior trigger.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

extractA

Extract structured data from a web page using an LLM. Provide either a JSON schema or a natural language prompt. Automatically falls back to the webclaw cloud API when bot protection is detected.

ParametersJSON Schema
NameRequiredDescriptionDefault
promptNoNatural language prompt describing what to extract
schemaNoJSON schema describing the structure to extract
urlYesURL to fetch and extract structured data from

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full transparency burden. It successfully discloses the automatic fallback to 'webclaw cloud API when bot protection is detected,' which is substantive behavioral context. However, it omits rate limits, auth requirements, and output format details.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three tightly constructed sentences: purpose declaration, input guidance, and fallback behavior. Every sentence earns its place with no redundancy or filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With 9 sibling tools sharing similar domains ('scrape', 'crawl', 'summarize'), the description lacks explicit differentiation to aid tool selection. While the core functionality is covered, the absence of output schema disclosure and sibling comparisons leaves gaps for agent decision-making.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, establishing a baseline of 3. The description adds value by clarifying the relationship between 'schema' and 'prompt' parameters ('either... or'), indicating they are alternative specification methods—a semantic constraint not explicit in the schema alone.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Specific verb ('Extract') and resource ('structured data from a web page') are clear, plus method ('using an LLM'). However, it does not explicitly differentiate from siblings like 'scrape' (raw HTML) or 'crawl' (multiple pages), only implying the distinction via the LLM mention.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Implies usage by stating input options ('Provide either a JSON schema or a natural language prompt'), but lacks explicit when-to-use guidance versus alternatives like 'scrape' or 'summarize', and does not state prerequisites or exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

mapA

Discover URLs from a website's sitemaps (robots.txt + sitemap.xml).

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesBase URL to discover sitemaps from (e.g. `<https://example.com>`)

TDQS

A3.8/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so description carries full burden. It adds valuable behavioral context by specifying it checks both 'robots.txt' and 'sitemap.xml'. However, missing critical behavioral details: whether it follows sitemap index files, rate limiting, output format, or error handling when sitemaps are absent.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Extremely efficient at 9 words. Front-loaded with the action 'Discover URLs' and immediately specifies the source mechanism. Every word earns its place with zero redundancy or boilerplate.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Appropriately complete for a single-parameter discovery tool. The description covers the core function and specific data sources. Minor gap: does not describe the return value (list of URLs), though this is somewhat implied by 'Discover URLs'.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Input schema has 100% description coverage with the 'url' parameter well-documented as 'Base URL to discover sitemaps from'. The description does not add additional semantics about the parameter (e.g., protocol requirements, trailing slashes), warranting the baseline score for high-coverage schemas.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Specific verb 'Discover' with clear resource 'URLs' and distinct mechanism 'sitemaps (robots.txt + sitemap.xml)'. This effectively distinguishes the tool from siblings like 'crawl' or 'scrape' by specifying it extracts URLs from sitemap files rather than page content.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Implies usage through the specific mechanism mentioned (sitemaps), suggesting use when architectural URL discovery is needed. However, lacks explicit guidance on when to prefer this over 'crawl' or 'search' siblings, or prerequisites like requiring valid sitemap files.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

researchA

Run a deep research investigation on a topic or question. Requires WEBCLAW_API_KEY. Starts an async research job on the webclaw cloud API, then polls until complete.

ParametersJSON Schema
NameRequiredDescriptionDefault
deepNoEnable deep research mode for more thorough investigation (default: false)
queryYesResearch query or question to investigate
topicNoTopic hint to guide research focus (e.g. "technology", "finance", "science")

TDQS

A3.5/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Discloses async execution model (starts job, polls) and authentication requirement, which is valuable given no annotations. However, missing critical behavioral details: no description of return format (text? JSON? report?), no mention of rate limits, cost implications, or error states for the cloud API operation.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three sentences with zero waste: purpose front-loaded, prerequisites stated, execution model explained. No redundancy with structured fields.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Adequate for basic invocation but gaps remain: no output schema exists yet description doesn't specify return format (critical for a research tool), and lacks guidance on expected duration or result structure given the async polling pattern described.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema has 100% description coverage, establishing baseline 3. Description reinforces parameters by mentioning 'deep research' (deep), 'topic' (topic), and 'question' (query), but adds no syntax details, examples, or constraints beyond what the schema already provides.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clear verb+resource ('Run a deep research investigation') and distinguishes from siblings like 'search' by emphasizing 'deep' investigation and async execution ('polls until complete'). However, could more explicitly contrast with 'search' or 'summarize' siblings.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides prerequisite ('Requires WEBCLAW_API_KEY') and implies usage context through 'deep' and async behavior, but lacks explicit when-to-use guidance versus alternatives like 'search' for quick lookups or 'scrape' for specific extraction.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

scrapeA

Scrape a single URL and extract its content as markdown, LLM-optimized text, plain text, or full JSON. Automatically falls back to the webclaw cloud API when bot protection or JS rendering is detected.

ParametersJSON Schema
NameRequiredDescriptionDefault
browserNoBrowser profile: "chrome" (default), "firefox", or "random"
exclude_selectorsNoCSS selectors to exclude from output
formatNoOutput format: "markdown" (default), "llm", "text", or "json"
include_selectorsNoCSS selectors to include (only extract matching elements)
only_main_contentNoIf true, extract only the main content (article/main element)
urlYesURL to scrape

TDQS

A4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so description carries full disclosure burden. Adds valuable operational context about automatic fallback to webclaw cloud API for bot protection and JS rendering, but omits rate limits, authentication requirements, timeout behavior, and error handling.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences with zero waste. First sentence establishes core functionality and output options; second provides critical implementation detail about fallback behavior. Well-structured and appropriately front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Adequate for a 6-parameter tool with complete schema coverage. Description compensates for missing output schema by detailing return format options, though could enhance with error behavior or rate limit documentation given the web scraping domain.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema has 100% description coverage (baseline 3). Description adds semantic value by clarifying that 'llm' format means 'LLM-optimized text' and elaborating on format intentions beyond the schema's terse descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clear specific verb 'scrape' with scope 'single URL' that explicitly distinguishes from sibling tools like 'batch' and 'crawl'. Lists output formats (markdown, LLM-optimized text, plain text, JSON) to clarify extraction capabilities.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Implicitly differentiates from batch/crawl via 'single URL' phrasing, but lacks explicit when-to-use guidance versus siblings like 'extract' or 'map'. No mention of prerequisites such as URL accessibility or when to prefer local vs cloud fallback.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

summarizeA

Summarize the content of a web page using an LLM. Automatically falls back to the webclaw cloud API when bot protection is detected.

ParametersJSON Schema
NameRequiredDescriptionDefault
max_sentencesNoNumber of sentences in the summary (default: 3)
urlYesURL to fetch and summarize

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so description carries full burden. It effectively discloses two key behaviors: (1) it uses an LLM (not just extraction), and (2) it 'automatically falls back to the webclaw cloud API when bot protection is detected' - crucial resilience behavior. Missing: output format, error handling on failure, or rate limit warnings.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, both essential. First sentence front-loads the core purpose (LLM summarization). Second sentence provides critical operational detail (fallback mechanism). Zero waste, appropriately sized for the tool's complexity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 2-parameter tool with no output schema, the description adequately covers core function and resilience behavior. Minor gap: no mention of return value format (string vs object) or what happens if the URL is unreachable even after fallback.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema has 100% description coverage ('URL to fetch and summarize', 'Number of sentences...'), establishing a baseline of 3. Description reinforces the 'web page' concept aligning with the url parameter, but adds no syntax details, format constraints, or semantic clarification beyond the schema definitions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly states the tool 'Summarize[s] the content of a web page using an LLM' - providing specific verb (summarize), resource (web page content), and method (LLM). This distinguishes it from siblings like 'scrape' (raw extraction), 'extract' (structured data), and 'crawl' (multi-page), which don't imply LLM-based condensation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Description implies usage by mentioning LLM-based summarization, but provides no explicit guidance on when to choose this over siblings like 'scrape' or 'extract'. The fallback behavior hints at resilience but doesn't state prerequisites or exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. 10 tool updatesv0.1.0
    • First observedbatch
    • First observedbrand
    • First observedcrawl
    • First observeddiff
    • First observedextract
    • First observedmap
    • First observedresearch
    • First observedscrape
    • First observedsearch
    • First observedsummarize

TDQS

A3.8/5.0
Disambiguation4/5

Most tools have distinct purposes, but there is some potential overlap between 'extract' (structured data via LLM) and 'scrape' (content extraction in various formats), which could cause confusion. Other tools like 'brand', 'diff', 'map', and 'summarize' are clearly specialized, helping to minimize misselection.

Naming Consistency5/5

All tool names follow a consistent pattern of single, descriptive verbs (e.g., batch, brand, crawl, diff, extract, map, research, scrape, search, summarize). There are no mixed conventions or deviations, making the set predictable and easy to navigate.

Tool Count5/5

With 10 tools, the count is well-scoped for a web scraping and research server. Each tool appears to serve a specific function in the domain, from basic scraping to advanced research, without feeling excessive or insufficient for the intended purpose.

Completeness4/5

The toolset covers a broad range of web-related operations, including extraction, crawling, searching, and analysis. Minor gaps exist, such as the lack of explicit update or delete operations for stored data, but these are not critical for the server's primary focus on content retrieval and processing.

Maintenance

ActivityActive
ResponsivenessResponsive

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

  • A
    license
    A
    quality
    C
    maintenance
    A Model Context Protocol (MCP) server implementation that integrates with FireCrawl for advanced web scraping capabilities.
    26
    40,139
    7,395
    MIT
  • A
    license
    A
    quality
    B
    maintenance
    Fast, token-efficient web content extraction tool that converts websites to clean Markdown for AI agents, featuring smart caching, content extraction with Mozilla Readability, and polite crawling capabilities.
    1
    534
    161
    MIT
  • A
    license
    B
    quality
    C
    maintenance
    🚀 Active Fork of executeautomation/mcp-playwright This repository is an actively maintained continuation of the original MCP Playwright server: >👉 https://github.com/executeautomation/mcp-playwright A Model Context Protocol server that provides browser automation capabilities using Playwright.
    6
    32
    18,122
    1
    MIT
  • A
    license
    Not graded
    quality
    B
    maintenance
    The web data platform for AI agents. Fetch, search, crawl, extract, monitor, and screenshot any URL. 55+ domain extractors, 65-98% token savings. 7 MCP tools included.
    332
    12
    AGPL 3.0

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/0xMassi/webclaw'

If you have feedback or need assistance with the MCP directory API, please join our Discord server