Skip to main content
Glama


What's New

v0.4.0 (2026-04-22)

  • feat: add crw-browse — interactive browser automation MCP server over CDP

  • feat: add SOCKS5 proxy support in crw-renderer

  • feat: extract crw-mcp-proto crate — shared JSON-RPC 2.0 types

Full changelog →


fastCRW — Open Source Web Scraping API for AI Agents

Power AI agents with clean web data. Single Rust binary, zero config, Firecrawl-compatible API. The open-source Firecrawl alternative you can self-host for free — or use our managed cloud.

Don't want to self-host? Sign up free → — managed cloud with global proxy network, web search, and dashboard. Same API, zero infra. 500 free credits, no credit card required.


Related MCP server: webpeel

Why CRW? — Firecrawl & Crawl4AI Alternative

  • Single binary, 6 MB RAM — no Redis, no Node.js, no containers. Firecrawl needs 5 containers and 4 GB+. Crawl4AI requires Python + Playwright

  • 5.5x faster than Firecrawl — 833ms avg vs 4,600ms (see benchmarks). P50 at 446ms

  • 73/100 search win rate — beats Firecrawl (25/100) and Tavily (2/100) in head-to-head benchmarks

  • Free self-hosting — $0/1K scrapes vs Firecrawl's $0.83–5.33. No infra, no cold starts (85ms). No API key required for local mode

  • Agent ready — add to any MCP client in one command. Embedded mode: no server needed

  • Firecrawl-compatible API — drop-in replacement. Same /v1/scrape, /v1/crawl, /v1/map endpoints. HTML to markdown, structured data extraction, website crawler — all built-in

  • Built for RAG pipelines — clean LLM-ready markdown output for vector databases and AI data ingestion

  • Open source — AGPL-3.0, developed transparently. Join our community

Metric

CRW (self-hosted)

fastcrw.com (cloud)

Firecrawl

Tavily

Crawl4AI

Coverage (1K URLs)

92.0%

92.0%

77.2%

Avg Scrape Latency

833ms

833ms

4,600ms

Avg Search Latency

880ms

880ms

954ms

2,000ms

Search Win Rate

73/100

73/100

25/100

2/100

Idle RAM

6.6 MB

0 (managed)

~500 MB+

— (cloud)

Cold start

85 ms

0 (always-on)

30–60 s

Self-hosting

Single binary

Multi-container

No

Python + Playwright

Cost / 1K scrapes

$0 (self-hosted)

From $13/mo

$0.83–5.33

$0

License

AGPL-3.0

Managed

AGPL-3.0

Proprietary

Apache-2.0


Web Scraping & Crawling Features

Core

Feature

Description

Scrape

Convert any URL to markdown, HTML, JSON, or links

Crawl

Async BFS website crawler with rate limiting

Map

Discover all URLs on a site instantly

Search

Web search + content scraping (cloud)

More

Feature

Description

LLM Extraction

Send a JSON schema, get validated structured data back

JS Rendering

Auto-detect SPAs, render via LightPanda or Chrome

CLI

Scrape any URL from your terminal — no server needed

MCP Server

Built-in stdio + HTTP transport for any AI agent

Use Cases: RAG pipelines · AI agent web access · content monitoring · data extraction · HTML to markdown conversion · web archiving


Quick Start

# Install:
curl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | CRW_BINARY=crw sh

# Scrape:
crw example.com

# Add to Claude Code (local):
claude mcp add crw -- npx crw-mcp
# Add to Claude Code (cloud — includes web search, 500 free credits at fastcrw.com):
claude mcp add -e CRW_API_URL=https://fastcrw.com/api -e CRW_API_KEY=your-key crw -- npx crw-mcp

Or: pip install crw (Python SDK) · npx crw-mcp (zero install) · brew install us/crw/crw (Homebrew) · All install options →

Scrape

Convert any URL to clean markdown, HTML, or structured JSON.

from crw import CrwClient

client = CrwClient(api_url="https://fastcrw.com/api", api_key="YOUR_API_KEY")  # local: CrwClient()
result = client.scrape("https://example.com")
print(result["markdown"])

Local mode: CrwClient() with no arguments runs a self-contained scraping engine — no server, no API key, no setup. The SDK automatically downloads the crw-mcp binary on first use.

CLI:

crw example.com
crw example.com --format html
crw example.com --js --css 'article'

Self-hosted (crw-server running on :3000):

curl -X POST http://localhost:3000/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

Cloud:

curl -X POST https://fastcrw.com/api/v1/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

Output:

# Example Domain

This domain is for use in illustrative examples in documents.
You may use this domain in literature without prior coordination.

Crawl

Scrape all pages of a website asynchronously.

from crw import CrwClient

client = CrwClient(api_url="https://fastcrw.com/api", api_key="YOUR_API_KEY")  # local: CrwClient()
pages = client.crawl("https://docs.example.com", max_depth=2, max_pages=50)
for page in pages:
    print(page["metadata"]["sourceURL"], page["markdown"][:80])
# Start crawl
curl -X POST http://localhost:3000/v1/crawl \
  -H "Content-Type: application/json" \
  -d '{"url": "https://docs.example.com", "maxDepth": 2, "maxPages": 50}'

# Check status (use job ID from above)
curl http://localhost:3000/v1/crawl/JOB_ID

Map

Discover all URLs on a site instantly.

from crw import CrwClient

client = CrwClient(api_url="https://fastcrw.com/api", api_key="YOUR_API_KEY")  # local: CrwClient()
urls = client.map("https://example.com")
print(urls)  # ["https://example.com", "https://example.com/about", ...]
curl -X POST http://localhost:3000/v1/map \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

Search the web and get full page content from results.

from crw import CrwClient

# Cloud only — requires fastcrw.com API key
client = CrwClient(api_url="https://fastcrw.com/api", api_key="YOUR_KEY")
results = client.search("open source web scraper 2026", limit=10)

Cloud only: search() requires a fastcrw.com API key (500 free credits, no credit card). Local/embedded mode provides scrape, crawl, and map.

curl -X POST https://fastcrw.com/api/v1/search \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query": "open source web scraper 2026", "limit": 10}'

API Endpoints

Method

Endpoint

Description

POST

/v1/scrape

Scrape a single URL, optionally with LLM extraction

POST

/v1/crawl

Start async BFS crawl (returns job ID)

GET

/v1/crawl/:id

Check crawl status and retrieve results

DELETE

/v1/crawl/:id

Cancel a running crawl job

POST

/v1/map

Discover all URLs on a site

POST

/v1/search

Web search with optional content scraping (cloud only)

GET

/health

Health check (no auth required)

POST

/mcp

Streamable HTTP MCP transport

Full API reference →


Connect to AI Agents — MCP, Skill, Onboarding

Add CRW to any AI agent or MCP client in seconds.

Skill

Install the CRW skill to all detected agents with one command:

npx crw-mcp init --all

Restart your agent after installing. Works with Claude Code, Cursor, Gemini CLI, Codex, OpenCode, and Windsurf.

MCP Server for AI Agents

Add CRW to any MCP-compatible client:

{
  "mcpServers": {
    "crw": {
      "command": "npx",
      "args": ["crw-mcp"]
    }
  }
}

Works with Claude Desktop, Cursor, Windsurf, Cline, Continue.dev, and any MCP client.

Config file locations: Claude Code — claude mcp add (no file edit). Claude Desktop — ~/Library/Application Support/Claude/claude_desktop_config.json. Cursor — .cursor/mcp.json. Windsurf — ~/.codeium/windsurf/mcp_config.json. All clients →

Cloud mode — adds crw_search for web search:

{
  "mcpServers": {
    "crw": {
      "command": "npx",
      "args": ["crw-mcp"],
      "env": {
        "CRW_API_URL": "https://fastcrw.com/api",
        "CRW_API_KEY": "your-api-key"
      }
    }
  }
}

MCP Tools

Mode

Tool

Description

Embedded + Cloud

crw_scrape

Scrape a URL → markdown, HTML, JSON, links

Embedded + Cloud

crw_crawl

Start async BFS crawl (returns job ID)

Embedded + Cloud

crw_check_crawl_status

Poll crawl job status (use with crw_crawl)

Embedded + Cloud

crw_map

Discover all URLs on a site

Cloud only

crw_search

Web search + optional content scraping

Embedded mode (default): no server, no API key, no setup — the MCP binary runs a self-contained scraping engine. Cloud mode (CRW_API_URL set): forwards calls to fastcrw.com, adds crw_search.

Full MCP docs →

Claude Code shortcut:

# Local (embedded — no server, no API key):
claude mcp add crw -- npx crw-mcp

# Cloud (+ web search):
claude mcp add -e CRW_API_URL=https://fastcrw.com/api -e CRW_API_KEY=your-key crw -- npx crw-mcp

Listed on the MCP Registry

Agent Onboarding

Are you an AI agent? Fetch this skill file to configure CRW for the user's coding agents. The SKILL.md contains tool descriptions, usage examples, authentication setup, and edge case handling:

curl -s https://fastcrw.com/agent-onboarding/SKILL.md

Benchmark

Search — CRW vs Firecrawl vs Tavily (100 queries, concurrent)

Metric

CRW

Firecrawl

Tavily

Avg Latency

880ms

954ms

2,000ms

Median Latency

785ms

932ms

1,724ms

Win Rate

73/100

25/100

2/100

CRW is 2.3x faster than Tavily and won 73% of latency races. Full search benchmark →

Scrape — CRW vs Firecrawl (1,000 URLs, JS rendering enabled)

Tested on Firecrawl's scrape-content-dataset-v1:

Metric

CRW

Firecrawl v2.5

Coverage

92.0%

77.2%

Avg Latency

833ms

4,600ms

P50 Latency

446ms

Noise Rejection

88.4%

noise 6.8%

Idle RAM

6.6 MB

~500 MB+

Cost / 1K scrapes

$0 (self-hosted)

$0.83–5.33

Metric

CRW

Firecrawl

Min RAM

~7 MB

4 GB

Recommended RAM

~64 MB (under load)

8–16 GB

Docker images

single ~8 MB binary

~2–3 GB total

Cold start

85 ms

30–60 seconds

Containers needed

1 (+optional sidecar)

5

Full benchmark details →

Run the benchmark yourself:

pip install datasets aiohttp
python bench/run_bench.py

Install

MCP Server (crw-mcp) — recommended for AI agents

npx crw-mcp                           # zero install (npm)
pip install crw                        # Python SDK (auto-downloads binary)
brew install us/crw/crw-mcp            # Homebrew
cargo install crw-mcp                  # Cargo
docker run -i ghcr.io/us/crw crw-mcp  # Docker

CLI (crw) — scrape URLs from your terminal

brew install us/crw/crw

# One-line install (auto-detects OS & arch):
curl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | CRW_BINARY=crw sh

# APT (Debian/Ubuntu):
curl -fsSL https://apt.fastcrw.com/gpg.key | sudo gpg --dearmor -o /usr/share/keyrings/crw.gpg
echo "deb [signed-by=/usr/share/keyrings/crw.gpg] https://apt.fastcrw.com stable main" | sudo tee /etc/apt/sources.list.d/crw.list
sudo apt update && sudo apt install crw

cargo install crw-cli

API Server (crw-server) — Firecrawl-compatible REST API

For serving multiple apps, other languages (Node.js, Go, Java), or as a shared microservice.

brew install us/crw/crw-server

# One-line install:
curl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | CRW_BINARY=crw-server sh

# Docker:
docker run -p 3000:3000 ghcr.io/us/crw

Custom port:

CRW_SERVER__PORT=8080 crw-server                                       # env var
docker run -p 8080:8080 -e CRW_SERVER__PORT=8080 ghcr.io/us/crw       # Docker

When do you need crw-server? Only if you want a REST API endpoint. The Python SDK (CrwClient()) and MCP binary (crw-mcp) both run a self-contained engine — no server required.


SDKs

Python

pip install crw
from crw import CrwClient

# Cloud (fastcrw.com — includes web search):
client = CrwClient(api_url="https://fastcrw.com/api", api_key="YOUR_API_KEY")
# Local (embedded, no server needed):
# client = CrwClient()

# Scrape
result = client.scrape("https://example.com", formats=["markdown", "links"])
print(result["markdown"])

# Crawl (blocks until complete)
pages = client.crawl("https://docs.example.com", max_depth=2, max_pages=50)

# Map
urls = client.map("https://example.com")

# Search (cloud only)
results = client.search("AI news", limit=10, sources=["web", "news"])

Requires: Python 3.9+. Local mode auto-downloads the crw-mcp binary on first use — no manual setup.

Community SDKs

Node.js: No official SDK yet — use the REST API directly or npx crw-mcp for MCP. SDK examples →


Integrations

Frameworks: CrewAI · LangChain · Agno · Dify

Platforms: n8n · Flowise

Missing your favorite tool? Open an issue → · All integrations →


LLM Structured Extraction

Send a JSON schema, get validated structured data back using LLM function calling. Full extraction docs →

curl -X POST http://localhost:3000/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/product",
    "formats": ["json"],
    "jsonSchema": {
      "type": "object",
      "properties": {
        "name": { "type": "string" },
        "price": { "type": "number" }
      },
      "required": ["name", "price"]
    }
  }'

Configure the LLM provider:

[extraction.llm]
provider = "anthropic"        # "anthropic" or "openai"
api_key = "sk-..."            # or CRW_EXTRACTION__LLM__API_KEY env var
model = "claude-sonnet-4-20250514"

JS Rendering

CRW auto-detects SPAs and renders them via a headless browser. Full JS rendering docs →

crw-server setup   # downloads LightPanda, creates config.local.toml

Renderer

Protocol

Best for

LightPanda

CDP over WebSocket

Low-resource environments (default); simple sites

Chrome

CDP over WebSocket

Modern React/Vite/Next SPAs; recommended for production

Playwright

CDP over WebSocket

Full browser compatibility

Renderer choice matters for SPAs. LightPanda is fast and cheap but its JS runtime does not fully cover every modern bundle format. For React / Vite / Next sites whose content appears only after hydration, configure Chrome (or Playwright) alongside LightPanda — CRW will fall back to Chrome automatically when LightPanda returns a loading placeholder. Leaving LightPanda as the only renderer may silently return "Loading..."-style shell content for these sites.

With Docker Compose, LightPanda runs as a sidecar automatically:

docker compose up

CLI

Scrape any URL from your terminal — no server, no config. Full CLI docs →

crw example.com                        # markdown to stdout
crw example.com --format html          # HTML output
crw example.com --format links         # extract all links
crw example.com --js                   # with JS rendering
crw example.com --css 'article'        # CSS selector
crw example.com --stealth              # stealth mode (rotate UAs)
crw example.com -o page.md             # write to file

Self-Hosting

Once installed, start the server and optionally enable JS rendering:

crw-server                    # start REST API on :3000
crw-server setup              # optional: downloads LightPanda for JS rendering
docker compose up             # alternative: Docker with LightPanda sidecar

See the self-hosting guide for production hardening, auth, reverse proxy, and resource tuning.


Open Source vs Cloud

Self-hosted (free)

fastcrw.com Cloud

Core scraping

JS rendering

✅ (LightPanda/Chrome)

Web search

Global proxy network

Dashboard

Commercial use without open-sourcing

Requires AGPL compliance

✅ Included

Cost

$0

From $13/mo

Sign up free →500 free credits, no credit card required.


Architecture

┌─────────────────────────────────────────────┐
│                 crw-server                  │
│         Axum HTTP API + Auth + MCP          │
├──────────┬──────────┬───────────────────────┤
│ crw-crawl│crw-extract│    crw-renderer      │
│ BFS crawl│ HTML→MD   │  HTTP + CDP(WS)      │
│ robots   │ LLM/JSON  │  LightPanda/Chrome   │
│ sitemap  │ clean/read│  auto-detect SPA     │
├──────────┴──────────┴───────────────────────┤
│                 crw-core                    │
│        Types, Config, Errors                │
└─────────────────────────────────────────────┘

Crate

Description

crw-core

Core types, config, and error handling

crates.io

crw-renderer

HTTP + CDP browser rendering engine

crates.io

crw-extract

HTML → markdown/plaintext extraction

crates.io

crw-crawl

Async BFS crawler with robots.txt & sitemap

crates.io

crw-server

Axum API server (Firecrawl-compatible)

crates.io

crw-mcp

MCP stdio server (embedded + proxy mode)

crates.io

crw-cli

Standalone CLI (crw binary, no server)

crates.io

Full architecture docs →


Configuration

Layered TOML config with environment variable overrides:

  1. config.default.toml — built-in defaults

  2. config.local.toml — local overrides (or CRW_CONFIG=myconfig)

  3. Environment variables — CRW_ prefix, __ separator (e.g. CRW_SERVER__PORT=8080)

[server]
host = "0.0.0.0"
port = 3000
rate_limit_rps = 10

[renderer]
mode = "auto"  # auto | lightpanda | playwright | chrome | none

[crawler]
max_concurrency = 10
requests_per_second = 10.0
respect_robots_txt = true

[auth]
# api_keys = ["fc-key-1234"]

See full configuration reference.


Security

  • SSRF protection — blocks loopback, private IPs, cloud metadata (169.254.x.x), IPv6 mapped addresses, and non-HTTP schemes (file://, data:)

  • Auth — optional Bearer token with constant-time comparison

  • robots.txt — RFC 9309 compliant with wildcard patterns

  • Rate limiting — token-bucket algorithm, returns 429 with error_code

  • Resource limits — max body 1 MB, max crawl depth 10, max pages 1000

Full security docs →


Resources


Contributing

Contributions are welcome! Please open an issue or submit a pull request.

  1. Fork the repository

  2. Install pre-commit hooks: make hooks

  3. Create your feature branch (git checkout -b feat/my-feature)

  4. Commit your changes (git commit -m 'feat: add my feature')

  5. Push to the branch (git push origin feat/my-feature)

  6. Open a Pull Request

The pre-commit hook runs the same checks as CI (cargo fmt, cargo clippy, cargo test). Run manually with make check.

Contributors


License

CRW is open-source under AGPL-3.0. For a managed version without AGPL obligations, see fastcrw.com.


Get Started

  • Self-host free: curl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | sh — works in 30 seconds

  • Cloud: Sign up free →500 free credits, no credit card required

  • Questions? Join our Discord


It is the sole responsibility of end users to respect websites' policies when scraping. Users are advised to adhere to applicable privacy policies and terms of use. By default, CRW respects robots.txt directives.

Available Tools

8 tools
crw_cancel_extractCancel extract jobA
DestructiveIdempotent
Inspect

Request cancellation of an extract job. Returns the canonical status; cancelling remains non-terminal until the claimed URL settles.

ParametersJSON Schema
NameRequiredDescriptionDefault
idYesExtract job id from crw_extract

Output Schema

ParametersJSON Schema
NameRequiredDescription
idYes
errorNo
statusYes
resultsYes
successYes
expiresAtYes
tokensUsedYes
creditsUsedNo

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Despite annotations already indicating destructive and non-read-only behavior, the description adds valuable context: cancellation is non-terminal until the claimed URL settles, and it returns canonical status. This goes beyond the structured annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tightly worded sentences. The first states the action; the second adds a critical caveat. No redundant information; front-loaded with purpose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a cancellation tool, the description covers purpose and the important async behavior. An output schema exists, so return values need not be described. Slightly more guidance on subsequent steps (e.g., checking status) could help, but it is not essential.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, with the single 'id' parameter documented as 'Extract job id from crw_extract.' The description text itself adds no parameter details, so it relies entirely on the schema, which is adequate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states the action: 'Request cancellation of an extract job.' The verb 'cancel' and resource 'extract job' are specific, and it is distinct from sibling tools like crw_extract (create) and crw_check_extract_status (status check).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies use when you want to cancel an extract job, but it does not explicitly contrast with alternatives or mention when not to use it. No sibling tool is referenced, so guidance is implied rather than explicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

crw_check_crawl_statusCheck crawl statusA
Read-onlyIdempotent
Inspect

Poll an async crawl job and retrieve its pages.

ParametersJSON Schema
NameRequiredDescriptionDefault
idYesCrawl job id from crw_crawl
maxLengthNoMax chars per page content field; 0 = unbounded (default ~15000)

TDQS

A4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint and idempotentHint, so the description adds only moderate behavioral context by confirming it polls and retrieves pages. The description does not disclose any additional traits beyond the annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, concise sentence that is front-loaded and contains no unnecessary words. It efficiently conveys the tool's purpose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple polling tool with two parameters and no output schema, the description adequately explains the action (poll and retrieve pages). It could hint at the return format, but 'retrieve its pages' is sufficient for an agent.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, with clear parameter descriptions in the schema. The tool description adds no additional meaning beyond what is already in the schema, meeting the baseline for high coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool polls an async crawl job and retrieves its pages, providing a specific verb and resource. It distinguishes from siblings like crw_crawl (start) and crw_scrape (synchronous scrape).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage context by mentioning 'poll an async crawl job', indicating it's for checking ongoing crawls. However, it does not explicitly state when not to use or name alternatives, which is a minor gap.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

crw_check_extract_statusCheck extract job statusA
Read-onlyIdempotent
Inspect

Poll an extract job; returns status and, when complete, a per-URL results array.

ParametersJSON Schema
NameRequiredDescriptionDefault
idYesExtract job id from crw_extract

Output Schema

ParametersJSON Schema
NameRequiredDescription
idYes
errorNo
statusYes
resultsYes
successYes
expiresAtYes
tokensUsedYes
creditsUsedNo

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The annotations already indicate a safe, idempotent read operation. The description adds the behavioral detail that the tool returns status immediately and, upon completion, includes a per-URL results array, which is useful for understanding polling behavior. No contradiction with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, front-loaded sentence that directly states the action and outcome with no extraneous text. It earns a top score.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given that an output schema exists and the tool has a single parameter, the description adequately covers the tool's purpose, return behavior, and usage context. It is complete for the tool's complexity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema already defines 'id' as the extract job id from crw_extract, and the description does not add any additional parameter semantics. With 100% schema coverage, the baseline of 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses the specific verb 'Poll' and identifies the resource as 'an extract job,' clearly distinguishing it from the similar sibling 'crw_check_crawl_status' by specifying 'extract' and the per-URL results array. It states exactly what the tool does and its output.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It clearly implies usage when checking the status of a previously submitted extract job, providing clear context. However, it does not explicitly mention alternatives or when not to use this tool, such as for crawl jobs, so it lacks explicit exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

crw_crawlCrawl siteAInspect

Start an async site crawl; returns a job id to poll with crw_check_crawl_status.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesStarting URL
waitForNoMs to wait after JS render per page
maxDepthNoMax crawl depth (default 2)
maxPagesNoMax pages to crawl (default 10)
renderJsNoForce JS render (true), HTTP-only (false), omit = auto
rendererNoPin renderer; non-auto hard-pins and implies renderJs:true (default auto). 'camoufox' requires the server's opt-in camoufox tier to be configured.
jsonSchemaNoOptional. A JSON Schema (draft 2020-12) describing fields to extract from each page via an LLM, e.g. {"type":"object","properties":{"title":{"type":"string"}}}. Free-form object. Omit to crawl without structured extraction.

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations declare readOnlyHint=false, destructiveHint=false, idempotentHint=false, openWorldHint=true. The description adds the important behavioral trait that the crawl is async and returns a job id for polling, which informs the agent about the non-immediate nature and required follow-up. This goes beyond what annotations alone provide.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences with no superfluous text. The first sentence states the action and output, the second ties it to the sibling tool for follow-up. Every word earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite having 7 parameters and no output schema, the tool description covers the key context: async nature, job id return, and polling via a specific sibling. It does not explain the crawling scope or error handling, but the input schema fills in parameter details. For a start-crawl tool, this is fairly complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% (all 7 parameters have descriptions). The tool description does not add any additional parameter semantics beyond the schema, so baseline score of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ('Start') and resource ('async site crawl'), clearly states the output (job id), and distinguishes from the sibling tool crw_check_crawl_status by indicating the polling workflow.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context for usage: start a crawl then poll with the sibling tool. It does not explicitly list when to avoid using it or compare with other siblings like crw_map or crw_scrape, but the implication is strong enough for a straightforward tool.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

crw_extractExtract structured dataAInspect

Extract structured JSON from URLs via a prompt and/or JSON schema. Async job — poll crw_check_extract_status with the returned id. Needs an LLM.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlsYesURLs to extract from
basisNoReturn per-field evidence: each top-level scalar property comes back with a source url, verbatim excerpt and honest status (supported/unverified/unsupported/notFound). Requires schema.
promptNoFree-text extraction objective (required unless schema is given)
schemaNoJSON Schema constraining the extracted output
llmModelNo
llmApiKeyNoBYOK LLM API key
llmProviderNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
idYes
urlsYes
statusYes
successYes

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already indicate readOnlyHint=false and openWorldHint=true. The description adds behavioral nuance by disclosing the async job nature ('Async job — poll crw_check_extract_status with the returned id') and a key prerequisite ('Needs an LLM'). This provides context beyond the annotations, such as the non-blocking execution model and the requirement to track progress via a returned identifier.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is exactly two sentences: the first states the core purpose and method, the second covers the async workflow and prerequisite. Every clause adds value, and the most important information is front-loaded. There is no redundancy or filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 7 parameters, nested objects, and an output schema, the description efficiently covers the essential workflow (async, polling, LLM requirement). The existence of an output schema means return values are documented elsewhere. It does not discuss error handling or rate limits, but these are less critical given the asynchronous pattern and available schema documentation. Overall, it is complete enough for an agent to select and invoke the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema covers 71% of parameters with descriptions, so the baseline is 3. The description adds meaningful semantics by explaining that extraction works 'via a prompt and/or JSON schema', clarifying the relationship between prompt and schema parameters. It also highlights the LLM dependency, tying together llmModel/llmProvider/llmApiKey even though the schema doesn't explicitly state they are required. This goes beyond the schema's bare parameter list.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description begins with 'Extract structured JSON from URLs via a prompt and/or JSON schema', which clearly states the verb ('extract'), the resource ('URLs'), and the output format ('structured JSON'). It also distinguishes from sibling tools like crw_scrape (raw scraping) and crw_map by emphasizing structured extraction. The async note differentiates it from synchronous tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides context that it is an async job and must be polled via crw_check_extract_status, and that it needs an LLM. However, it does not explicitly state when to use this tool over alternatives such as crw_scrape or crw_crawl, nor any exclusions. The usage context is clear but lacks explicit when-to-use/when-not-to-use guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

crw_mapMap site URLsA
Read-onlyIdempotent
Inspect

Discover URLs on a site via sitemap and/or a short crawl. Returns a URL list only, no page content.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesURL to map
limitNoMax URLs to discover AND return; 0 = unbounded (default 100). Raise it (e.g. 50000) to pull deep/large sitemaps.
maxDepthNoMax discovery depth (default 2)
useSitemapNoUse sitemap.xml (default true)
crawlFallbackNoSupplement sitemap with a short BFS crawl (default true; false = sitemap-only)

TDQS

A3.8/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already provide readOnly, idempotent, and non-destructive hints. The description adds that output is URL list only, consistent with annotations, but no extra behavioral details beyond annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two concise, front-loaded sentences with no wasted words; every part adds value.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity and rich schema+annotations, the description covers the main action and output. Minor gap: interaction of sitemap and crawl fallback is explained in schema, so description is complete enough.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema has 100% parameter coverage with clear descriptions (e.g., limit 0 = unbounded, defaults). The description adds no further parameter meaning beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool discovers URLs on a site via sitemap/crawl and explicitly says it returns only URLs, no content, distinguishing it from sibling crw_scrape and implying it's different from crw_crawl.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for URL discovery without content, but does not explicitly contrast with siblings like crw_crawl or provide when-to-use/not-use guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

crw_parse_fileParse PDFA
Read-onlyIdempotent
Inspect

Parse a local PDF (base64 in contentBase64) to markdown. No OCR: scanned PDFs return empty markdown with a warning.

ParametersJSON Schema
NameRequiredDescriptionDefault
formatsNoOutput formats (default ["markdown"]); json/summary need a server LLM
parsersNoParsers to apply (default ["pdf"])
filenameNoOriginal filename (optional)
maxLengthNoMax chars per content field; 0 = unbounded (default ~15000)
jsonSchemaNoOptional. A JSON Schema (draft 2020-12) describing fields to extract when formats includes "json", e.g. {"type":"object","properties":{"title":{"type":"string"}}}. Free-form object.
contentBase64YesBase64-encoded PDF bytes

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The annotations already declare readOnlyHint=true, idempotentHint=true, destructiveHint=false, so the safety profile is clear. The description adds the behavioral trait that no OCR is performed and scanned PDFs return empty markdown with a warning, which is valuable context beyond annotations. It also clarifies that the input is base64-encoded local PDF bytes, but that's already in schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is one sentence, front-loaded with the main action ('Parse a local PDF...to markdown') and then a brief caveat. No wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool has 6 parameters and no output schema, the description covers the core functionality and a key limitation. However, it doesn't describe the response structure or that other output formats (json, summary) require a server LLM, though that's in the schema. The description is adequate for a simple parse tool but leaves some gap regarding output shape.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema provides descriptions for all 6 parameters (100% coverage), so the description doesn't need to add much. The description does reference contentBase64 and the markdown output, implicitly mapping to formats, but it doesn't explain the formats, jsonSchema, or maxLength parameters – though those are well-documented in the schema. Thus, the description adds minimal additional parameter semantics.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool parses a local PDF (base64 in contentBase64) to markdown, which is a specific verb with resource and output format. It distinguishes itself from sibling tools by specifying 'local PDF' rather than URLs, aligning with crw_scrape/crw_crawl. The 'No OCR' caveat further defines its scope.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context for when to use it: when you have a local PDF as base64 and want markdown. It explicitly states a when-not scenario: scanned PDFs return empty markdown with a warning, which tells the agent to avoid using it for those. However, it doesn't name alternative tools for OCR or other formats, so it falls just short of explicit alternatives guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

crw_scrapeScrape URLA
Read-onlyIdempotent
Inspect

Scrape one URL to markdown, HTML, or links.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesURL to scrape
formatsNoOutput formats (default ["markdown"])
waitForNoMs to wait after JS render for late content
renderJsNoForce JS render (true), HTTP-only (false), omit = auto
rendererNoPin renderer; non-auto hard-pins and implies renderJs:true (default auto). 'camoufox' requires the server's opt-in camoufox tier to be configured.
maxLengthNoMax chars per content field; 0 = unbounded (default ~15000)
excludeTagsNoCSS selectors to exclude
includeTagsNoCSS selectors to include
onlyMainContentNoStrip nav/footer; main content only (default true)

TDQS

A3.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so the agent knows this is a safe, non-destructive operation. The description adds minimal behavioral context beyond the 'one URL' scope, but does not disclose any additional behavior such as JS rendering defaults, pagination, or output limits. It does not contradict the annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single 9-word sentence that is direct and front-loaded. Every word contributes to the core purpose. It is extremely concise without being verbose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (9 parameters, 4 output formats, multiple renderer options), the description is quite thin. The rich schema compensates for parameter details, but the description fails to mention the 'images' format and provides no context about renderer behavior or when to use optional parameters. For a simple scraper, it may be sufficient, but for an AI agent selecting among multiple similar tools, more context would be helpful.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema covers 100% of parameters with descriptions, so the baseline is 3. The description mentions markdown, HTML, and links, but omits 'images' from the formats enum, which is a minor gap. The description does not add meaning beyond the schema for the remaining parameters.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action (scrape), the resource (one URL), and the output formats (markdown, HTML, or links). The phrase 'one URL' distinguishes it from siblings like crw_crawl, which implies multi-page crawling. It is specific and unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies this tool is for scraping a single URL, since it says 'one URL'. However, it does not explicitly mention when to prefer this over crw_crawl, crw_map, or crw_extract, nor does it list any exclusions or prerequisites. Usage is implied rather than stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. 5 tool updatesv0.30.0
    • Addedcrw_cancel_extract
    • Changedcrw_check_extract_status1 field changed
      • changedOutput schema / (root)
        Previous value: -nullNew value: +{
        +  "additionalProperties": false,
        +  "properties": {
        +    "creditsUsed": {
        +      "type": "integer"
        +    },
        +    "error": {
        +      "type": "string"
        +    },
        +    "expiresAt": {
        +      "format": "date-time",
        +      "type": "string"
        +    },
        +    "id": {
        +      "type": "string"
        +    },
        +    "results": {
        +      "items": {
        +        "additionalProperties": false,
        +        "properties": {
        +          "basis": {
        +            "items": {
        +              "type": "object"
        +            },
        +            "type": "array"
        +          },
        +          "basisWarnings": {
        +            "items": {
        +              "type": "object"
        +            },
        +            "type": "array"
        +          },
        +          "data": {
        +            "additionalProperties": true,
        +            "type": "object"
        +          },
        +          "error": {
        +            "type": "string"
        +          },
        +          "llmInputHash": {
        +            "type": "string"
        +          },
        +          "llmUsage": {
        +            "type": "object"
        +          },
        +          "status": {
        +            "enum": [
        +              "processing",
        +              "completed",
        +              "failed",
        +              "cancelled"
        +            ],
        +            "type": "string"
        +          },
        +          "url": {
        +            "type": "string"
        +          }
        +        },
        +        "required": [
        +          "url",
        +          "status"
        +        ],
        +        "type": "object"
        +      },
        +      "type": "array"
        +    },
        +    "status": {
        +      "enum": [
        +        "processing",
        +        "cancelling",
        +        "completed",
        +        "failed",
        +        "cancelled"
        +      ],
        +      "type": "string"
        +    },
        +    "success": {
        +      "type": "boolean"
        +    },
        +    "tokensUsed": {
        +      "type": "integer"
        +    }
        +  },
        +  "required": [
        +    "success",
        +    "id",
        +    "status",
        +    "results",
        +    "expiresAt",
        +    "tokensUsed"
        +  ],
        +  "type": "object"
        +}
    • Changedcrw_extract1 field changed
      • changedOutput schema / (root)
        Previous value: -nullNew value: +{
        +  "additionalProperties": false,
        +  "properties": {
        +    "id": {
        +      "type": "string"
        +    },
        +    "status": {
        +      "enum": [
        +        "processing"
        +      ],
        +      "type": "string"
        +    },
        +    "success": {
        +      "type": "boolean"
        +    },
        +    "urls": {
        +      "minimum": 0,
        +      "type": "integer"
        +    }
        +  },
        +  "required": [
        +    "success",
        +    "id",
        +    "status",
        +    "urls"
        +  ],
        +  "type": "object"
        +}
    • Changedcrw_parse_file1 field changed
      • changedInput schema / properties / formats / items / enum
        Previous value: -[
        -  "markdown",
        -  "plainText",
        -  "links",
        -  "json",
        -  "summary"
        -]New value: +[
        +  "markdown",
        +  "plainText",
        +  "links",
        +  "images",
        +  "json",
        +  "summary"
        +]
    • Changedcrw_scrape1 field changed
      • changedInput schema / properties / formats / items / enum
        Previous value: -[
        -  "markdown",
        -  "html",
        -  "links"
        -]New value: +[
        +  "markdown",
        +  "html",
        +  "links",
        +  "images"
        +]
  2. 1 tool updatev0.24.1
    • Changedcrw_extract1 field changed
      • addedInput schema / properties / basis
        Added value: +{
        +  "description": "Return per-field evidence: each top-level scalar property comes back with a source url, verbatim excerpt and honest status (supported/unverified/unsupported/notFound). Requires schema.",
        +  "type": "boolean"
        +}
  3. 2 tool updatesv0.22.0
    • Addedcrw_check_extract_status
    • Addedcrw_extract
  4. 1 tool updatev1.0.1
    • Changedcrw_map1 field changed
      • changedInput schema / properties / limit / description
        Previous value: -"Max URLs returned; 0 = unbounded (default 100)"New value: +"Max URLs to discover AND return; 0 = unbounded (default 100). Raise it (e.g. 50000) to pull deep/large sitemaps."
  5. 3 tool updatesv0.18.0
    • Changedcrw_crawl4 fields changed
      • addedInput schema / properties / jsonSchema / additionalProperties
        Added value: +true
      • changedInput schema / properties / jsonSchema / description
        Previous value: -"JSON schema for LLM extraction per page"New value: +"Optional. A JSON Schema (draft 2020-12) describing fields to extract from each page via an LLM, e.g. {\"type\":\"object\",\"properties\":{\"title\":{\"type\":\"string\"}}}. Free-form object. Omit to crawl without structured extraction."
      • changedInput schema / properties / renderer / description
        Previous value: -"Pin renderer; non-auto hard-pins and implies renderJs:true (default auto)"New value: +"Pin renderer; non-auto hard-pins and implies renderJs:true (default auto). 'camoufox' requires the server's opt-in camoufox tier to be configured."
      • changedInput schema / properties / renderer / enum
        Previous value: -[
        -  "auto",
        -  "lightpanda",
        -  "chrome",
        -  "playwright"
        -]New value: +[
        +  "auto",
        +  "lightpanda",
        +  "chrome",
        +  "playwright",
        +  "camoufox"
        +]
    • Changedcrw_parse_file2 fields changed
      • addedInput schema / properties / jsonSchema / additionalProperties
        Added value: +true
      • changedInput schema / properties / jsonSchema / description
        Previous value: -"JSON schema for LLM extraction (when formats has json)"New value: +"Optional. A JSON Schema (draft 2020-12) describing fields to extract when formats includes \"json\", e.g. {\"type\":\"object\",\"properties\":{\"title\":{\"type\":\"string\"}}}. Free-form object."
    • Changedcrw_scrape2 fields changed
      • changedInput schema / properties / renderer / description
        Previous value: -"Pin renderer; non-auto hard-pins and implies renderJs:true (default auto)"New value: +"Pin renderer; non-auto hard-pins and implies renderJs:true (default auto). 'camoufox' requires the server's opt-in camoufox tier to be configured."
      • changedInput schema / properties / renderer / enum
        Previous value: -[
        -  "auto",
        -  "lightpanda",
        -  "chrome",
        -  "playwright"
        -]New value: +[
        +  "auto",
        +  "lightpanda",
        +  "chrome",
        +  "playwright",
        +  "camoufox"
        +]
  6. 6 tool updatesv0.16.0
    • Changedcrw_check_crawl_status2 fields changed
      • changedInput schema / properties / id / description
        Previous value: -"The crawl job ID returned by crw_crawl"New value: +"Crawl job id from crw_crawl"
      • addedInput schema / properties / maxLength
        Added value: +{
        +  "description": "Max chars per page content field; 0 = unbounded (default ~15000)",
        +  "minimum": 0,
        +  "type": "integer"
        +}
    • Changedcrw_crawl7 fields changed
      • changedInput schema / properties / jsonSchema / description
        Previous value: -"JSON schema for LLM-based structured data extraction on each crawled page"New value: +"JSON schema for LLM extraction per page"
      • changedInput schema / properties / maxDepth / description
        Previous value: -"Maximum crawl depth (default: 2)"New value: +"Max crawl depth (default 2)"
      • changedInput schema / properties / maxPages / description
        Previous value: -"Maximum number of pages to crawl (default: 10)"New value: +"Max pages to crawl (default 10)"
      • changedInput schema / properties / renderJs / description
        Previous value: -"Render JavaScript on every crawled page (true = force JS, false = HTTP only, omit = auto-detect or use the server's render_js_default)"New value: +"Force JS render (true), HTTP-only (false), omit = auto"
      • changedInput schema / properties / renderer / description
        Previous value: -"Pin every crawled page to a specific renderer. \"auto\" (default if omitted) uses the configured fallback chain. Other values hard-pin with no fallback. Pinning a non-auto value implies renderJs:true unless renderJs:false is set explicitly."New value: +"Pin renderer; non-auto hard-pins and implies renderJs:true (default auto)"
      • changedInput schema / properties / url / description
        Previous value: -"The starting URL to crawl"New value: +"Starting URL"
      • changedInput schema / properties / waitFor / description
        Previous value: -"Milliseconds to wait after JS rendering on each page"New value: +"Ms to wait after JS render per page"
    • Changedcrw_map5 fields changed
      • changedInput schema / properties / crawlFallback / description
        Previous value: -"If true (default), supplements sitemap discovery with a short BFS crawl when the sitemap returns enough URLs. Set false for sitemap-only mode (faster, may miss pages not in the sitemap)."New value: +"Supplement sitemap with a short BFS crawl (default true; false = sitemap-only)"
      • addedInput schema / properties / limit
        Added value: +{
        +  "description": "Max URLs returned; 0 = unbounded (default 100)",
        +  "minimum": 0,
        +  "type": "integer"
        +}
      • changedInput schema / properties / maxDepth / description
        Previous value: -"Maximum crawl depth for discovery (default: 2)"New value: +"Max discovery depth (default 2)"
      • changedInput schema / properties / url / description
        Previous value: -"The URL to map"New value: +"URL to map"
      • changedInput schema / properties / useSitemap / description
        Previous value: -"Whether to use the site's sitemap.xml (default: true)"New value: +"Use sitemap.xml (default true)"
    • Changedcrw_parse_file6 fields changed
      • changedInput schema / properties / contentBase64 / description
        Previous value: -"Base64-encoded bytes of the PDF file"New value: +"Base64-encoded PDF bytes"
      • changedInput schema / properties / filename / description
        Previous value: -"Original filename (optional; echoed in metadata.sourceFilename)"New value: +"Original filename (optional)"
      • changedInput schema / properties / formats / description
        Previous value: -"Output formats (default: [\"markdown\"]). json/summary require a server LLM."New value: +"Output formats (default [\"markdown\"]); json/summary need a server LLM"
      • changedInput schema / properties / jsonSchema / description
        Previous value: -"JSON schema for LLM-based structured extraction (when formats includes json)"New value: +"JSON schema for LLM extraction (when formats has json)"
      • addedInput schema / properties / maxLength
        Added value: +{
        +  "description": "Max chars per content field; 0 = unbounded (default ~15000)",
        +  "minimum": 0,
        +  "type": "integer"
        +}
      • changedInput schema / properties / parsers / description
        Previous value: -"Document parsers to apply (default: [\"pdf\"])"New value: +"Parsers to apply (default [\"pdf\"])"
    • Changedcrw_scrape9 fields changed
      • changedInput schema / properties / excludeTags / description
        Previous value: -"CSS selectors to exclude from output"New value: +"CSS selectors to exclude"
      • changedInput schema / properties / formats / description
        Previous value: -"Output formats (default: [\"markdown\"])"New value: +"Output formats (default [\"markdown\"])"
      • changedInput schema / properties / includeTags / description
        Previous value: -"CSS selectors to include (only content matching these selectors)"New value: +"CSS selectors to include"
      • addedInput schema / properties / maxLength
        Added value: +{
        +  "description": "Max chars per content field; 0 = unbounded (default ~15000)",
        +  "minimum": 0,
        +  "type": "integer"
        +}
      • changedInput schema / properties / onlyMainContent / description
        Previous value: -"Extract only the main content, removing nav/footer/etc (default: true)"New value: +"Strip nav/footer; main content only (default true)"
      • changedInput schema / properties / renderJs / description
        Previous value: -"Render JavaScript before extracting (true = force JS, false = HTTP only, omit = auto-detect or use the server's render_js_default)"New value: +"Force JS render (true), HTTP-only (false), omit = auto"
      • changedInput schema / properties / renderer / description
        Previous value: -"Pin this request to a specific renderer. \"auto\" (default if omitted) uses the configured fallback chain. Other values hard-pin to a single renderer with no fallback. Pinning a non-auto value implies renderJs:true unless renderJs:false is set explicitly."New value: +"Pin renderer; non-auto hard-pins and implies renderJs:true (default auto)"
      • changedInput schema / properties / url / description
        Previous value: -"The URL to scrape"New value: +"URL to scrape"
      • changedInput schema / properties / waitFor / description
        Previous value: -"Milliseconds to wait after JS rendering for late content/XHRs"New value: +"Ms to wait after JS render for late content"
    • Removedcrw_search
  7. 2 tool updatesv0.15.2
    • Addedcrw_parse_file
    • Changedcrw_search4 fields changed
      • changedInput schema / properties / categories / description
        Previous value: -"Bias the search towards a category. `pdf` appends `filetype:pdf` to the query; `github`/`research` switch to topical engines."New value: +"Bias the search towards a category. Curated values: `pdf` appends `filetype:pdf` to the query; `github`/`research` switch to topical engines. Any other value (e.g. `science`, `it`, `news`, `files`) is passed straight through to SearXNG's native `categories` routing."
      • removedInput schema / properties / categories / items / enum
        Removed value: -[
        -  "github",
        -  "research",
        -  "pdf"
        -]
      • addedInput schema / properties / country
        Added value: +{
        +  "description": "Country code for results (e.g. \"us\", \"tr\"). Hint to bias regional results; ignored if the underlying engine does not support it.",
        +  "type": "string"
        +}
      • changedOutput schema / (root)
        Previous value: -nullNew value: +{
        +  "$defs": {
        +    "searchResultItem": {
        +      "properties": {
        +        "category": {
        +          "type": "string"
        +        },
        +        "description": {
        +          "description": "Body snippet for the result. `snippet` is an alias of this field.",
        +          "type": "string"
        +        },
        +        "position": {
        +          "type": "integer"
        +        },
        +        "score": {
        +          "type": "number"
        +        },
        +        "snippet": {
        +          "description": "Alias of `description`. Always populated.",
        +          "type": "string"
        +        },
        +        "title": {
        +          "type": "string"
        +        },
        +        "url": {
        +          "type": "string"
        +        }
        +      },
        +      "required": [
        +        "url",
        +        "title",
        +        "description",
        +        "snippet",
        +        "position"
        +      ],
        +      "type": "object"
        +    }
        +  },
        +  "properties": {
        +    "data": {
        +      "properties": {
        +        "answer": {
        +          "type": "string"
        +        },
        +        "citations": {
        +          "type": "array"
        +        },
        +        "llmUsage": {
        +          "type": "object"
        +        },
        +        "results": {
        +          "oneOf": [
        +            {
        +              "items": {
        +                "$ref": "#/$defs/searchResultItem"
        +              },
        +              "type": "array"
        +            },
        +            {
        +              "properties": {
        +                "images": {
        +                  "type": "array"
        +                },
        +                "news": {
        +                  "items": {
        +                    "$ref": "#/$defs/searchResultItem"
        +                  },
        +                  "type": "array"
        +                },
        +                "web": {
        +                  "items": {
        +                    "$ref": "#/$defs/searchResultItem"
        +                  },
        +                  "type": "array"
        +                }
        +              },
        +              "type": "object"
        +            }
        +          ]
        +        },
        +        "warnings": {
        +          "items": {
        +            "type": "string"
        +          },
        +          "type": "array"
        +        }
        +      },
        +      "required": [
        +        "results"
        +      ],
        +      "type": "object"
        +    },
        +    "error": {
        +      "type": "string"
        +    },
        +    "error_code": {
        +      "type": "string"
        +    },
        +    "success": {
        +      "type": "boolean"
        +    },
        +    "warning": {
        +      "type": "string"
        +    }
        +  },
        +  "required": [
        +    "success",
        +    "data"
        +  ],
        +  "type": "object"
        +}
  8. 5 tool updatesv0.9.1
    • Addedcrw_check_crawl_status
    • Addedcrw_crawl
    • Addedcrw_map
    • Addedcrw_scrape
    • Addedcrw_search
  9. 5 tool updatesv0.8.3
    • Removedcrw_check_crawl_status
    • Removedcrw_crawl
    • Removedcrw_map
    • Removedcrw_scrape
    • Removedcrw_search

TDQS

A4.2/5.0
Disambiguation5/5

Each tool has a clearly distinct purpose: extract, scrape, crawl, map, parse file, and status checks are all differentiated by target resource and action. The async job polling and cancellation tools are specific to their respective job types, leaving no ambiguity.

Naming Consistency5/5

All tools follow a consistent `crw_` prefix with snake_case and a verb_noun pattern (e.g., `crw_scrape`, `crw_check_extract_status`, `crw_parse_file`). The naming is predictable and uniform.

Tool Count5/5

With 8 tools, the server is well-scoped for a web scraping/crawling toolkit. Each tool serves a distinct function, and the count fits comfortably within the ideal 3-15 range.

Completeness4/5

The toolkit covers core workflows: single-page scraping, site crawling, URL discovery, structured extraction, PDF parsing, and async job management. However, there is no cancellation for crawl jobs (asymmetric with extract cancellation), which is a minor gap in the crawl lifecycle.

Maintenance

ActivityActive
ResponsivenessResponsive

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

  • A
    license
    Not graded
    quality
    D
    maintenance
    Web scraping MCP server for Al agents. 6 tools: extract clean text/markdown from any URL, structured scraping with CSS selectors, full-page screenshots via Playwright, link extraction with regex filtering, metadata extraction (OG tags, Twitter cards), and Google search. Free tier: 50 requests/IP/day.
    8
    MIT
  • A
    license
    Not graded
    quality
    B
    maintenance
    The web data platform for AI agents. Fetch, search, crawl, extract, monitor, and screenshot any URL. 55+ domain extractors, 65-98% token savings. 7 MCP tools included.
    332
    12
    AGPL 3.0
  • A
    license
    A
    quality
    A
    maintenance
    Web content extraction for AI agents. 10 tools: scrape, crawl, map, batch, extract, summarize, diff, brand, search, research. Uses TLS fingerprinting to bypass anti-bot without a headless browser. Outputs LLM-optimized markdown with 67% fewer tokens than raw HTML.
    10
    2,316
    AGPL 3.0
  • A
    license
    A
    quality
    A
    maintenance
    Enables LLM agents to read any website by scraping and crawling into clean Markdown, automatically bypassing bot detection with residential proxies.
    3
    21
    MIT

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/us/crw'

If you have feedback or need assistance with the MCP directory API, please join our Discord server