Automatically detects and crawls Docusaurus-based documentation sites, extracting core content and converting it to Markdown for LLM ingestion.
Automatically detects and crawls GitBook documentation sites, extracting core content and converting it to Markdown for LLM ingestion.
Automatically detects and crawls Sphinx-based documentation sites, extracting core content and converting it to Markdown for LLM ingestion.
doc-scraper)
A configurable, concurrent, and resumable web crawler written in Go. Specifically designed to scrape technical documentation websites, extract core content, convert it cleanly to Markdown format suitable for ingestion by Large Language Models (LLMs), and save the results locally.
This project provides a powerful command-line tool to crawl documentation sites based on settings defined in a config.yaml file. It navigates the site structure, extracts content from specified HTML sections using CSS selectors, and converts it into clean Markdown files.
Built for LLM Training & RAG Systems - Creates clean, consistent Markdown optimized for ingestion
Preserves Documentation Structure - Maintains the original site hierarchy for context preservation
Production-Ready Features - Offers resumable crawls, rate limiting, and graceful error handling
High Performance - Uses Go's concurrency model for efficient parallel processing
Related MCP server: go-docs-mcp
The main objective of this tool is to automate the often tedious process of gathering and cleaning web-based documentation for use with Large Language Models. By converting structured web content into clean Markdown, it aims to provide a dataset that is:
Text-Focused: Prioritizes the textual content extracted via CSS selectors
Structured: Maintains the directory hierarchy of the original documentation site, preserving context
Cleaned: Converts HTML to Markdown, removing web-specific markup and clutter
Locally Accessible: Provides the content as local files for easier processing and pipeline integration
Feature | Description |
Configurable Crawling | Uses YAML for global and site-specific settings |
Scope Control | Limits crawling by domain, path prefix, and disallowed path patterns (regex) |
Content Extraction | Extracts main content using CSS selectors |
HTML-to-Markdown | Converts extracted HTML to clean GitHub-Flavored Markdown (tables, task lists, strikethrough) |
Image Handling | Opt-in downloading and local rewriting of image links with domain and size filtering (disabled by default; doc-scraper is text-first) |
Link Rewriting | Rewrites internal links to relative paths for local structure |
JSONL Output | Optional one-record-per-page JSONL with a trailing crawl-summary record, for RAG ingestion |
Concurrency | Configurable worker pools and semaphore-based request limits (global and per-host) |
Rate Limiting | Configurable per-host delays with jitter |
Robots.txt & Sitemaps | Respects |
State Persistence | Uses BadgerDB for state; supports resuming crawls via |
Graceful Shutdown | Handles |
HTTP Retries | Exponential backoff with jitter for transient errors |
Observability | Structured logging ( |
Modular Code | Organized into packages for clarity and maintainability |
CLI Utilities | Built-in |
MCP Server Mode | Expose as Model Context Protocol server for Claude Code/Cursor integration |
Full-Text Search | Offline BM25 search over crawled docs (SQLite FTS5) via the |
Auto Content Detection | Automatic framework detection (Docusaurus, MkDocs, Sphinx, GitBook, ReadTheDocs) with readability fallback |
Parallel Site Crawling | Crawl multiple sites concurrently with shared resource management |
Watch Mode | Scheduled periodic re-crawling with state persistence |
Go: Version 1.26 or later
Git: For cloning the repository
Disk Space: Sufficient for storing crawled content and state database
Option 1: Direct Installation (Recommended)
Install the latest version directly from GitHub:
go install github.com/Sriram-PR/doc-scraper/v2/cmd/doc-scraper@latestThis installs the doc-scraper binary to your GOPATH/bin directory (usually ~/go/bin or %USERPROFILE%\go\bin). Make sure this directory is in your PATH.
Option 2: Clone and Build
Clone the repository:
git clone https://github.com/Sriram-PR/doc-scraper.git
cd doc-scraperInstall Dependencies:
go mod tidyBuild the Binary:
make build
# or: go build -o doc-scraper ./cmd/doc-scraperThis creates an executable named doc-scraper in the project root.
Create a minimal config.yaml in the project root:
output_base_dir: "./crawled_docs"
state_dir: "./crawler_state"
enable_jsonl_output: true
sites:
rust_cli_book:
start_urls:
- "https://rust-cli.github.io/book/index.html"
allowed_domain: "rust-cli.github.io"
allowed_path_prefix: "/book/"
content_selector: "#content, main"
max_depth: 2 # seed plus one level; set 0 for the whole bookRun the crawl:
./doc-scraper crawl -site rust_cli_book -loglevel infoThe Markdown, plus pages.jsonl, llms.txt, and llms-full.txt, lands under ./crawled_docs/rust_cli_book/ (output is organized by site key). A small book like this finishes in a few seconds; large sites can take minutes, so start with a low max_depth to gauge size before removing the bound.
config.yaml)A config.yaml file is required to run the crawler. Create this file in the project root or specify its path using the -config flag.
When configuring for LLM documentation processing, pay special attention to these settings:
sites.<your_site_key>.content_selector: Define precisely to capture only relevant text
sites.<your_site_key>.allowed_domain / allowed_path_prefix: Define scope accurately
skip_images: Images are not downloaded by default (text-first). Set to false globally or per-site to download and localize images for offline consumption
Adjust concurrency/delay settings based on the target site and your resources
# Global settings (applied if not overridden by site)
default_delay_per_host: 500ms
num_workers: 8
num_image_workers: 8
max_requests: 48
max_requests_per_host: 4
output_base_dir: "./crawled_docs"
state_dir: "./crawler_state"
max_retries: 4
initial_retry_delay: 1s
max_retry_delay: 30s
global_crawl_timeout: 0s
skip_images: true # Default. Set to false to download and localize images
max_image_size_bytes: 10485760 # 10 MiB (applies only when images are downloaded)
enable_jsonl_output: true
jsonl_output_filename: "pages.jsonl"
# HTTP Client Settings
http_client_settings:
timeout: 45s
max_idle_conns_per_host: 6
# Site-specific configurations
sites:
# Key used with -site flag
pytorch_docs:
start_urls:
- "https://pytorch.org/docs/stable/"
allowed_domain: "pytorch.org"
allowed_path_prefix: "/docs/stable/"
content_selector: "article.pytorch-article .body"
max_depth: 0 # 0 for unlimited depth
skip_images: false # Opt in to downloading images for this site
disallowed_path_patterns:
- "/docs/stable/.*/_modules/.*"
- "/docs/stable/.*\.html#.*"
tensorflow_docs:
start_urls:
- "https://www.tensorflow.org/guide"
- "https://www.tensorflow.org/tutorials"
allowed_domain: "www.tensorflow.org"
allowed_path_prefix: "/"
content_selector: ".devsite-article-body"
max_depth: 0
delay_per_host: 1s # Site-specific override
# Disable JSONL output for this site, overriding global
enable_jsonl_output: false
disallowed_path_patterns:
- "/install/.*"
- "/js/.*"Option | Type | Description | Default |
| String | Default User-Agent header for requests |
|
| Duration | Time to wait between requests to the same host |
|
| Integer | Number of concurrent crawl workers |
|
| Integer | Number of concurrent image download workers | same as |
| Integer | Maximum concurrent requests (global) |
|
| Integer | Maximum concurrent requests per host |
|
| String | Base directory for crawled content |
|
| String | Directory for BadgerDB state data |
|
| Integer | Maximum retry attempts for HTTP requests. To disable retries, set this to |
|
| Duration | Initial delay for retry backoff |
|
| Duration | Maximum delay for retry backoff |
|
| Duration | Overall timeout for the entire crawl |
|
| Duration | Timeout for processing a single page |
|
| Boolean | Whether to skip downloading images. Image downloading is opt-in |
|
| Integer | Maximum allowed image size (applies only when images are downloaded) |
|
| Integer | Maximum HTML page body size |
|
| Boolean | Enable JSONL page output (one record per page plus a trailing crawl_meta record) for RAG pipelines |
|
| String | Filename for JSONL output |
|
| Boolean | Enable incremental crawling globally |
|
| Integer | Number of past crawls per site kept in the SQLite history index (powers |
|
| Object | HTTP client configuration | (see below) |
| Map | Site-specific configurations | (required) |
HTTP Client Settings:
(Global; cannot be overridden per site. Pool, dialer, and TLS timings are baked into pkg/fetch with sane defaults and are not exposed as config knobs.)
timeout: Overall request timeout (default 45s)
max_idle_conns_per_host: Idle connections per host (default 2)
allow_private_networks: Disables the SSRF guard that blocks dials to loopback / private / link-local / CGNAT / multicast addresses. Default false. Set to true only if you intentionally crawl internal documentation servers reachable via private IPs.
Site-Specific Configuration Options:
start_urls: Array of starting URLs for crawling (Required)
allowed_domain: Restrict crawling to this domain (Required)
allowed_path_prefix: Restrict crawling to URLs under this path prefix (Optional; defaults to /, the whole domain). Setting it is strongly recommended to bound scope
content_selector: CSS selector for main content extraction, or "auto" for automatic detection (Required)
max_depth: Exclusive upper bound on crawl depth from start URLs. Start pages are depth 0, so 1 crawls only the start pages, 2 adds their directly-linked pages, and so on. 0 = unlimited. URLs discovered from a sitemap.xml are seeded at depth 1 (one hop from the site root), so they are still bounded by max_depth: max_depth: 1 stays start-only and skips sitemap expansion
delay_per_host: Override global delay setting for this site
disallowed_path_patterns: Array of regex patterns for URLs to skip
link_extraction_selectors: Array of CSS selectors for additional link extraction areas
respect_nofollow: Boolean. Whether to respect rel="nofollow" links
user_agent: String. Override global user agent for this site
skip_images: Override the global image setting for this site. Images are skipped unless this (or the global skip_images) is set to false
max_image_size_bytes: Integer. Override global max image size for this site
allowed_image_domains: Array of domains from which to download images
disallowed_image_domains: Array of domains to block image downloads from
enable_jsonl_output: true or false. Override global JSONL output enablement for this site
jsonl_output_filename: String. Override global JSONL output filename for this site
Execute the compiled binary from the project root directory:
./doc-scraper <command> [options]Command | Description |
| Start a crawl (add |
| Probe a docs site and draft a config entry for it: detects the framework, proposes crawl scope from the sitemap, previews one extracted page, and writes only after confirmation |
| Validate configuration file without crawling |
| List available site keys from config |
| Start MCP server for AI tool integration |
| Ranked full-text search over the crawled corpus (BM25, stemming, section anchors) |
| Watch sites and re-crawl on schedule |
| Show version information |
| Read a JSON task spec from stdin and dispatch a crawl or watch (for orchestration/automation) |
crawl:
Flag | Description | Default |
| Path to config file |
|
| Site key from config (single site) | - |
| Comma-separated site keys for parallel crawling | - |
| Crawl all configured sites in parallel |
|
| Resume an interrupted crawl from existing state |
|
| Log level ( |
|
| Emit logs as JSON (one record per line) instead of text |
|
| pprof server address. Only effective in builds with |
|
| Enable incremental crawling (skip unchanged pages) |
|
| Force full crawl (ignore incremental settings) |
|
Note: One of -site, -sites, or --all-sites is required.
add:
doc-scraper add https://vitepress.dev/guide/what-is-vitepressProbes the site with a handful of polite requests (the page, robots.txt, llms.txt, the sitemap), then shows what it found before anything is written: the detected framework and content selector (validated against the fetched page), a crawl scope clustered from the sitemap with page counts as evidence, sibling version/locale trees proposed as exclusions, and a markdown preview of the extracted page with code-block fidelity numbers. The entry is appended to your config only after you confirm; the rest of the file is preserved byte-for-byte, comments included.
Flag | Description | Default |
| Path to config file (created if missing) |
|
| Site key to use instead of the derived one | - |
| Content CSS selector, skipping auto-detection | - |
| Override the proposed max_depth | - |
| Write without prompting |
|
| Draft only, never write (exit code 2) |
|
| Emit the draft as JSON on stdout (human text goes to stderr) |
|
Exit codes: 0 written, 1 error, 2 drafted but not written. For agents and scripts: add -dry-run -json <url> inspects, then add -yes <url> commits; with no terminal attached the command fails fast instead of waiting on stdin. Sites whose robots.txt disallows crawling the given path are refused, and robots rules that restrict AI crawlers are surfaced as a warning.
config validate:
Flag | Description | Default |
| Path to config file |
|
| Site key to validate (optional, validates all if empty) | - |
| Emit a single JSON object instead of human-readable text |
|
config list:
Flag | Description | Default |
| Path to config file |
|
| Emit a single JSON object instead of human-readable text |
|
mcp-server: (stdio transport only; the SSE transport was removed in v2.x)
Flag | Description | Default |
| Path to config file |
|
| Log level ( |
|
watch:
Flag | Description | Default |
| Path to config file |
|
| Site key to watch (single site) | - |
| Comma-separated site keys to watch | - |
| Watch all configured sites |
|
| Crawl interval (e.g., |
|
| Log level ( |
|
| Emit logs as JSON (one record per line) instead of text |
|
Note: One of -site, -sites, or --all-sites is required.
Basic Crawl:
./doc-scraper crawl -site tensorflow_docs -loglevel infoResume a Large Crawl:
./doc-scraper crawl -site pytorch_docs --resume -loglevel infoValidate Configuration:
./doc-scraper config validate -config config.yaml
./doc-scraper config validate -site pytorch_docs # Validate specific siteList Available Sites:
./doc-scraper config listHigh Performance Crawl with Profiling:
./doc-scraper crawl -site small_docs -loglevel warn -pprof localhost:6060Debug Mode for Troubleshooting:
./doc-scraper crawl -site test_site -loglevel debugParallel Crawl of Multiple Sites:
./doc-scraper crawl -sites pytorch_docs,tensorflow_docs,langchain_docsCrawl All Configured Sites:
./doc-scraper crawl --all-sitesStart MCP Server for Claude Desktop:
./doc-scraper mcp-server -config config.yamlcrawl -incremental (which implies --resume, and is also what watch mode uses) re-fetches every previously-crawled page and re-checks it for changes:
Change detection is content-scoped: it hashes the extracted content-selector region, not the raw page. Churn in the page shell (navigation, analytics, build timestamps, CSRF tokens) outside the content selector does not count as a change.
Pages whose content region is unchanged are skipped without re-converting, re-downloading images, or rewriting output.
Pages whose content region changed are fully reprocessed and their output is rewritten.
A page that now returns an error (e.g. 404) on re-crawl leaves its previously-crawled output as-is; nothing is pruned.
Because there is no conditional-request support yet, incremental mode still performs the HTTP fetch for each known page; the savings come from skipping the downstream processing of unchanged pages.
Crawled content is saved under the output_base_dir defined in the config, organized by site key and preserving the site structure. Keying by site key (rather than domain) keeps two site configs that target the same domain in separate trees:
<output_base_dir>/
└── <sanitized_site_key>/ # e.g., flask_docs
├── images/ # Always created; only populated when skip_images: false
│ ├── image1.png
│ └── image2.jpg
├── index.md # Markdown for the root path
├── <jsonl_output_filename> # If enable_jsonl_output: true
├── llms.txt # Manifest of pages (auto-generated, when JSONL is enabled)
├── llms-full.txt # Full content concatenated (auto-generated, when JSONL is enabled)
├── topic_one/
│ ├── index.md
│ └── subtopic_a.md
└── topic_two.mdWhen JSONL output is enabled, the crawler also emits llms.txt and llms-full.txt following the llmstxt.org convention. llms.txt is a markdown manifest (H1 + summary blockquote + ## Pages list of every crawled page with title and URL). llms-full.txt concatenates the full markdown content of every page, with section separators. Both files are regenerated on every crawl from the JSONL source of truth, so resumed crawls produce a complete updated manifest.
Each generated Markdown file begins with a YAML frontmatter block carrying page metadata, followed by the converted content:
YAML frontmatter (delimited by ---) with title, url (source URL), crawled_at (RFC3339 timestamp), content_hash (SHA-256 of the content, matching the JSONL record), and depth
Clean content converted from HTML to GitHub-Flavored Markdown, preserving tables
Relative links to other pages (when within the allowed domain)
Local image references (if images are enabled)
Example:
---
title: 'Authentication'
url: https://docs.example.com/api/auth
crawled_at: "2026-08-09T12:00:00Z"
content_hash: 9f2b...c1a4
depth: 2
---
# Authentication
...page content as Markdown...When enabled, the crawler writes one JSON object per line to a JSONL file. This format is designed for ingestion into RAG pipelines and downstream indexers.
Enable it:
enable_jsonl_output: true
jsonl_output_filename: "pages.jsonl" # defaultThe file mixes two record kinds, distinguished by the record_type field:
page records, one per crawled page.
A single crawl_meta record as the final line, holding the crawl-level summary. Resuming rewrites the file to drop any leftover crawl_meta record before appending a fresh one at close, so a closed file always contains exactly one crawl_meta record.
page record fields (from PageJSONL):
Field | Description |
| Always |
| Final absolute URL of the page |
| Page title |
| Full markdown content |
| Array of headings extracted from the page |
| Array of links found in the content |
| Array of image URLs found in the content |
| SHA-256 hash of the content (used for incremental crawling) |
| Timestamp of when the page was crawled |
| Crawl depth from the start URL |
crawl_meta record fields (from CrawlMetaJSONL):
Field | Description |
| Always |
| Site key from the config |
| The crawled domain |
| Crawl start timestamp |
| Crawl end timestamp |
| Number of pages recorded in this crawl |
The output file is written to each site's output directory. Both the enable flag and filename can be overridden per site.
When you set content_selector: "auto" for a site, the crawler automatically detects the documentation framework and applies the appropriate content selector.
Detection recognizes 30+ documentation generators and hosted platforms, checked in three tiers of decreasing trust: the <meta name="generator"> tag, structural DOM signatures (attributes, ids, classes), and asset path patterns. Covered families include Docusaurus, VitePress, VuePress, Starlight/Astro, Nextra, Fumadocs, Mintlify, GitBook, MkDocs (Material, ReadTheDocs theme, and plain), Sphinx (furo, pydata, book, RTD, and classic themes), Antora, Docsy, hugo-book, Geekdoc, just-the-docs, mdBook, rustdoc, pkg.go.dev, Javadoc, Doxygen, TypeDoc, Writerside, ReadMe.com, Intercom, and Docus.
Every detected selector is validated against the live page before it is trusted: if it matches nothing or captures too little text, the crawler falls back instead of extracting empty content. Client-rendered shells (Docsify, Swagger UI, Redoc, Scalar, Document360, and generic empty-body SPAs) are recognized and reported as needing JavaScript rendering rather than silently producing an empty crawl.
If no known framework is detected (or the detected selectors do not match the page), the crawler uses Mozilla's Readability algorithm to extract the main content. This works well on classic server-rendered docs, but can drop code blocks on some modern sites, so doc-scraper add's preview reports code-block fidelity before you commit a config.
sites:
pytorch_docs:
start_urls:
- "https://pytorch.org/docs/stable/"
allowed_domain: "pytorch.org"
allowed_path_prefix: "/docs/stable/"
content_selector: "auto" # Auto-detect framework
max_depth: 0Crawl multiple documentation sites concurrently with shared resource management. The orchestrator coordinates multiple crawlers while respecting global rate limits and semaphores.
# Crawl specific sites in parallel
./doc-scraper crawl -sites pytorch_docs,tensorflow_docs,langchain_docs
# Crawl all configured sites
./doc-scraper crawl --all-sites
# Resume parallel crawl
./doc-scraper crawl -sites pytorch_docs,tensorflow_docs --resumeWhen running parallel crawls, the following resources are shared across all site crawlers:
Global semaphore: Limits total concurrent requests across all sites
HTTP client: Shared connection pooling
Rate limiter: Respects per-host delays
Each site still maintains its own:
BadgerDB store for state persistence
Output directory for crawled content
Per-host semaphores for domain-specific limiting
After all sites complete, the orchestrator outputs a summary:
===========================================
Parallel crawl completed in 2m30s
Site Results:
pytorch_docs: SUCCESS - 1500 pages in 1m20s
tensorflow_docs: SUCCESS - 2000 pages in 2m15s
langchain_docs: FAILED - 0 pages in 3s
Error: initial fetch failed for start URL (see logs)
-------------------------------------------
Total: 3 sites (2 success, 1 failed), 3500 pages processed
===========================================Unknown or misspelled site keys are rejected before the crawl starts, so they never appear as a FAILED row in this summary. For example, crawl -sites pytorch_docs,typo_key exits immediately (non-zero) with:
Invalid site keys: site 'typo_key' not found. Available sites: [pytorch_docs tensorflow_docs langchain_docs]The FAILED rows in the summary are for sites that exist in the config but errored during the crawl itself.
Watch mode enables scheduled periodic re-crawling of documentation sites. The scheduler tracks the last run time for each site and automatically triggers crawls when the configured interval has elapsed.
# Watch a single site with 24-hour interval
./doc-scraper watch -site pytorch_docs -interval 24h
# Watch multiple sites
./doc-scraper watch -sites pytorch_docs,tensorflow_docs -interval 12h
# Watch all configured sites weekly
./doc-scraper watch --all-sites -interval 7dThe interval supports standard Go duration format plus day units:
30m - 30 minutes
1h - 1 hour
24h - 24 hours
7d - 7 days
1d12h - 1 day and 12 hours
Watch mode persists state to <state_dir>/watch_state.json, tracking:
Last run time for each site
Success/failure status
Pages processed
Error messages (if any)
This allows the scheduler to resume correctly after restarts, only running sites when their interval has elapsed.
INFO Starting watch mode for 2 sites with interval 24h0m0s
INFO Watch schedule:
INFO pytorch_docs: last run 2024-01-15T10:30:00Z (success, 1500 pages), next run 2024-01-16T10:30:00Z
INFO tensorflow_docs: never run, will run immediately
INFO Running crawl for 1 due sites: [tensorflow_docs]
...
INFO Next crawl: pytorch_docs in 23h45m (at 10:30:00)Watch mode handles SIGINT/SIGTERM gracefully: it stops the scheduler and cancels any in-progress crawl, letting the crawler flush its BadgerDB state and partial output first, so the interrupted crawl resumes cleanly on the next run.
The run command reads a single JSON object from stdin and dispatches the equivalent crawl or watch. It is meant for orchestration agents that would rather build a JSON payload than assemble shell flags. Unknown fields are rejected so typos surface immediately; logs go to stderr and the exit code matches the equivalent flag-driven subcommand.
{
"command": "crawl" | "watch", // required
"config": "config.yaml", // optional, defaults to config.yaml
"site": "site_key", // exactly one of site | sites | all_sites
"sites": ["a", "b"],
"all_sites": true,
"resume": false, // crawl only
"incremental": false, // crawl only (implies resume)
"full": false, // crawl only (mutually exclusive with incremental)
"interval": "24h", // watch only, defaults to 24h
"loglevel": "info", // defaults to info
"json_logs": false, // emit slog records as JSON on stderr
"pprof": "" // crawl only, e.g. localhost:6060
}Examples:
echo '{"command":"crawl","site":"pytorch_docs"}' | doc-scraper run
echo '{"command":"crawl","all_sites":true,"incremental":true,"json_logs":true}' | doc-scraper run
echo '{"command":"watch","sites":["pytorch_docs","tensorflow_docs"],"interval":"6h"}' | doc-scraper runThe crawler can run as a Model Context Protocol (MCP) server, enabling integration with AI assistants like Claude Code and Cursor.
Tool | Description |
| Orientation manifest: server identity + sites + recent jobs in one call (call this first) |
| List all configured sites from config file |
| Fetch a single URL live over the network and return content as markdown |
| Start a background crawl for a site (returns job ID) |
| Check the status of a background crawl job |
| Cancel a running or pending crawl job by job ID |
| Enumerate crawled pages for a site (paginated, metadata only) |
| Return a crawled page's markdown from the stored output, without network access |
| Full-text search across crawled docs (BM25, stemming, snippets), without network access |
| Report how stale a site's latest crawl is, from the crawl-history index |
| Report pages added, removed, or changed since a given timestamp |
The MCP server uses the stdio transport, compatible with Claude Desktop, Claude Code, and Cursor.
./doc-scraper mcp-server -config config.yamlAdd to your Claude Code configuration (claude_code_config.json):
{
"mcpServers": {
"doc-scraper": {
"command": "/path/to/doc-scraper",
"args": ["mcp-server", "-config", "/path/to/config.yaml"]
}
}
}List available sites:
Tool: list_sites
Result: Returns all configured sites with their domains and crawl statusFetch a single page:
Tool: get_page
Arguments: { "url": "https://docs.example.com/guide", "content_selector": "article" }
Result: Returns page content as markdown with metadataStart a background crawl:
Tool: crawl_site
Arguments: { "site_key": "pytorch_docs", "incremental": true }
Result: Returns job ID for tracking progressCheck crawl progress:
Tool: get_job_status
Arguments: { "job_id": "abc-123-def" }
Result: Returns status, pages processed, and completion infoEnumerate crawled pages:
Tool: list_pages
Arguments: { "site_key": "pytorch_docs", "max_results": 50, "offset": 0 }
Result: Returns up to 50 page entries (URL, title, depth, crawled_at, content_length), sorted by URL. Use offset for pagination.Cancel a running crawl:
Tool: cancel_crawl
Arguments: { "job_id": "abc-123-def" }
Result: Returns cancelled: true/false and the job's current status. Has no effect on jobs already in a terminal state.Contributions are welcome! Please feel free to open an issue to discuss bugs, suggest features, or propose changes.
Pull Request Process:
Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add some amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request
Please ensure code adheres to Go best practices and includes appropriate documentation.
doc-scraper collects nothing: no telemetry, no analytics, no accounts. All output and state stays on your machine, and the only network requests it makes are the crawls and fetches you explicitly ask for. Full policy: PRIVACY.md.
This project is licensed under the Apache-2.0 License.
GoQuery for HTML parsing
html-to-markdown for conversion
BadgerDB for state persistence
mcp-go for MCP server implementation
go-readability for content extraction fallback
modernc.org/sqlite for the pure-Go crawl-history index
Cancel a running or pending crawl job by job ID. Has no effect on jobs already in a terminal state.
| Name | Required | Description | Default |
|---|---|---|---|
| job_id | Yes | The job ID returned by crawl_site |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already provide idempotentHint=true and destructiveHint=false. The description adds the terminal-state no-op disclosure and clarifies the exact operation scope, giving the agent a precise expectation of side effects beyond what annotations alone convey.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with action-first phrasing and no filler. The second sentence earns its place by defining the boundary condition, making the whole description compact yet complete.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a single-parameter action with full schema coverage and helpful annotations, the description fully states the target states, the no-op condition, and the identifying parameter. No output schema exists, but return values are not necessary for correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and the job_id description already explains its origin from crawl_site. The description only repeats 'by job ID' without adding new semantics, so the baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('Cancel'), a specific resource ('crawl job'), and the key selector ('by job ID'). The action is unmistakably distinct from siblings like get_job_status and crawl_site, even without naming them.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly says the tool applies to running or pending jobs and gives a clear when-not condition: no effect on terminal-state jobs. It does not name alternative tools, but the boundary condition is enough to guide correct use.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Start a background crawl for a configured site. Returns immediately with a job ID.
| Name | Required | Description | Default |
|---|---|---|---|
| site_key | Yes | Site key from config file (e.g., 'langchain_py', 'rust_docs') | |
| incremental | No | Enable incremental mode (skip unchanged pages) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Adds key behavioral context beyond annotations: the crawl runs in the background and the call returns immediately with a job ID, implying the actual work happens asynchronously. This helps the agent know to poll get_job_status later. It also clarifies that the site must already be configured, setting a prerequisite not visible from annotations alone.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with no filler: the first states the action and scope, the second states the return behavior. Information is front-loaded and every sentence adds value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple 2-parameter tool with no output schema, the description is sufficient: it explains the start action, background execution, and the immediate return of a job ID. It could mention how to monitor progress, but the existence of a get_job_status sibling makes that omission minor.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so both site_key and incremental are already fully documented. The description does not add any additional meaning for these parameters, so the baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a clear action: start a background crawl for a configured site, and explicitly notes it returns a job ID immediately. This distinguishes it from sibling tools like cancel_crawl or get_job_status, which operate on existing crawls.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies when to use it—when you want to kick off a crawl of a configured site—but does not explicitly discuss alternatives or exclusion conditions. An agent must infer that cancel_crawl or diff_crawl serve different purposes, so some routing burden remains on the agent.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Returns server identity, configured sites, and recent crawl jobs in one call. Call this first to orient yourself: it consolidates what would otherwise require list_sites plus several get_job_status calls. The MCP tool list is already advertised by the protocol so it is not duplicated here.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, idempotentHint, and destructiveHint, so the safety profile is covered. The description adds behavioral value by explaining that this is a single combined call and that the MCP tool list is intentionally not duplicated, which clarifies a potential expectation gap.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three concise sentences, each earning its place: the first states the purpose, the second gives usage guidance, and the third prevents a likely confusion about omissions. Information is front-loaded and there is no redundancy with the annotations or schema.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no parameters, no output schema, and complete annotations, the description provides the needed orientation: what the tool returns and when to call it. Nothing required to invoke it correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, so there are no parameters to document. The baseline for a 0-parameter tool is 4, and the description appropriately focuses on return content rather than input semantics.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: it returns server identity, configured sites, and recent crawl jobs. It distinguishes itself from siblings by explicitly noting that it consolidates list_sites plus several get_job_status calls, so an agent can tell it apart without inspecting other tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives clear usage context with 'Call this first to orient yourself' and names the alternatives it consolidates (list_sites, get_job_status). It does not explicitly state when not to use it, but the orient-first guidance and consolidation rationale are sufficient for correct selection.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Return added/removed/changed pages between the latest crawl and the most recent crawl whose crawl_ended_at <= since. Hash-based verdicts from the SQLite history index. Pair with get_freshness: pass the last_crawl_ended_at value as since after running crawl_site with incremental set to true to see exactly what changed.
| Name | Required | Description | Default |
|---|---|---|---|
| since | Yes | RFC3339 timestamp; the most recent crawl whose crawl_ended_at <= since is the baseline (e.g. 2026-05-23T22:00:00Z) | |
| offset | No | Pagination offset, 0-based | |
| site_key | Yes | Site key from config | |
| max_results | No | Maximum diff entries to return (default 100, max 1000) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, idempotentHint, and destructiveHint, so the safety profile is covered. The description adds behavioral context by noting 'hash-based verdicts from the SQLite history index' and precisely defining the baseline selection rule, which goes beyond the structured annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences, with the core behavior first and the workflow example last. Each sentence carries distinct information (what, how, when) with no filler or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given there is no output schema, the description gives a sufficient high-level contract: added/removed/changed pages with hash-based verdicts. It also reduces ambiguity by naming the exact sequence with crawl_site and get_freshness, so an agent can correctly select and chain the tool; minor gaps like response entry shape remain but are not essential for invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema covers 100% of parameters, so the baseline is 3. The description adds cross-tool value by explaining where `since` comes from ('pass the last_crawl_ended_at value as since'), which is not present in the input schema; the other parameters are already well documented in the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a clear action and resource: 'Return added/removed/changed pages between the latest crawl and the most recent crawl whose crawl_ended_at <= since.' This is specific enough to distinguish it from sibling tools like get_freshness, and the title 'Diff crawls over time' reinforces the same function.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides a concrete workflow: 'Pair with get_freshness: pass the last_crawl_ended_at value as since after running crawl_site with incremental set to true.' This tells an agent when to call it and how to chain it with siblings, though it does not explicitly state exclusions or when-not-to-use alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Return the most recent crawl summary for a site (last_crawl_started_at/ended_at, total_pages, mode, age_seconds) plus output/state dir presence and any running job. Use this to decide whether to query the existing crawl or run crawl_site first.
| Name | Required | Description | Default |
|---|---|---|---|
| site_key | Yes | Site key from config (use list_sites to discover available keys) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, idempotentHint, and destructiveHint, so the safety profile is covered. The description goes further by disclosing what the tool returns (crawl summary fields, output/state dir presence, running job), which is useful behavioral detail beyond the structured annotations. It accurately describes a read-only inspection operation without contradicting any annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two dense sentences: the first lists the return content, the second gives the decision context. No filler and the key return information is front-loaded. The description achieves high information density in a compact format.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a single-parameter read-only tool with no output schema, this description is complete. It states what is returned, how to decide to use the tool, and the annotations cover idempotence and safety. The provided information fully supports correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema documents site_key at 100% coverage, including the hint to use list_sites to discover available keys. The description adds no parameter-level details, but with full schema coverage the baseline score of 3 is appropriate. No extra semantics are needed beyond what the schema already provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource ('Return the most recent crawl summary for a site') and enumerates the returned fields, making the operation unmistakable. It also injects a decision cue ('Use this to decide whether to query the existing crawl or run crawl_site first') that distinguishes it from the crawl_site sibling. This is a clear, specific purpose statement that differentiates it from other tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly states when to use the tool: before deciding between querying an existing crawl and running crawl_site. This names the alternative and the condition, satisfying the usage guidance dimension. The tool's role as a freshness check is directly tied to a decision, giving clear context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Get the status of a crawl job
| Name | Required | Description | Default |
|---|---|---|---|
| job_id | Yes | The job ID returned by crawl_site |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so the safety profile is established. The description adds no behavioral context beyond the word 'Get', such as what status values may appear or whether the call reflects updated progress. No contradiction exists.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, focused sentence with no filler. The verb and object are front-loaded, making it immediately scannable for an agent.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a one-parameter, read-only status lookup with strong annotations, the definition is mostly sufficient. However, with no output schema, it does not indicate what statuses or fields the response contains, and the absence of usage guidance leaves minor but real ambiguity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and the single job_id parameter is adequately documented as the ID returned by crawl_site. The description itself adds no parameter-level meaning beyond what the schema already provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Get') and resource ('status of a crawl job'), making the tool's purpose clear. It is naturally distinct from siblings like crawl_site or cancel_crawl, though it does not explicitly name or contrast alternatives.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is only implied: checking the status of a crawl job makes sense after calling crawl_site, and the schema's job_id description references crawl_site as the source of the ID. However, the description itself gives no explicit when-to-use or when-not-to-use guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Fetch a URL live over the network and return its content as markdown. This is an on-demand fetch independent of any crawl: it does not read the stored crawl output, and it ignores the site's configured content_selector and scope.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to fetch | |
| content_selector | No | CSS selector for main content (defaults to 'body') |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The annotations already provide readOnlyHint, openWorldHint, idempotentHint, and destructiveHint=false. The description adds substantial behavioral detail beyond these: live network access, markdown output, independence from crawl output, and intentional disregard for site-configured content_selector/scope. There is no contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with action and output, then a compact contrast that disambiguates from crawl-based tools. Every sentence earns its place with no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a two-parameter read-only tool with rich annotations, 100% schema coverage, and no output schema, the description is sufficient. It explains what, when, and one key quirk, leaving no ambiguity about invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema covers 100% of the parameters, including the content_selector default of 'body'. The description adds no meaningful parameter-level detail; its mention of content_selector refers to site configuration rather than the argument itself. Baseline 3 is appropriate because the schema carries the parameter documentation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource: 'Fetch a URL live over the network and return its content as markdown.' It also distinguishes itself from siblings by explicitly noting it does not read stored crawl output, which clearly separates it from read_page and other crawl-related tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It provides a clear context: use this for an on-demand, live fetch independent of any crawl, as opposed to reading stored crawl output. It doesn't name an explicit alternative such as read_page, but the contrast with crawl-based reads gives an agent enough guidance to choose correctly.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
List crawled pages for a site, paginated and sorted by URL. Returns metadata only (URL, title, depth, crawled_at, content_length). Pass any URL returned here to read_page to get its stored markdown; get_page re-fetches a URL live rather than returning the crawled copy.
| Name | Required | Description | Default |
|---|---|---|---|
| offset | No | Pagination offset, 0-based (default: 0) | |
| site_key | Yes | Site key from config (use list_sites to discover available keys) | |
| max_results | No | Maximum pages to return (default: 100, max: 1000) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already cover read-only, idempotent, and non-destructive behavior. The description adds useful behavioral context: it returns metadata only (URL, title, depth, crawled_at, content_length), is paginated and sorted by URL, and reflects stored crawled pages rather than live fetches. It doesn't discuss errors or partial failures, so it is strong but not exhaustive.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three focused sentences front-load the core action, state output scope and shape, and route to sibling tools. Every sentence earns its place with no wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a read-only listing tool with fully documented parameters and safety annotations, the description covers action, scope, output fields, pagination ordering, and downstream tool usage. The schema supplies the remaining parameter details, so nothing essential is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the baseline applies. The description's mention of pagination and 'for a site' mildly reinforces site_key, offset, and max_results, but it does not add parameter meaning beyond what the schema already documents.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description uses a specific verb 'List' with a clear resource 'crawled pages for a site' and adds pagination and sort order. It also distinguishes itself from read_page and get_page by explicitly stating that it returns metadata only and that the pages are crawled, not live.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly routes downstream usage: URLs returned here should be passed to read_page for stored markdown, while get_page is for live re-fetching rather than crawled copies. This gives an agent clear when-to-use and when-not-to-use guidance for the relevant siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
List all configured sites available for crawling
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, idempotentHint, and destructiveHint, so the safety profile is fully covered. The description adds that the tool lists only configured sites, which is useful scoping, but it does not mention return format or pagination. This is acceptable given the simple read-only nature.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, front-loaded sentence that communicates the action, resource, and scope with no wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a parameterless, read-only list tool with comprehensive annotations, the description is sufficiently complete. An agent knows exactly what this tool does and can invoke it without missing information.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, so there is nothing for the description to elaborate on. The baseline of 4 applies here since no parameter documentation is needed.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states a specific verb ('List') and resource ('all configured sites'), and the phrase 'available for crawling' adds useful context that distinguishes this from sibling tools like list_pages. An agent can immediately tell what the tool does without opening the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies the usage context: this is for discovering which sites are configured and eligible for crawling. However, it does not explicitly state when to use this tool versus alternatives such as crawl_site or list_pages, leaving some room for inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Return a page's markdown from the stored crawl output, without any network access. This is the counterpart to get_page: read_page serves the crawled copy that already had the site's content_selector applied, while get_page re-fetches the URL live. Use list_pages to discover URLs, then read_page to read them. Large pages are truncated at max_bytes; follow next_offset to read the rest.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL of a crawled page, as reported by list_pages | |
| offset | No | Byte offset into the page content, for reading a truncated page in parts (default: 0) | |
| site_key | Yes | Site key from config (use list_sites to discover available keys) | |
| max_bytes | No | Maximum content bytes to return (default: 102400, max: 1048576) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the readOnly/idempotent annotations, the description reveals important behavior: no network access, content_selector already applied, truncation at max_bytes, and a next_offset mechanism for reading the rest. These are not derivable from the annotations or schema alone.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Four sentences, each earning its place: core function, sibling distinction, discovery workflow, and pagination behavior. The most important information is front-loaded, and there is no filler or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool without an output schema, the description adequately covers return format, truncation, and continuation via next_offset. It also provides the surrounding workflow with list_pages and clarifies the live-crawl tradeoff, leaving no critical gap for an agent to call it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, and the schema already explains each parameter's meaning. The description's mention of max_bytes truncation and next_offset adds behavioral context, but it does not add new semantic detail about the parameters themselves.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Return a page's markdown from the stored crawl output, without any network access.' It explicitly contrasts itself with get_page, making the tool's role unambiguous and differentiating it from the most similar sibling.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives clear workflow guidance: 'Use list_pages to discover URLs, then read_page to read them.' It also explains the exact tradeoff versus get_page (crawled copy vs live re-fetch), so an agent knows when to select this tool over the alternative.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Full-text search across all crawled documentation, ranked by relevance (BM25 with stemming), with zero network access. Results carry the page URL, its section heading path, and a snippet with match terms marked [like this]; follow up with read_page for the full page. Supports FTS5 syntax: quoted phrases, OR, and trailing * for prefix matching. Searches the stored index only; run crawl_site first for uncrawled sites.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Maximum results to return (default: 10, max: 50) | |
| query | Yes | Search terms. Plain words are ANDed; "quoted phrases", OR, and prefix* work too. | |
| site_key | No | Limit results to one site (use list_sites to discover keys). Omit to search all sites. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the annotations (readOnlyHint, idempotentHint, destructiveHint), the description reveals important behavioral traits: zero network access, BM25 ranking with stemming, result format including URL and section path, FTS5 syntax support, and the stored-index-only scope. This adds substantial context about what happens when calling the tool and what the output contains, with no contradiction to annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Four dense sentences, each earning its place: purpose and ranking, result format and follow-up, query syntax, and index caveat. The most important scoping constraints are front-loaded, and there is no fluff or repetition of schema content.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given there is no output schema, the description properly explains what results contain (URL, heading path, snippet with highlighted matches) and how to get the full content. It also covers prerequisites (crawl_site), querying syntax, and scope limitations. The agent has enough context to invoke the tool correctly without missing behavioral details.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already describes all three parameters at 100% coverage, including the query syntax, limit bounds, and site_key usage. The description reiterates the FTS5 syntax but does not meaningfully extend parameter-level understanding beyond the schema. Baseline 3 is appropriate because the schema carries the parameter-heavy lifting.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Full-text search across all crawled documentation'. It further distinguishes the tool from siblings by stating it is ranked by relevance, works offline, and searches only the stored index, which clearly differentiates it from list_pages, get_page, and read_page.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives clear context for when to use the tool: when you need to search crawled documentation. It also provides practical guidance by telling the agent to run crawl_site first for uncrawled sites and to follow up with read_page for the full page. However, it does not explicitly mention when not to use this tool in favor of a sibling like list_pages or get_page, so it falls just short of a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
cancel_crawlcrawl_sitedescribe_serverdiff_crawlget_freshnessget_job_statusget_pagelist_pageslist_sitesread_pagesearch_docsThis server cannot be installed
Each tool has a distinct role, and the get_page/read_page vs list_pages/search_docs boundaries are clearly explained. The only mild overlap is get_freshness and get_job_status, and describe_server partly duplicates list_sites/job status, though it is positioned as an orientation call.
All tools use snake_case verb_noun names like list_sites, crawl_site, cancel_crawl, read_page, and search_docs. There is no mixing of styles or vague verbs; even diff_crawl follows the verb-first pattern.
11 tools is right-sized for a documentation crawling and retrieval server. Each tool maps to a meaningful operation without redundant surfaces.
The crawl-and-read workflow is fully covered: start/cancel/status, list/read/search stored pages, and diff/freshness for updates. The main gap is that site management is limited to listing pre-configured sites, with no tool to add, remove, or update site definitions, so agents cannot onboard new documentation sources without out-of-band configuration.
Agent-native MCP server over the public saagarpatel.dev corpus. Read-only, stateless.
Scrape, crawl, map & search the web. Open-source, self-hostable Rust crawler & search for AI agents.
Document-to-Markdown MCP server — convert PDF, Office and HTML into LLM-ready Markdown.
SEO MCP server: crawl your site, find AI-visibility gaps, and ship the fix from your coding agent.
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/Sriram-PR/doc-scraper'
If you have feedback or need assistance with the MCP directory API, please join our Discord server
