Skip to main content
Glama
GeorgesAdSim

Fetch Crawl MCP

by GeorgesAdSim

Crawl Site

crawl_site
Read-only

Recursively crawl a website from a starting URL to build a list of discovered pages with titles and HTTP status codes. Set crawl depth, page limits, concurrency, and URL filters to control scope and impact.

Instructions

Crawl a website recursively starting from a URL. Follows internal links up to a specified depth and max pages. Returns a list of discovered pages with their titles and status codes.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
urlYesThe starting URL to crawl
delayNoBase delay in ms between requests. A random jitter of ±30% is applied automatically (default: 300ms)
maxDepthNoMaximum crawl depth (0 = only the starting page)
maxPagesNoMaximum number of pages to crawl
concurrencyNoNumber of pages to fetch in parallel (default: 3). Lower values are safer for small sites
excludePatternNoRegex pattern: skip URLs matching this pattern
includePatternNoRegex pattern: only crawl URLs matching this pattern
respectRobotsTxtNoRespect robots.txt rules and Crawl-delay (default: true)

Schema Changelog

Changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. First observedv4.2.0

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The readOnlyHint annotation already communicates that the tool is non-destructive. The description adds meaningful behavioral detail beyond that: it follows internal links, respects depth and page limits, and returns discovered pages with titles and status codes. This is a reasonable level of transparency for a read-only crawler.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences and every sentence adds value: it states the core action, the recursive behavior, the key limits, and the output format. There is no redundant or filler content.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description, combined with a fully described input schema and the readOnlyHint annotation, gives an agent enough context to invoke the tool correctly. It explains the main behavior and output, even without an output schema. Minor details like how external links are handled or exact error conditions are not described, but they are not essential for calling the tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the parameter descriptions already document each input. The description adds little beyond the schema; it mentions depth and max pages but these are already described directly in the input schema. It does not introduce any new meaning that the schema does not already provide.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states a specific action ('Crawl a website recursively starting from a URL'), names the resource, and describes the recursive link-following behavior and output. It is easy to distinguish from sibling tools like fetch_page or parse_sitemap because it uniquely identifies full-site recursive crawling.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context for when this tool is appropriate: when you need to recursively crawl a website from a starting URL and discover pages. It does not explicitly name alternatives or say 'when not to use', but the recursive crawling language strongly implies this is for site-wide discovery rather than single-page fetches or sitemap parsing.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/GeorgesAdSim/fetch-crawl-mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server