How do I use crawler-mcp?

1. Click on "Install Server". 2. Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state. 3. In the chat, type @ followed by the MCP server name and your instructions, e.g., "@crawler-mcp Fetch the text content of https://example.com" That's it! The server will respond to your query, and you can continue using it as needed. Here is a step-by-step guide with screenshots.

crawler-mcp

by shadab15github

Overview Schema Related Servers Score Discussions

TypeScript

Hybrid

crawler-mcp

A web-crawler MCP server, built in the same style as weather-mcp (TypeScript + @modelcontextprotocol/sdk + Express + Zod).

It fetches pages with a fast axios + cheerio path and automatically falls back to a Playwright headless browser when a page needs JavaScript to render. Designed to be used as a Claude connector (local stdio or remote HTTP).

Tools

Tool	What it does
`fetch_page`	Fetch one URL and return its content as Markdown, text, or raw HTML.
`extract_links`	List all hyperlinks on a page (absolute URLs, optional same-domain filter).
`crawl_site`	Recursively crawl from a start URL up to a max depth/page count.
`extract_by_selector`	Pull specific data via a CSS selector (text or an attribute).

Each tool accepts a render option: auto (default — static first, browser if needed), static (never use a browser), or browser (always render with Playwright).

Related MCP server: turbowebfetch

Setup

cd crawler-mcp
npm install

# Optional — only needed for JavaScript-heavy / SPA sites:
npx playwright install chromium

cp .env.example .env   # then edit if you want

Run

# Local (stdio) — for Claude Desktop:
npm start

# Remote (HTTP) — for a custom/remote connector:
npm run start:http     # serves http://localhost:3001/mcp

Use it as a Claude connector

Claude Desktop (local, stdio)

Add to your claude_desktop_config.json (Settings → Developer → Edit Config):

{
  "mcpServers": {
    "crawler": {
      "command": "npx",
      "args": ["tsx", "D:\\MCP\\crawler-mcp\\server.ts"]
    }
  }
}

Restart Claude Desktop. The four crawler tools appear under the connectors menu.

Remote HTTP connector (claude.ai / Desktop "Add custom connector")

Start the server in HTTP mode and expose it over HTTPS (e.g. behind a reverse proxy or a tunnel such as cloudflared / ngrok):
```
npm run start:http
```
In Claude → Settings → Connectors → Add custom connector, enter the URL:
```
https://your-host/mcp
```

Remote connectors on claude.ai generally require a public HTTPS URL. http://localhost:3001/mcp works for local testing tools but not for the hosted claude.ai web app.

Configuration (`.env`)

Variable	Default	Purpose
`MCP_TRANSPORT`	`stdio`	`stdio` or `http`.
`MCP_HTTP_PORT`	`3001`	Port for HTTP mode.
`CRAWLER_USER_AGENT`	`crawler-mcp/1.0 …`	User-Agent for all requests.
`CRAWLER_TIMEOUT_MS`	`15000`	Per-request timeout.
`CRAWLER_MAX_CHARS`	`20000`	Default cap on returned page content.

Notes

Fetched pages are cached in memory for 5 minutes to avoid refetching during a crawl.
crawl_site is bounded (max_depth ≤ 3, max_pages ≤ 50) and adds a politeness delay between requests.
Playwright is an optional dependency — if it isn't installed, render: auto still works for static sites and render: browser returns a clear error.
This server does not currently parse robots.txt; crawl responsibibly and only sites you are authorized to crawl.

Available Tools

4 tools

crawl_siteA

Recursively crawl a website starting from a URL, following links up to a maximum depth and page count. Returns a short content summary for each page visited. Stays on the same domain by default.

ParametersJSON Schema

Name	Required	Description	Default
`start_url`	Yes	Page URL (http/https). Scheme-less input like `example.com` is allowed.
`max_depth`	No	How many link-hops to follow from the start URL (0 = only the start page).
`max_pages`	No	Hard cap on the total number of pages fetched.
`same_domain`	No	Only follow links on the start URL's hostname.
`format`	No	Output format for each page's content summary.	markdown
`chars_per_page`	No	Characters of content to include per page.
`delay_ms`	No	Politeness delay between requests, in milliseconds.
`render`	No	auto=fast static fetch, fall back to headless browser if the page needs JS; static=never use a browser; browser=always render with Playwright (handles SPAs).	auto

TDQS

A4/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description should fully disclose behavior. It mentions returning a short content summary and staying on the same domain, but does not explicitly state that the tool is read-only or discuss rate limiting, authorization, or potential side effects. The lack of such details leaves some ambiguity.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise: only two sentences that convey the essential purpose, behavior, and default constraint. No unnecessary words or redundant repetition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (8 parameters, no output schema), the description covers the core functionality and default behavior. It does not describe error handling or output structure beyond a 'short content summary,' but the format parameter addresses output style. Slightly lacking in completeness but adequate.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the input schema already describes all parameters adequately. The description adds no extra parameter-specific meaning beyond what the schema provides. Baseline score of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb 'crawl' and the resource 'website', specifying recursive behavior, depth and page limits, and domain restriction. It differentiates from sibling tools like fetch_page (single page) and extract_by_selector (targeted extraction).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies when to use this tool (multi-page, domain-restricted crawling) but does not explicitly contrast with siblings or state when not to use it. The context of staying on the same domain is helpful but could be more explicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

extract_by_selectorB

Extract specific data from a page using a CSS selector. Returns each matching element's text, or an attribute value when attribute is given (e.g. selector='a.product', attribute='href').

ParametersJSON Schema

Name	Required	Description	Default
`url`	Yes	Page URL (http/https). Scheme-less input like `example.com` is allowed.
`selector`	Yes	CSS selector, e.g. 'h2', '.price', 'table tr td', 'a.result-link'.
`attribute`	No	Return this attribute instead of text (e.g. 'href', 'src', 'data-id').
`limit`	No	Maximum number of matches to return.
`render`	No	auto=fast static fetch, fall back to headless browser if the page needs JS; static=never use a browser; browser=always render with Playwright (handles SPAs).	auto

TDQS

B3.4/5.0

Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries full burden for behavioral disclosure. It does not mention whether the tool is read-only, error handling, rate limits, or what happens with no matches. The description gives limited behavioral insight beyond basic functionality.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences: the first defines the tool's purpose, and the second provides a concrete example. No unnecessary words or redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite having 5 parameters and no output schema, the description lacks details on return format, error cases, or typical usage patterns. The example is helpful, but overall completeness is low for a data extraction tool that returns results.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3. The description adds value by showing an example of the 'attribute' parameter and briefly explaining 'render' options, but most parameter meaning is already clear from the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's function: extract specific data from a page using CSS selectors, returning text or attribute values. It effectively differentiates from sibling tools like 'extract_links' and 'fetch_page' by specifying selective extraction via selectors.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides an example of using the 'attribute' parameter but does not explicitly state when to use this tool versus its siblings or when not to use it. The usage context is implied but not clearly delineated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

extract_linksB

Extract all hyperlinks from a web page, resolved to absolute URLs. Optionally restrict to links on the same domain.

ParametersJSON Schema

Name	Required	Description	Default
`url`	Yes	Page URL (http/https). Scheme-less input like `example.com` is allowed.
`same_domain`	No	Only return links pointing to the same hostname as the page.
`limit`	No	Maximum number of links to return.
`render`	No	auto=fast static fetch, fall back to headless browser if the page needs JS; static=never use a browser; browser=always render with Playwright (handles SPAs).	auto

TDQS

B3.4/5.0

Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so the description must disclose behaviors. It mentions URL resolution and same-domain filtering but omits critical details like error handling, performance impact of 'render' modes, or what happens if no links are found.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences covering the core purpose and optional restriction. No unnecessary words, front-loaded with the main action.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Missing output schema and no description of return format. Behavioral and edge-case details (e.g., empty results, error states) are absent. Sibling differentiation is adequate but incomplete for an agent to fully assess suitability.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with detailed descriptions. The description adds value by explaining 'resolved to absolute URLs' and 'restrict to same domain', but doesn't go beyond the schema's own documentation significantly.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states it extracts hyperlinks from a web page, resolves them to absolute URLs, and optionally restricts to same domain. It effectively distinguishes from sibling tools like crawl_site (multi-page) and extract_by_selector (CSS-based).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for extracting links from a single page but lacks explicit guidance on when to use this vs. siblings (e.g., craw_site for multiple pages). No mention of prerequisites or when not to use.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

fetch_pageA

Fetch a single web page and return its readable content as Markdown, plain text, or raw HTML. Automatically renders JavaScript-heavy pages with a headless browser when needed.

ParametersJSON Schema

Name	Required	Description	Default
`url`	Yes	Page URL (http/https). Scheme-less input like `example.com` is allowed.
`format`	No	Output format for the page content.	markdown
`render`	No	auto=fast static fetch, fall back to headless browser if the page needs JS; static=never use a browser; browser=always render with Playwright (handles SPAs).	auto
`max_chars`	No	Cap on returned characters (default 20000).

TDQS

A4/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description discloses the automatic use of a headless browser for JavaScript-heavy pages, a key behavioral trait. However, since there are no annotations, it could further clarify that it is a read-only operation.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences, front-loaded with the core purpose and output formats, with no extraneous information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description adequately covers the return value (readable content in three formats) and highlights JS rendering. It does not mention edge cases or error handling, but for a simple fetch tool it is sufficient.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3. The description adds no additional parameter details beyond what the schema already provides.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool fetches a single web page and returns content in multiple formats. It distinguishes itself from sibling tools like crawl_site (multiple pages) and extract_by_selector (specific elements).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for single-page content retrieval but does not explicitly mention when to use this tool versus siblings like crawl_site or extract_links. No alternatives are suggested.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

4 tool updatesv1.0.0Jun 4, 2026
- First observedcrawl_site
- First observedextract_by_selector
- First observedextract_links
- First observedfetch_page

Install Server

RayrunBuild, deploy, and operate MCP servers without becoming an infrastructure team.ray.runAd

TDQS

A3.9/5.0

Disambiguation5/5

Each tool has a clearly distinct purpose: crawl_site handles multi-page crawling with summaries, fetch_page retrieves single-page content in various formats, extract_links focuses on hyperlinks, and extract_by_selector targets specific elements via CSS selectors. There is no overlap or ambiguity between them.

Naming Consistency5/5

All tool names follow a consistent verb_noun pattern in snake_case (e.g., crawl_site, extract_links). The naming is predictable and intuitive, making it easy for an agent to infer functionality from the name.

Tool Count4/5

With 4 tools, the server covers the core operations of web crawling and extraction without being overly large. The count is slightly on the lower side but still well-scoped for its purpose, and each tool earns its place.

Completeness4/5

The tool set covers the essential workflow: fetching pages, crawling sites, extracting links, and extracting specific elements. Minor gaps exist, such as the absence of tools for setting request headers or handling authentication, but the core functionality is solid.

Maintenance

ActivityInactive

ResponsivenessSyncing

Resources

GitHub Repository

Need Help?

Related Servers

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

MESSORA A
dev.messora
Scrape, crawl and search the web for AI agents via MCP.
Web Scraping Browser Automation Search
51
luminati-io-brightdata-mcp
ai.smithery
One MCP for the Web. Easily search, crawl, navigate, and extract websites without getting blocked.…
Browser Automation Web Scraping Search
2
mcp-webextratorOAuth
io.github.AceDataCloud
MCP server for web extraction and rendering via AceDataCloud WebExtrator
Web Scraping Browser Automation
2
VPN.FAIL MCP A
io.github.vpnfail
Free remote MCP server for fetching public web pages through a rotating proxy pool.
Web Scraping Browser Automation
31

Related MCP Servers

MCP Web Scraper
Web Scraping Browser Automation Developer Tools
navin4078
F
license
Not graded
quality
D
maintenance
A lightweight web scraping server that allows Claude Desktop users to extract various types of data from websites, including text, links, images, tables, headlines, and metadata using CSS selectors.
Updated a year ago (2025-06-10 12:37 UTC)
4
-
turbowebfetch
Browser Automation Web Scraping
aza-ali
A
license
A
quality
B
maintenance
MCP server that lets Claude Code fetch web content using real Chrome browsers. Renders JavaScript-heavy pages, handles bot mitigation, and runs up to 14 parallel browsers locally with zero API keys. Makes outbound HTTP requests only to URLs the user explicitly asks Claude to fetch.
Updated 4 months ago (2026-05-15 17:28 UTC)
2
14
91
MIT
agentfetch
Browser Automation Web Scraping Search
SID1ART
A
license
Not graded
quality
B
maintenance
An open-source web retrieval MCP server that fetches, crawls, and searches the web, returning clean markdown for AI agents. It integrates with Claude MCP, LangChain, and other frameworks for agentic web access.
Updated 3 months ago (2026-06-19 08:41 UTC)
1
MIT
fast-webfetch-mcp
Browser Automation Web Scraping
NikkeTryHard
A
license
Not graded
quality
B
maintenance
A high-performance MCP server for web fetching in Claude Code using Firecrawl backend with automatic fallback.
Updated 14 days ago (2026-08-22 22:17 UTC)
2
MIT

Latest Blog Posts

Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)
By Om-Shree-0709 on July 25, 2026.
mcp
Agent Identity
OAuth 2.1
Your AI Chatbot Just Exposed Your CEO's Salary to an Intern
By Om-Shree-0709 on July 2, 2026.
Agent Identity
MCP Security
OAuth Delegation
Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)
By Om-Shree-0709 on June 30, 2026.
Agentic Ai
Prompt Injection
WebAssembly

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/shadab15github/crawler-mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server

crawler-mcp

Tools

Setup

Run

Use it as a Claude connector

Claude Desktop (local, stdio)

Remote HTTP connector (claude.ai / Desktop "Add custom connector")

Configuration (.env)

Notes

Available Tools

Tool Schema Changelog

Maintenance

Resources

Looking for Admin?

Related MCP Connectors

Related MCP Servers

Latest Blog Posts

MCP directory API

Configuration (`.env`)