What can you do with this server?

This server lets agents search the web and extract page content as Markdown via MCP tools. * search(query, limit=5): SearXNG-first search with automatic Tavily keyless fallback; returns title/url/description results, with optional semantic reranking and rate limiting. * extract(url, max_words=8000): fetches a page, denoises navigation/ads, converts to clean Markdown, retries on failure; returns title/markdown/length.

Which integrations are available for this server?

Provides web search through a self-hosted SearXNG meta-search instance, with automatic fallback to Tavily keyless API, shared caching, and optional semantic reranking of results.

How do I use CrawlEyes?

1. Click on "Install Server". 2. Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state. 3. In the chat, type @ followed by the MCP server name and your instructions, e.g., "@CrawlEyes Find the latest MCP Python SDK docs and extract the top result" That's it! The server will respond to your query, and you can continue using it as needed. Here is a step-by-step guide with screenshots.

CrawlEyes

by waiky-github

Overview Schema Related Servers Score Discussions

Python

Hybrid

CrawlEyes — Web Scraping & Search Toolkit for AI Agents

GitHub stars GitHub license Python MCP MIT

CrawlEyes gives AI agents reliable full-text extraction (web_extract) and robust search (web_search) backends — the "eyes" that let agents see and read the web. Built and tested against Hermes Agent.

Also ships as a standard MCP server, so any MCP client (Claude Desktop, Cursor, other agents) can reuse the same search + extraction capabilities.

CrawlEyes demo

Why CrawlEyes?

Most agent toolkits cover one slice of the pipeline. CrawlEyes is the rare all-in-one that you can actually run behind the Great Firewall without external accounts.

	Typical agent toolkit	CrawlEyes
🔍 Search	API key required, often blocked in CN	✅ SearXNG (self-hosted) + Tavily keyless fallback — zero config, zero key
📄 Extraction	Separate scraper, or Firecrawl SaaS	✅ Built-in Crawl4AI full-text extract, ~89% noise removal
🧠 Semantic rerank	Rarely included	✅ Local fastembed rerank — no torch, ~50MB model
🔌 MCP server	Often missing	✅ Standard MCP tools (`search` + `extract` + `deep_research` + `sitemap`), any client
🌐 China-friendly	Mostly English/GFW-blocked	✅ Tested on a real mainland China server (baidu + yandex)

Zero API keys. Zero external accounts. One command. CrawlEyes is the only toolkit in this space that combines search + extraction + semantic reranking + MCP in a single, China-friendly, self-hosted package.

Related MCP server: Web Search MCP Server

Features

Capability	Where	Why it matters
Full-text extraction	`scripts/crawl4ai_cli.py`	Headless-browser scraping → clean Markdown; handles ~80% of JS/dynamic/UA-blocked pages
Content denoising (P1)	`crawl4ai_cli.py --noise-filter`	Prunes nav/ads/comments via Crawl4AI's `PruningContentFilter` — measured 24.6k→2.8k chars (~89% noise removed) on a typical article
Retry with backoff (P3)	`crawl4ai_cli.py --retry N`	Exponential backoff (1s/2s/4s) on transient failures
Browser session reuse (P4)	`crawl4ai_cli.py --session NAME`	Reuses the browser context across scrapes in one process — no cold-start per URL
Keyword-focused extraction	`crawl4ai_cli.py --bm25 KEYWORD`	Keeps only paragraphs relevant to a keyword (experimental — BM25 is English-centric; works best on English docs)
Search (primary)	SearXNG (self-hosted meta-search)	Privacy-friendly search aggregator
Search (fallback)	Tavily keyless API	Zero-config, no-key fallback when SearXNG is down/empty
Search orchestration	`plugins/searxng-tavily/`	Hermes plugin provider: SearXNG first → auto-fallback to Tavily keyless; three-state circuit breaker (3 fails → 60s cooldown → half-open) + shared SQLite cache (TTL 3600s)
Semantic reranking (P2)	`scripts/crawl_search_standalone.py`	Local embedding rerank of search results with `fastembed` + `BAAI/bge-small-zh-v1.5` (512-dim, no torch dependency, ~50MB, cached) — puts relevant results first. Measured: crawler-relevant items 0.817/0.732 float to top, irrelevant 0.302/0.139 sink
MCP server (P5)	`scripts/mcp_crawl_server.py`	Exposes `search` + `extract` + `deep_research` + `sitemap` as standard MCP tools (stdio default, or streamable-http for remote clients). Works in any MCP client, no Hermes dependency. Extracted content is sanitized against prompt-injection (strips invisible chars + prompt-hijack lines). Unified rate limiting + exponential backoff guard every tool (sliding window, per-tool cost) so concurrent agent calls can't hammer downstream services
Sitemap discovery (P1)	`crawleyes/sitemap.py`	`sitemap(origin)` → parses `sitemap.xml` (plain / gzip / index-recursion) with `robots.txt` fallback, returns a deduped URL map. Zero-key way to discover a site's URL surface for whole-site fetch or deep-research seeding
Multi-format extract (P0)	`extract(..., format=)`	`markdown` (default) / `fit` (denoised) / `raw` (unfiltered) / `markdown_with_citations` — pick the level of cleanup you need
RAG-ready interfaces	`crawleyes/rag.py`	One-liners `markdown(url)` / `search_markdown(query)` → clean, sanitized, LLM-ready Markdown for RAG corpora
Deep research	`crawleyes/deep_research.py`	`deep_research(topic)` → decomposes topic into sub-questions → searches → extracts → synthesizes a cited Markdown report. Optional LLM (any OpenAI-compatible endpoint); degrades to evidence-aggregate mode without one
Verification	`scripts/`	Clean subprocess scripts to verify each backend end-to-end per Hermes profile

Project layout

plugins/searxng-tavily/   Hermes web-search provider plugin (SearXNG → Tavily keyless fallback)
                          + three-state circuit breaker + shared SQLite cache
scripts/
  crawl4ai_cli.py          Universal scraping CLI (URL → Markdown), with denoise/retry/session/BM25
  crawl_search_standalone.py  Standalone search (SearXNG → Tavily) + optional semantic rerank.
                             No Hermes dependency — usable anywhere, powers the MCP server.
  mcp_crawl_server.py      Standard MCP server exposing search + extract + deep_research + sitemap (stdio)
  single_env_check.py      Verify crawl4ai provider registered+available+extracts (one profile)
  verify_searxng_tavily.py Verify searxng-tavily provider: normal path + forced fallback
  agent_link_check.py      Verify full agent tool chain: web_search_tool dispatch + logs

Quick start

1. Install Crawl4AI (China-friendly mirrors)

python3 -m venv .venv
# Use Tsinghua PyPI mirror for speed (or any mirror you prefer)
.venv/bin/pip install -i https://pypi.tuna.tsinghua.edu.cn/simple crawl4ai
# Playwright browser kernel — use npmmirror binary mirror if cdn.playwright.dev is blocked
PLAYWRIGHT_DOWNLOAD_HOST=https://registry.npmmirror.com/-/binary/playwright \
  .venv/bin/python -m playwright install chromium
.venv/bin/crawl4ai-setup

2. Scrape a page

.venv/bin/python scripts/crawl4ai_cli.py https://example.com          # stdout Markdown
.venv/bin/python scripts/crawl4ai_cli.py https://example.com -o out.md  # to file
.venv/bin/python scripts/crawl4ai_cli.py URL --text --max-words 5000   # plain text, truncated

# Multi-format extraction (markdown|fit|raw|markdown_with_citations)
.venv/bin/python scripts/crawl4ai_cli.py URL --format raw              # unfiltered source markdown
.venv/bin/python scripts/crawl4ai_cli.py URL --format markdown_with_citations  # + source URLs

# Respect robots.txt (opt-in, default off)
.venv/bin/python scripts/crawl4ai_cli.py URL --respect-robots

# Denoise nav/ads + retry 3x + reuse session across scrapes
.venv/bin/python scripts/crawl4ai_cli.py URL --noise-filter --retry 3 --session s1

3. Use the search + rerank (standalone, no Hermes)

# Optional: local semantic rerank of results (fastembed + bge-small-zh, auto-downloaded)
.venv/bin/pip install -i https://pypi.tuna.tsinghua.edu.cn/simple fastembed

# SearXNG first, Tavily keyless fallback, then rerank
SEARXNG_URL=https://your-searxng .venv/bin/python -c "
import sys; sys.path.insert(0, 'scripts')
from crawl_search_standalone import CrawlSearch
r = CrawlSearch(rerank=True).search('your query')
print(r['data']['web'])"

China-network note: the embedding model downloads from HuggingFace, which is blocked on mainland networks. Set HF_ENDPOINT=https://hf-mirror.com and HF_HUB_DISABLE_XET=1 (hf-mirror doesn't support the xet protocol and returns 401 without this).

4. Run as an MCP server (any client)

# Any MCP client can connect via stdio (default):
.venv/bin/python scripts/mcp_crawl_server.py
# Exposes tools:
#   search(query, limit)                 - SearXNG → Tavily keyless, rerank, retry+rate-limit
#   extract(url, max_words, format)      - markdown|fit|raw|markdown_with_citations
#   deep_research(topic, num_questions)  - multi-round cited report
#   sitemap(origin, max_urls)            - URL map from sitemap.xml / robots.txt

# Or serve over HTTP (streamable-http) for remote clients:
.venv/bin/python -m crawleyes.mcp_crawl_server --transport http --port 8765 --host 127.0.0.1
#   → clients connect to http://127.0.0.1:8765/mcp
#   (host/port configurable; default 127.0.0.1:8765)

For Hermes specifically, add to config.yaml:

mcp_servers:
  crawl:
    command: "/path/to/crawl/.venv/bin/python"
    args: ["/path/to/crawl/scripts/mcp_crawl_server.py"]
    timeout: 90
    connect_timeout: 60

4b. Firecrawl-compatible `/scrape` endpoint

Already using Firecrawl's Python SDK? Point it at CrawlEyes and keep your code:

.venv/bin/python -m crawleyes.firecrawl_api --port 8899 --host 127.0.0.1
#   POST /v2/scrape  →  { success, data: { markdown, metadata } }
#   GET  /healthz    →  health check

from firecrawl import Firecrawl
fc = Firecrawl(api_url="http://127.0.0.1:8899", api_key="ignored")
doc = fc.scrape(url="https://example.com")   # → { markdown, metadata }

This is a pragmatic subset of the Firecrawl API — the core /scrape contract (success + data.markdown + data.metadata), backed by CrawlEyes' own extraction engine. It does not implement Firecrawl's async /crawl queue, /search, or /map — see the design notes for the rationale.

5. Install the search plugin (Hermes)

Copy plugins/searxng-tavily/ into a Hermes plugins dir, then:

hermes plugins enable web/searxng-tavily
hermes config set web.search_backend searxng-tavily

Set SEARXNG_URL in your Hermes profile .env to point at your SearXNG instance. If unset or unreachable, the provider automatically falls back to the Tavily keyless API (no API key required).

Note: the plugin only takes effect for newly started agent sessions.

6. Verify

# Requires the Hermes source tree + its venv
venv/bin/python scripts/verify_searxng_tavily.py $HERMES_HOME
venv/bin/python scripts/agent_link_check.py $HERMES_HOME

Design notes

Layered composition: no single tool covers everything. Crawl4AI handles extraction; SearXNG + Tavily cover search; each layer has a tested fallback.
Tavily keyless works with zero configuration and no account — a cheap resilience net for the whole search path.
Circuit breaker is SearXNG-only: a Tavily fallback success does not reset the breaker (otherwise it would never trip). record_success() is only called when SearXNG itself succeeds.
Shared SQLite cache lives in the real user home (via pwd.getpwuid, not $HOME — which Hermes profiles override), so all profiles share one cache. WAL + 5s timeout + try/except degrade-to-no-cache under concurrency.
Semantic rerank is cheap: fastembed (ONNX) avoids the ~2GB torch dependency; model loads in ~0.6s once cached, embeddings in ~50ms.
MCP server is standalone: it does not import Hermes internals, so it runs on any Python 3.12 env and serves any MCP client.
MCP transport is dual: stdio (default, standard MCP clients) or streamable-http (--transport http), so a single codebase serves both local process and remote HTTP clients.
Unified rate limiting is layered: MCP tools and deep-research's internal search/extract all share one sliding-window limiter (per-tool cost), so concurrent agent fan-out can't hammer SearXNG/Tavily/Crawl4AI even through multi-round deep research.

Credits & inspiration

This project builds on a set of excellent open-source tools. All code here is an independent implementation (no copied code), but the ideas and interfaces are drawn from the following projects — full credit to their authors:

Feature in this repo	Inspired by	License
Extraction engine (Crawl4AI wrapper)	Crawl4AI — direct dependency	Apache-2.0
Content denoising (P1)	Readability, GeneralNewsExtractor (idea)	Apache-2.0 / MIT
Semantic reranking (P2)	Vane, Perplexica (idea)	MIT / MIT
Retry with backoff (P3)	Crawlee (idea)	Apache-2.0
Browser session reuse (P4)	camoufox (idea)	MIT
MCP server (P5)	playwright-mcp, exa-mcp-server (idea)	Apache-2.0 / MIT
Search orchestration / fallback	SearXNG — self-hosted (official Docker image, no source modification), accessed via HTTP API only · Tavily keyless	AGPL-3.0 (server software, not linked/embedded) / proprietary API

Design independence: the implementations here are written from scratch — we studied the above projects' approaches (denoising thresholds, rerank pipelines, backoff strategies, MCP tool patterns) but did not copy their source code. Dependencies are declared in requirements.txt. If you believe any attribution is missing or incorrect, please open an issue.

Compliance

CrawlEyes is a general-purpose fetch toolkit for legitimate research and personal use. It deliberately does not include proxy pools, fingerprint rotation, or CAPTCHA-solving (anti-scraping evasion) — those are out of scope.

Robots.txt is opt-in (default off): pass respect_robots=True to extract / markdown (or --respect-robots on the CLI) to check each target's robots.txt (RFC 9309) and refuse URLs it explicitly disallows. It's default-off so legitimate scraping isn't silently blocked by aggressive or broken robots rules — compliance is the caller's informed choice per use case. Always review each site's terms of service before scraping at scale.

License

MIT — see LICENSE.

Available Tools

2 tools

extractB

抓取网页正文为 Markdown。自动去噪（过滤导航/广告），失败自动重试。 Returns JSON with title/markdown/length.

ParametersJSON Schema

Name	Required	Description	Default
`url`	Yes
`max_words`	No

Output Schema

ParametersJSON Schema

Name	Required	Description
`result`	Yes

TDQS

B3.3/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations present, the description carries the behavioral disclosure burden. It usefully discloses automatic denoising (filtering navigation/ads), automatic retry on failure, and the JSON return format with title/markdown/length. These go beyond the obvious, though it omits rate limits, auth, or failure edge cases.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences, front-loaded with the core purpose, then adds denoising, retry, and return format details. Every sentence earns its place with zero filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple extraction tool with an output schema, the description covers the core function and return format. However, it omits parameter semantics and provides no usage context relative to its sibling tool, leaving the agent to infer when and how to invoke it fully.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate for parameter meaning. It does not explain 'max_words' at all, and only indirectly implies 'url' via the extraction context. The description focuses on behavior and output, leaving parameter semantics unexplained.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's function: extracting webpage main content as Markdown, with denoising and retry behavior. However, it does not explicitly distinguish itself from the sibling tool 'search', so it stops short of full sibling differentiation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided on when to use 'extract' versus 'search' or any other alternative. The description explains what the tool does but gives no context about when to select it, prerequisites, or exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

searchB

搜索网页。SearXNG 优先，失败自动 fallback Tavily keyless（无需 API key）。 Returns JSON with title/url/description for each result.

ParametersJSON Schema

Name	Required	Description	Default
`limit`	No
`query`	Yes

Output Schema

ParametersJSON Schema

Name	Required	Description
`result`	Yes

TDQS

B3.1/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full behavioral disclosure burden. It does reveal the SearXNG-first fallback to keyless Tavily and the JSON return fields, which is useful. However, it does not mention failure modes, rate limits, result ordering, or how the limit parameter affects behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is very short, front-loaded with the core operation, and every sentence carries relevant information. The fallback behavior and output shape are packed into two compact sentences with no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the simple schema and presence of an output schema, the description covers the main search behavior and return format. However, it omits limit semantics and usage guidance versus the sibling tool, leaving an agent to infer those details.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It implies 'query' via '搜索网页', but it never explains 'limit' or how it constrains the result set. The output format is mentioned, but neither parameter gets meaningful semantic detail.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states '搜索网页' (search web pages) and specifies the return format as JSON with title/url/description. It identifies a concrete operation with a resource, though it does not explicitly contrast itself with the sibling tool 'extract', so it stops short of a 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no guidance on when to use this tool versus the sibling 'extract', nor any exclusions or prerequisites. The intent is implied by the name and '搜索网页', but the description never states usage conditions or alternative selection.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

2 tool updatesv0.1.0Aug 31, 2026
- First observedextract
- First observedsearch

Install Server

RayrunBuild, deploy, and operate MCP servers without becoming an infrastructure team.ray.runAd

TDQS

B3.4/5.0

Disambiguation5/5

Search and extract have clearly distinct purposes: one discovers URLs via web search, the other fetches and cleans page content. There is no overlap or ambiguity between the two tools.

Naming Consistency5/5

Both tools use simple, consistent single-word imperative verbs: search and extract. The naming style is uniform and predictable.

Tool Count3/5

Two tools is borderline thin for a server named CrawlEyes. The pair is coherent for a basic search-then-extract workflow, but the surface feels minimal for a crawling-focused server.

Completeness3/5

The core web research pipeline of searching and extracting content is covered, but there is no explicit crawling, pagination, or multi-page navigation capability implied by the server name. Agents can work around this by chaining searches and extracts, but it is a notable gap.

Maintenance

ActivityMaintained

ResponsivenessNo issues

Resources

GitHub Repository

Need Help?

Related Servers

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

MESSORA A
dev.messora
Scrape, crawl and search the web for AI agents via MCP.
Web Scraping Browser Automation Search
51
pixserp A
com.pixserp
Live AI-native web search with citations. One tool for every MCP client. Flat per-request pricing.
Search Browser Automation
12
Not Human Search A
ai.nothumansearch
Search the agentic web. 4,100+ sites, 11 tools incl. check_url + verify_mcp for probe-before-use.
Search Browser Automation Web Scraping
143
Ninelayer
in.ninelayer
Agent-native search engine with live web research optimized for AI agents.
Search Agent Orchestration
2

Related MCP Servers

websearch-skill
Search Web Scraping Browser Automation
hec-ovi
A
license
A
quality
B
maintenance
Enables AI agents to perform multi-engine web search, fetch web pages, and extract clean Markdown content via MCP, with no API keys required.
Updated 24 days ago (2026-08-13 14:14 UTC)
3
5
MIT
Web Search MCP Server
Browser Automation Web Scraping Search
lalit9168
F
license
Not graded
quality
C
maintenance
Enables AI agents to perform web searches, extract webpage content, and conduct end-to-end search-and-extract operations using multiple search providers and content extraction methods.
Updated 2 months ago (2026-07-01 07:18 UTC)
-
qsearch
Search RAG Systems Knowledge & Memory
theYahia
A
license
Not graded
quality
B
maintenance
Enables AI agents to perform web searches with full content retrieval and multi-engine provenance, including trust scoring and local corpus persistence, via MCP integration.
Updated 3 days ago (2026-09-03 04:54 UTC)
3
2
Apache 2.0
jiro
Browser Automation Web Scraping Search
DevAnimecx
A
license
Not graded
quality
A
maintenance
Enables AI agents to perform live web searches across 9 engines, scrape web pages into clean formats, and run agentic research with citations via MCP.
Updated 13 hours ago (2026-09-05 18:42 UTC)
1
MIT

Latest Blog Posts

Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)
By Om-Shree-0709 on July 25, 2026.
mcp
Agent Identity
OAuth 2.1
Your AI Chatbot Just Exposed Your CEO's Salary to an Intern
By Om-Shree-0709 on July 2, 2026.
Agent Identity
MCP Security
OAuth Delegation
Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)
By Om-Shree-0709 on June 30, 2026.
Agentic Ai
Prompt Injection
WebAssembly

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/waiky-github/CrawlEyes'

If you have feedback or need assistance with the MCP directory API, please join our Discord server

CrawlEyes — Web Scraping & Search Toolkit for AI Agents

Why CrawlEyes?

Features

Project layout

Quick start

1. Install Crawl4AI (China-friendly mirrors)

2. Scrape a page

3. Use the search + rerank (standalone, no Hermes)

4. Run as an MCP server (any client)

4b. Firecrawl-compatible /scrape endpoint

5. Install the search plugin (Hermes)

6. Verify

Design notes

Credits & inspiration

Compliance

License

Available Tools

Output Schema

TDQS

Output Schema

TDQS

Tool Schema Changelog

TDQS

Maintenance

Resources

Looking for Admin?

Related MCP Connectors

Related MCP Servers

websearch-skill

Web Search MCP Server

qsearch

jiro

Latest Blog Posts

MCP directory API

4b. Firecrawl-compatible `/scrape` endpoint