Skip to main content
Glama
Prog-up

Web Scraper MCP

by Prog-up

Web Scraper MCP

Самодостаточный сервер Model Context Protocol, который предоставляет LLM-клиенту (Claude Code, Cursor, ChatGPT) тот же набор инструментов, что и платные сервисы скрапинга — scrape, crawl, map, search, extract, interact, deep_research — работающий полностью на вашем собственном оборудовании.

Никаких платных прокси/CAPTCHA-сервисов: антибот самодостаточен (headless Chromium + playwright-stealth, robots.txt, вежливое ограничение частоты запросов). Защищённые сайты всё ещё могут блокировать; см. Ограничения.

Tools

Tool

Что делает

scrape

Один URL → чистый markdown (без шаблонного кода). Сначала статический, автоматический переход на браузер для JS-страниц.

crawl / check_crawl_status

Фоновая задача обхода BFS (дедупликация, ограничения по глубине/страницам); опрос результатов.

map

Список ссылок на странице (опционально только тот же домен) — решите, что обходить.

search

Веб-поиск. Подключаемый бэкенд: DuckDuckGo (по умолчанию), SearXNG, Brave или Tavily.

extract

Получить страницу и извлечь структурированный JSON, соответствующий вашей схеме, с помощью LLM.

browser_navigate / browser_act / browser_close

Управление постоянной сессией браузера (клик/заполнение/нажатие) с использованием дешёвых по токенам ARIA-снимков.

deep_research

Поиск → чтение лучших источников → возврат отчёта с цитатами.

extract и deep_research требуют ANTHROPIC_API_KEY.

Related MCP server: Universal Web Data Extraction Platform

Quick start

uv sync                      # install deps (uses the pinned uv.lock)
uv run playwright install chromium
export SCRAPER_AUTH_TOKEN=$(openssl rand -hex 32)
uv run web-scraper-mcp       # HTTP server on http://127.0.0.1:8000/mcp

Stdio (локально, для настольного клиента): SCRAPER_TRANSPORT=stdio uv run web-scraper-mcp.

Docker

Самый быстрый способ начать — загрузить готовый образ с DockerHub.

# Pull the latest image
docker pull PROG_UP_USERNAME/web-scraper-mcp:latest

# Run the container (with Anthropic / Claude)
docker run -p 8000:8000 \
  -e SCRAPER_AUTH_TOKEN=your_secure_token_here \
  -e ANTHROPIC_API_KEY=sk-ant-api03-... \
  PROG_UP_USERNAME/web-scraper-mcp:latest

# Or run the container over stdio (useful for local MCP clients)
docker run -i --rm \
  -e SCRAPER_TRANSPORT=stdio \
  -e SCRAPER_AUTH_TOKEN=your_secure_token_here \
  PROG_UP_USERNAME/web-scraper-mcp:latest

(Убедитесь, что вы заменили PROG_UP_USERNAME на ваше реальное имя пользователя DockerHub).

Using Local Models (Ollama) & Context Windows

Если вы предпочитаете запускать модели локально вместо использования API Anthropic, сервер полностью поддерживает Ollama в качестве альтернативного бэкенда для инструментов extract и deep_research.

docker run -p 8000:8000 \
  -e SCRAPER_AUTH_TOKEN=your_secure_token_here \
  -e SCRAPER_OLLAMA_HOST=http://host.docker.internal:11434 \
  -e SCRAPER_EXTRACT_MODEL=qwen3.5:2b \
  -e SCRAPER_RESEARCH_MODEL=qwen3.5:2b \
  PROG_UP_USERNAME/web-scraper-mcp:latest

[!WARNING] Контекстные окна критически важны! Веб-скрапинг создаёт огромное количество Markdown. extract может отправлять до 100 000 символов, а deep_research — до 16 000 символов в LLM.

По умолчанию Claude изначально обрабатывает большие контексты. Однако контекстное окно Ollama (num_ctx) часто настроено всего на 2 048 токенов. Если вы передадите огромную страницу Википедии локальной модели, она молча обрежет промпт (отбросив ваши инструкции) и вернёт пустые строки!

Мы автоматически передаём "num_ctx": 32768 в Ollama в полезной нагрузке API, чтобы предотвратить это обрезание. Убедитесь, что на вашей локальной машине достаточно RAM/VRAM для поддержки контекстного окна 32K при использовании Ollama!

Register in a client (mcp.json)

Если запуск через HTTP:

{
  "mcpServers": {
    "web-scraper": {
      "url": "http://127.0.0.1:8000/mcp",
      "headers": { "Authorization": "Bearer your_secure_token_here" }
    }
  }
}

Если запуск через stdio (Docker):

{
  "mcpServers": {
    "web-scraper": {
      "command": "docker",
      "args": [
        "run", "-i", "--rm",
        "-e", "SCRAPER_TRANSPORT=stdio",
        "-e", "SCRAPER_OLLAMA_HOST=http://host.docker.internal:11434",
        "-e", "SCRAPER_EXTRACT_MODEL=qwen3.5:2b",
        "-e", "SCRAPER_RESEARCH_MODEL=qwen3.5:2b",
        "PROG_UP_USERNAME/web-scraper-mcp:latest"
      ]
    }
  }
}

Configuration

Все настройки — переменные окружения (префикс SCRAPER_) или файл .env — см. .env.example.

Var

Default

Примечания

SCRAPER_AUTH_TOKEN

(unset)

Bearer-токен для HTTP-эндпоинта. Обязателен для любого сетевого развёртывания; не задан = без аутентификации (предупреждение).

SCRAPER_HOST / SCRAPER_PORT

127.0.0.1 / 8000

Адрес привязки. Docker-образ устанавливает хост 0.0.0.0.

SCRAPER_MAX_CONCURRENT_PAGES

8

Максимум одновременных headless-страниц (ограничение по RAM/CPU).

SCRAPER_MAX_CRAWL_PAGES / _DEPTH

100 / 3

Жёсткие пределы для задач обхода.

SCRAPER_PER_DOMAIN_DELAY_S

1.0

Вежливое ограничение частоты запросов на домен.

SCRAPER_RESPECT_ROBOTS

true

Уважать robots.txt.

SCRAPER_ALLOW_PRIVATE_NETWORKS

false

Оставьте false — при true отключает защиту SSRF.

ANTHROPIC_API_KEY

(unset)

Включает extract / deep_research.

SCRAPER_SEARXNG_URL, BRAVE_API_KEY, TAVILY_API_KEY

(unset)

Необязательные поисковые бэкенды (первый заданный имеет приоритет, иначе DuckDuckGo).

Security

  • Защита SSRF — каждый полученный URL (и каждый переход по редиректу) разрешается через DNS и отклоняется, если он указывает на частный / loopback / link-local / cloud-metadata адрес. Браузер также прерывает запросы подресурсов к частным IP.

  • Аутентификация — bearer-токен на HTTP-транспорте; по умолчанию привязка к localhost.

  • Ограничения ресурсов — размер ответа, таймаут, параллельность страниц, лимиты страниц/глубины обхода для защиты хоста.

  • robots.txt + ограничение частоты включены по умолчанию.

  • Контейнер — запускается от непривилегированного пользователя; секреты только через переменные окружения.

Supply chain — verifying the image

CI подписывает образ без ключей с помощью cosign (Sigstore), используя OIDC-идентичность GitLab, и прикрепляет аттестацию SPDX SBOM. Проверьте перед запуском:

cosign verify \
  --certificate-oidc-issuer https://gitlab.cri.epita.fr \
  --certificate-identity-regexp 'https://gitlab.cri.epita.fr/enzo.juhel/web-scraper//.*' \
  registry.gitlab.cri.epita.fr/enzo.juhel/web-scraper@sha256:...

Benchmark

benchmarks/run.py оценивает наши scrape/extract на публичных наборах данных и базовом уровне Crawl4AI, выводя таблицу результатов (F1/точность по типам страниц + раздел «Ограничения»). Запустите локально или через ручную задачу CI benchmark:

uv run python benchmarks/run.py --output scorecard.md

Limitations

  • Нет платных прокси/CAPTCHA: сильно защищённые сайты (LinkedIn, Amazon, Cloudflare challenges) иногда блокируют нас. Таблица результатов бенчмарка показывает, где именно.

  • Извлечение основного контента хорошо работает на статьях, хуже на форумах / страницах товаров / списках (известное свойство всех экстракторов).

  • Состояние обхода/сессии в памяти — однопроцессное, однопользовательское по замыслу.

Development

uv run pre-commit install        # local lint/secret hooks
uv run ruff check . && uv run ruff format --check .
uv run mypy src
uv run pytest -q

Available Tools

4 tools
deep_researchA

Search the web, read the top sources, and return a cited synthesis report.

ParametersJSON Schema
NameRequiredDescriptionDefault
queryYesThe research question.
max_sourcesNoHow many top results to read.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A3.8/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It discloses the general workflow (searching the web, reading top sources, producing a cited report) but does not mention read-only behavior, potential latency, rate limits, or any limitations. The description is truthful and somewhat informative, but leaves important operational behavior unaddressed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single sentence with a clear chronological structure, no filler, and immediate front-loading of the core action ('Search the web'). Every phrase earns its place: 'read the top sources' and 'cited synthesis report' are distinct and necessary components. This is appropriately concise for the tool's complexity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the two fully documented parameters and the presence of an output schema (likely covering the report structure), the description covers the essential process and outcome well. What is missing is more explicit context about when to use the tool versus siblings and a few behavioral notes, but the combination of description and schema is largely sufficient for an agent to invoke it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%: both 'query' and 'max_sources' have descriptions in the schema. The tool description's phrasing ('research question', 'top results') mirrors the schema descriptions without adding new meaning about formats, constraints, or interpretation. This meets the baseline of 3 but does not exceed what structured data already provides.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb sequence ('Search', 'read', 'return') with a clear deliverable ('a cited synthesis report'). It unambiguously identifies the tool's function and naturally distinguishes deep_research from siblings like search (just web results) or scrape/extract (page-level content gathering). The multi-step process is explicit and not a tautology.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for comprehensive research with cited synthesis, but it never explicitly states when to prefer this over search, scrape, or extract, nor does it give any 'instead of' guidance. Context makes the intended use reasonably clear, yet there are no exclusions or alternative routing as seen in higher-quality definitions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

extractB

Extract structured data (JSON matching json_schema) or a text answer from a page.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesURL to extract from.
promptNoNatural-language extraction instruction.
renderNoForce a browser render.
json_schemaNoJSON Schema describing the fields to extract (recommended).

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

B3.2/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description must carry the behavioral burden. It does disclose the two operating modes (structured JSON versus text answer), which is meaningful. However, it does not mention rendering defaults, page-fetch behavior, rate limits, or side effects. The render parameter is defined in the schema, so some of that context is available, but the description itself stays minimal.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, front-loaded sentence with no wasted words. It could be slightly better structured by explicitly separating the two modes, but as written it is efficient and easy to parse.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The output schema exists, so return-value details are covered elsewhere, and the input schema is fully documented. The main gaps are the lack of choice guidance between prompt and json_schema, and no indication of when to use extract instead of scrape. These gaps make the description adequate but not complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already defines url, prompt, render, and json_schema. The description adds a high-level mapping by linking json_schema to structured JSON and implying the text answer comes from the prompt, but it does not add much beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('extract'), the target resource ('a page'), and the two possible outputs: JSON matching json_schema or a text answer. It does not explicitly distinguish itself from sibling tools like scrape or search, though the verb and output specification make the core purpose clear.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives no guidance on when to use extract versus scrape, search, or deep_research, and no exclusions or conditions. The only context is 'from a page,' which is too thin to help an agent choose between this and its siblings.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

scrapeA

Scrape a single URL into clean markdown (boilerplate/ads stripped).

Tries a fast static fetch first and falls back to a stealth headless browser automatically when the page looks JS-gated.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesThe URL to scrape (http/https).
renderNoForce a headless browser render (for JS-heavy pages).
include_linksNoAlso return all links found on the page.
include_raw_htmlNoAlso return the raw HTML (large).

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the burden of behavioral disclosure. It clearly reveals the two-phase fetch strategy, the automatic fallback to headless browsing, and the output transformation to clean markdown. It does not discuss rate limits, authentication, or failure behavior, but the core operational traits are transparent.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences with no fluff, front-loading the primary purpose before the fallback behavior. Every sentence contributes useful information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The presence of an output schema means return values do not need to be explained in the description. Combined with fully documented parameters and a clear explanation of the scraping strategy, the definition is largely complete, though it omits failure semantics and any rate-limit or authentication caveats.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so all four parameters are already documented in the input schema. The tool description adds no additional parameter-level meaning, which matches the baseline for fully-covered schemas.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb ('Scrape a single URL') and a concrete outcome ('into clean markdown'), making the tool's purpose immediately clear. The boilerplate/ads-stripping detail further distinguishes it from general fetch or research siblings.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It explains that a fast static fetch is attempted first and that a stealth headless browser is used automatically for JS-gated pages, giving clear context on when the tool adapts. It does not explicitly name alternatives or say when to prefer sibling tools, so it stops short of full exclusion guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. 4 tool updatesv0.1.0
    • First observeddeep_research
    • First observedextract
    • First observedscrape
    • First observedsearch

TDQS

A3.8/5.0
Disambiguation4/5

scrape and extract both operate on a page but are clearly distinguished by output type: markdown document vs structured JSON/answer. search and deep_research are also distinct, though deep_research could be seen as a superset of search.

Naming Consistency5/5

All tools use lowercase snake_case imperative verbs: scrape, extract, search, deep_research. The naming pattern is consistent and predictable.

Tool Count5/5

Four tools is well-scoped for a web scraper/research server: one for raw page content, one for structured extraction, one for search, and one for synthesis. Each tool has a clear purpose with no redundant bloat.

Completeness4/5

The core web research workflow is covered: search, fetch, extract, and synthesize. A minor gap is the lack of batch crawling or link-following features, but most common agent tasks can be completed with these tools.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

  • A
    license
    Not graded
    quality
    C
    maintenance
    Enables web scraping and crawling capabilities for LLM clients, supporting single-page scraping, multi-page website crawling, and web search with multiple engines (Playwright, Cheerio, Puppeteer) and flexible output formats including markdown, HTML, text, and screenshots.
    14
    6
    MIT
  • F
    license
    Not graded
    quality
    D
    maintenance
    Enables LLMs to extract content from websites using automated static and dynamic scraping engines with built-in anti-bot protections. It provides tools for web data retrieval and stores results in MongoDB with support for JSON and CSV exports.
    -
  • A
    license
    A
    quality
    A
    maintenance
    Web scraping, crawling, and structured data extraction for AI agents. 5 tools: scrape (clean markdown from any URL), crawl (entire sites), map (discover URLs), extract (structured JSON), and search. 833ms avg latency, single binary, self-hostable.
    8
    935
    AGPL 3.0
  • A
    license
    Not graded
    quality
    C
    maintenance
    Enables LLMs to fetch and extract web content using browser automation, OCR, and multiple extraction methods, handling JavaScript rendering and anti-scraping techniques.
    17
    MIT

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/Prog-up/web-scraper-mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server