WebSearch Forge MCP

A lightweight, unified WebSearch MCP for AI agents.

Search, read, crawl, parse, and research the public web through one MCP server.

Quick Start · Tools · Architecture · Configure Engines · 中文文档

What It Is

WebSearch Forge MCP is a self-contained MCP service for agent-driven web research:

Question → search sources → fetch pages → extract content → crawl related pages → compile evidence

It is split into clear layers:

• transports exposes MCP stdio and the optional FastAPI adapter.

• core handles configuration, caching, security checks, orchestration, and research workflows.

• providers implement search, fetching, extraction, crawling, and media features.

• SearXNG is the single search gateway for Bing, Baidu, Brave, DuckDuckGo, and other configured engines.

Related MCP server: myscrape

Highlights

One MCP server, six capabilities

Connect once to transports/mcp_stdio.py:

Unified search gateway

The engines argument is passed to SearXNG as a filter. WebSearch Forge does not independently fan out to search websites:

WebSearch Forge MCP → SearXNG → Bing / Baidu / Brave / DuckDuckGo

Lightweight by default

MCP stdio needs no exposed application port or database server. The cache is SQLite. Optional packages add Trafilatura, Readability, stealth requests, Playwright, Office/PDF parsing, Scrapy, YouTube captions, and FastAPI without changing the MCP contract.

Bounded crawling

crawl_site supports a zero-dependency native backend and an optional Scrapy backend. It enforces page and depth limits, stays on the starting host, resolves relative links, and isolates page failures.

Quick Start

The following commands are for Windows PowerShell.

Install

cd D:\my-websearch\my_websearch
py -3.12 -m pip install -r requirements.txt

Start SearXNG

docker version
cd D:\my-websearch\my_websearch
docker compose up -d searxng
docker compose ps

The default gateway is http://127.0.0.1:8080. Configuration lives in config/searxng/settings.yml.

Configure the MCP client

Use mcp_config.example.json and keep an absolute path:

{
  "mcpServers": {
    "websearch-forge": {
      "command": "py",
      "args": [
        "-3.12",
        "D:\\my-websearch\\my_websearch\\transports\\mcp_stdio.py"
      ],
      "env": {
        "SEARXNG_URL": "http://127.0.0.1:8080",
        "CACHE_TTL": "300"
      }
    }
  }
}

The stdio adapter forces UTF-8 on Windows. Loopback traffic bypasses machine-wide proxy variables by default; set PROXY_URL explicitly when needed.

Tools

search_web

Search through SearXNG without downloading page bodies.

{
  "name": "search_web",
  "arguments": {
    "query": "Python 3.14 new features",
    "engines": ["bing", "baidu", "brave"],
    "limit": 5,
    "time_range": "month"
  }
}

fetch_web_content

Fetch and extract one public URL. HTML, PDF, DOCX, XLSX, PPTX, CSV, Markdown, and plain text are supported.

{
  "name": "fetch_web_content",
  "arguments": {
    "url": "https://www.python.org",
    "max_chars": 10000,
    "stealth_mode": "off",
    "render_mode": "auto",
    "extraction_mode": "auto"
  }
}

The response includes final URL, HTTP status, title, extraction method, word count, content, and discovered links.

search_and_fetch

Search first, then fetch the top results independently. A failed page is recorded on that item and does not cancel the batch.

{
  "name": "search_and_fetch",
  "arguments": {
    "query": "FastAPI MCP server",
    "limit": 3,
    "max_chars": 12000
  }
}

deep_research

Run related queries concurrently, fetch the strongest results, and return a Markdown report with source-level failures.

{
  "name": "deep_research",
  "arguments": {
    "queries": ["SearXNG engine configuration", "MCP stdio deployment"],
    "breadth": 3,
    "max_chars": 12000
  }
}

crawl_site

Crawl a same-host site with hard page and depth limits.

{
  "name": "crawl_site",
  "arguments": {
    "url": "https://www.python.org",
    "max_pages": 10,
    "max_depth": 2,
    "backend": "native",
    "stealth_mode": "off"
  }
}

native is the default. Install Scrapy and set backend to scrapy to use the optional backend. Each page reports URL, depth, status, fetch method, title, content, and word count.

youtube_transcript

Retrieve YouTube captions with optional source and translation languages.

Response Shape

{
  "query": "OpenAI",
  "provider": "searxng",
  "engines": ["bing", "baidu", "brave"],
  "total_results": 3,
  "results": [
    {
      "title": "OpenAI | Research & Deployment",
      "url": "https://openai.com/",
      "description": "...",
      "source": "openai.com",
      "engine": "bing",
      "score": 1.0
    }
  ],
  "partial_failures": []
}

Successful partial results are preserved. Engine, page, and document errors are returned as structured failure entries.

Architecture

flowchart LR
    A[Agent / MCP Client] -->|stdio JSON-RPC| B[transports/mcp_stdio.py]
    B --> C[core/service.py]
    C --> D[providers/search]
    D --> E[SearXNG]
    E --> F[Bing / Baidu / Brave / DDG]
    C --> G[providers/content]
    G --> H[HTTP / stealth / Playwright]
    C --> I[providers/crawl]
    C --> J[providers/media]
    C --> K[(SQLite TTL cache)]

• transports adapts protocols; MCP and FastAPI share the same service.

• core owns orchestration, cache policy, configuration, and URL security.

• providers/search talks to SearXNG and validates engine names.

• providers/content handles HTTP, stealth transport, rendering, and extraction.

• providers/crawl contains native and Scrapy crawling backends.

• providers/media contains the YouTube transcript provider.

Configure Search Engines

The project-owned SearXNG source is:

config/searxng/settings.yml

Two settings decide whether an engine can be called:

1. settings.yml enables the engine inside SearXNG.

2. providers/search/registry.py lists the accepted name in SUPPORTED_ENGINES.

Restart after changes:

cd D:\my-websearch\my_websearch
docker compose up -d --force-recreate searxng

If SearXNG does not provide the engine yet, implement that SearXNG engine first.

Security and Reliability

• Only HTTP and HTTPS URLs are accepted.

• Localhost, loopback, private IPv4, link-local, and private IPv6 targets are rejected.

• Redirect destinations are validated again.

• Search and fetch operations use a SQLite TTL cache, 300 seconds by default.

• Partial failures do not discard successful work.

• stdout is reserved for MCP JSON-RPC.

Optional Capabilities

Project Layout

my_websearch/
├── assets/                     # Project logo
├── config/searxng/             # SearXNG settings.yml
├── core/                       # Config, cache, security, orchestration
├── providers/
│   ├── search/                 # SearXNG gateway and engine allow-list
│   ├── content/                # Requests, rendering, extraction
│   ├── crawl/                  # Native and Scrapy backends
│   └── media/                  # YouTube transcript provider
├── transports/                 # MCP stdio and FastAPI adapters
├── tests/                      # Dependency-free self-checks
├── docker-compose.yml          # Local SearXNG gateway
├── mcp_config.example.json     # MCP client template
├── requirements.txt            # Dependency entry point
├── README.md                  # English documentation
└── README.zh-CN.md            # 中文文档

Development Check

cd D:\my-websearch
py -3.12 -m my_websearch.tests.test_server

Expected output:

my_websearch self-check: ok

Optional FastAPI service:

cd D:\my-websearch\my_websearch
py -3.12 -m transports.api

Then open http://127.0.0.1:8787/docs.

License

No license is imposed yet. Add a root-level LICENSE file before public distribution.