Skip to main content
Glama

Web MCP Server

A privacy-focused web search MCP (Model Context Protocol) server that provides web search and content extraction capabilities. Uses SearxNG as the primary search engine with Google scraping as a fallback.

Features

  • Web Search - Search the web with category filters (general, news, images, videos, science, files)

  • Content Extraction - Extract readable content from URLs as markdown

  • Search Suggestions - Get query suggestions for better searches

  • Privacy-Focused - Uses SearxNG metasearch engine

  • Fallback Support - Automatically falls back to Google scraping if SearxNG is unavailable

  • Relevance Ranking - Query-aware reranking, deduplication, and low-signal filtering

  • Security-Aware Search - CVE/security queries prioritize trusted advisory sources

  • Rate Limiting - Built-in rate limiting to prevent abuse

  • Docker Ready - Single-container deployment with SearxNG included

Related MCP server: webmcp

Tools Provided

Tool

Description

web_search

Search the web with query, category, and limit options

fetch_content

Extract and convert webpage content to markdown

get_suggestions

Get search query suggestions

Installation

# Build the image
docker build -t web-mcp:latest .

# Run the container
docker run --rm -i web-mcp:latest

Option 2: Python Package

# Clone the repository
git clone https://github.com/your-org/web-mcp.git
cd web-mcp

# Install dependencies
pip install -r requirements.txt
pip install -r requirements-dev.txt  # Optional: tests, lint, type checks

# Or install as package
pip install -e .

# Run the server
python -m web_mcp.server

Option 3: With External SearxNG

If you have an existing SearxNG instance:

# Set the SearxNG URL
export SEARXNG_URL=http://your-searxng-instance:8080

# Run the MCP server
python -m web_mcp.server

Configuration

Environment Variables

Variable

Default

Description

SEARXNG_URL

http://localhost:8080

SearxNG server URL

SEARXNG_TIMEOUT

10

Request timeout in seconds

SEARCH_ENGINE_PROFILE_MODE

auto

Query-aware engine profile mode (auto or off)

SEARCH_SECURITY_ENGINES

brave,bing,duckduckgo,wikipedia,github,stackoverflow

Engines used for security/CVE queries

SEARCH_GENERAL_ENGINES

``

Engines for general queries (empty = SearxNG defaults)

SEARCH_CANDIDATE_MULTIPLIER

5

Candidate expansion before reranking

SEARCH_MAX_CANDIDATES

30

Maximum candidates before reranking

SEARCH_MIN_QUALITY_SCORE

2.5

Security-query quality threshold for fallback merge

FALLBACK_ENABLED

true

Enable Google scraping fallback

RATE_LIMIT_REQUESTS

30

Max requests per period

RATE_LIMIT_PERIOD

60

Rate limit period in seconds

MAX_CONTENT_LENGTH

10000

Max characters in fetched content

FETCH_ALLOW_PRIVATE_NETWORK

false

Allow fetching localhost/private network URLs

DEFAULT_SEARCH_LIMIT

5

Default number of search results

LOG_LEVEL

INFO

Logging level (DEBUG, INFO, WARNING, ERROR)

JSON_LOGS

false

Output logs in JSON format

Configuration File

Create a .env file in the project root:

SEARXNG_URL=http://localhost:8080
SEARXNG_TIMEOUT=10
SEARCH_ENGINE_PROFILE_MODE=auto
SEARCH_SECURITY_ENGINES=brave,bing,duckduckgo,wikipedia,github,stackoverflow
SEARCH_GENERAL_ENGINES=
SEARCH_CANDIDATE_MULTIPLIER=5
SEARCH_MAX_CANDIDATES=30
SEARCH_MIN_QUALITY_SCORE=2.5
FALLBACK_ENABLED=true
RATE_LIMIT_REQUESTS=30
RATE_LIMIT_PERIOD=60
MAX_CONTENT_LENGTH=10000
FETCH_ALLOW_PRIVATE_NETWORK=false
DEFAULT_SEARCH_LIMIT=5
LOG_LEVEL=INFO
JSON_LOGS=false

Usage with MCP Clients

Claude Desktop

Add to your Claude Desktop configuration (~/Library/Application Support/Claude/claude_desktop_config.json on macOS):

{
  "mcpServers": {
    "web-mcp": {
      "command": "docker",
      "args": ["run", "--rm", "-i", "web-mcp:latest"]
    }
  }
}

Or with Python:

{
  "mcpServers": {
    "web-mcp": {
      "command": "python",
      "args": ["-m", "web_mcp.server"],
      "env": {
        "SEARXNG_URL": "http://localhost:8080"
      }
    }
  }
}

Other MCP Clients

The server uses stdio transport, making it compatible with any MCP-compatible client.

Tool Reference

Search the web for information.

Parameters:

Parameter

Type

Required

Description

query

string

Yes

The search query

category

string

No

Search category: general, images, videos, news, science, files

limit

integer

No

Maximum results (default: 5, min: 1, max: 10)

Example:

{
  "name": "web_search",
  "arguments": {
    "query": "Python async programming",
    "category": "general",
    "limit": 5
  }
}

Response:

# Search Results for: Python async programming

*Provider: searxng | 5 results*

---

## 1. Async IO in Python: A Complete Guide
**URL:** https://realpython.com/async-io-python/

Complete guide to async programming in Python...

## 2. Python asyncio Documentation
**URL:** https://docs.python.org/3/library/asyncio.html

Official Python asyncio documentation...

fetch_content

Extract readable content from a URL. By default, only public http/https targets are allowed (FETCH_ALLOW_PRIVATE_NETWORK=false).

Parameters:

Parameter

Type

Required

Description

url

string

Yes

The URL to fetch content from

max_length

integer

No

Maximum content length (default: 10000, min: 500, max: 20000)

Example:

{
  "name": "fetch_content",
  "arguments": {
    "url": "https://example.com/article",
    "max_length": 5000
  }
}

Response:

# Article Title

> Brief description of the article

**Author:** John Doe
**Source:** example.com
**URL:** https://example.com/article

---

[Article content in markdown format...]

get_suggestions

Get search query suggestions.

Parameters:

Parameter

Type

Required

Description

query

string

Yes

The partial search query

Example:

{
  "name": "get_suggestions",
  "arguments": {
    "query": "python asyn"
  }
}

Response:

# Suggestions for: python asyn

1. python async await
2. python asyncio tutorial
3. python async http requests
4. python async context manager
5. python asyncio vs threading

Development

Setup

# Create virtual environment
python -m venv venv
source venv/bin/activate

# Install development dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Run linting
ruff check src tests

# Run type checking
mypy src

Manual MCP Smoke Test (Container + stdio)

Use this to verify the real MCP integration path used by CLI agents.

test.py starts the containerized MCP server as a child process with: docker run --rm -i web-mcp:latest and validates initialize, list_tools, and call_tool flows.

The container contract is stdio-only. Detached mode (docker run -d ...) is intentionally not supported for MCP clients.

# 1) Build image
docker build -t web-mcp:latest .

# 2) Run smoke script from repo root (with your virtualenv active)
.venv/bin/python test.py

# Optional: custom inputs
.venv/bin/python test.py \
  --image web-mcp:latest \
  --query "python asyncio" \
  --suggest-query "python asyn" \
  --content-url "https://example.com" \
  --limit 3 \
  --max-length 800

# Optional: full response blocks
.venv/bin/python test.py --verbose

What test.py verifies:

  • MCP session initialization against containerized server

  • Expected tools are registered: web_search, fetch_content, get_suggestions

  • Tool calls succeed over MCP stdio transport

Script behavior notes:

  • If you pass only one of --query or --suggest-query, that value is reused for both

  • test.py prints compact pass/fail summaries by default; use --verbose to show full tool outputs

  • Use --docker-command if your environment uses a different container runtime command

Project Structure

web-mcp/
├── src/web_mcp/
│   ├── __init__.py
│   ├── config.py           # Configuration management
│   ├── server.py           # MCP server entry point
│   ├── search/
│   │   ├── base.py         # SearchResult, SearchResponse, SearchProvider ABC
│   │   ├── searxng.py      # SearxNG provider
│   │   ├── google.py       # Google scraping fallback
│   │   ├── fallback.py     # Fallback orchestration + quality gate
│   │   ├── relevance.py    # Scoring, ranking, dedup, snippet cleaning
│   │   └── provider_registry.py # Shared provider singleton
│   ├── tools/
│   │   ├── web_search.py   # web_search tool
│   │   ├── fetch_content.py # fetch_content tool
│   │   └── suggestions.py  # get_suggestions tool
│   └── utils/
│       ├── logger.py       # Structured logging
│       ├── rate_limiter.py  # Rate limiting
│       ├── content_extractor.py # HTML-to-markdown extraction
│       └── validation.py   # Shared input validation
├── tests/                  # Test suite
├── docker/                 # Docker configuration
│   ├── searxng/           # SearxNG settings
│   └── entrypoint.sh      # Container entrypoint
├── Dockerfile             # Single-container Docker build
├── pyproject.toml         # Python project config
├── requirements.txt       # Runtime dependencies
└── requirements-dev.txt   # Test/lint/type dependencies

Troubleshooting

Common Issues

1. SearxNG Connection Refused

Error: Failed to connect to SearxNG
  • Ensure SearxNG is running: curl http://localhost:8080/config

  • Check SEARXNG_URL environment variable

  • If using Docker via MCP stdio, ensure the image is current (docker build -t web-mcp:latest .)

2. Google Rate Limiting

Error: Google rate limit hit (429)
  • Reduce request frequency

  • SearxNG should be used as primary; Google is fallback only

  • Wait a few minutes before retrying

3. Content Extraction Failed

Error: Failed to extract content from page
  • The page may use JavaScript rendering (not supported)

  • The page may block automated requests

  • Try with a different URL

4. Import Errors

ModuleNotFoundError: No module named 'web_mcp'
  • Ensure you're in the virtual environment

  • Install the package: pip install -e .

  • Check PYTHONPATH includes src/

Debug Mode

Enable debug logging:

export LOG_LEVEL=DEBUG
python -m web_mcp.server

Docker Debugging

# Run container interactively
docker run -it --entrypoint /bin/sh web-mcp:latest

# View logs
docker logs <container>

Security Considerations

  • SearxNG Secret: Change SEARXNG_SECRET in production

  • Rate Limiting: Configure RATE_LIMIT_REQUESTS to prevent abuse

  • Network: Container exposes port 8080 (for debugging only)

  • User Permissions: Container defaults to root-managed processes; harden users/permissions for production

License

MIT License - see LICENSE for details.

Contributing

  1. Fork the repository

  2. Create a feature branch

  3. Make your changes

  4. Run tests: pytest

  5. Submit a pull request

Acknowledgments

  • SearxNG - Privacy-respecting metasearch engine

  • MCP - Model Context Protocol

  • Trafilatura - Web content extraction

Available Tools

3 tools
fetch_contentB

Fetch and extract readable content from a URL. Returns content as markdown.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesThe URL to fetch content from
max_lengthNoMaximum content length in characters (default: 10000)

TDQS

B3.4/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so description carries full burden. It says 'readable content' but defines it only as markdown. Does not disclose whether it handles non-HTML, authentication, rate limits, or failure modes (e.g., unreachable URLs). Under-specified for safe autonomous use.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with verb and resource, second sentence adds output format. No redundancy. Every word is necessary.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with no annotations and sibling tools, the description is too minimal. Lacks guidance on edge cases, expected behavior for different URL types, and any prerequisites. Incomplete for an agent to use reliably.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so description adds minimal value beyond schema. The description does not elaborate on parameters beyond what schema provides. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clear verb ('Fetch and extract'), resource ('content from a URL'), and output format ('Returns as markdown'). Distinguishes from siblings: web_search returns search results, get_suggestions returns suggestions, not page content.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Implies usage for extracting content from a specific URL, but no explicit when-to-use or when-not-to-use versus alternatives like web_search. No prerequisites or limitations mentioned.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_suggestionsB

Get search query suggestions. Useful for autocomplete or query refinement.

ParametersJSON Schema
NameRequiredDescriptionDefault
queryYesThe partial search query to get suggestions for

TDQS

B3.1/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, and the description does not disclose any behavioral traits such as whether suggestions are personalized, require authentication, have rate limits, or are real-time. The tool's behavior is under-specified.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise with two sentences, no redundancy. It is front-loaded and efficient, though could include more detail without becoming verbose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no output schema, the description should hint at the return format (e.g., a list of suggestion strings). It lacks this detail, as well as information on result limits or sorting, making it incomplete for a simple suggestion tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and the parameter 'query' is adequately described. The description adds no additional meaning beyond the schema, so baseline score of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool retrieves search query suggestions, and the sibling tools ('fetch_content', 'web_search') indicate it is for suggestions rather than content retrieval or full searches. However, it could be more specific about the type of suggestions (e.g., completions).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description mentions 'useful for autocomplete or query refinement', which provides context, but does not explicitly state when to use alternatives like 'web_search' for full results or 'fetch_content' for content.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. 3 tool updatesv0.1.0
    • First observedfetch_content
    • First observedget_suggestions
    • First observedweb_search

TDQS

A3.5/5.0
Disambiguation5/5

Each tool has a clearly distinct purpose: fetching content from a URL, getting search suggestions, and performing a web search. There is no overlap or ambiguity between them.

Naming Consistency5/5

All tool names use a consistent verb_noun pattern in snake_case: fetch_content, get_suggestions, web_search. The naming is predictable and follows the same convention.

Tool Count5/5

With 3 tools covering search, suggestions, and content retrieval, the set is well-scoped. It is neither too few nor too many for a search-focused server.

Completeness4/5

The tool surface covers the core search workflow: querying, getting suggestions, and fetching results. A minor gap is the absence of a tool for advanced search parameters or filtering, but it is not critical.

Maintenance

ActivityInactive
ResponsivenessNo issues

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

  • A
    license
    A
    quality
    A
    maintenance
    MCP server for private web search via self-hosted SearXNG with local reranking, full-page content fetching via Firecrawl, and optional Ollama-powered query expansion and summaries.
    7
    116
    21
    MIT
  • A
    license
    Not graded
    quality
    C
    maintenance
    MCP server for web search and content extraction using DuckDuckGo or SearXNG, with Playwright-based fetching and LLM-powered data extraction.
    139
    MIT
  • F
    license
    Not graded
    quality
    D
    maintenance
    Privacy-focused web search MCP server using SearXNG with Streamable HTTP transport, supporting authentication and advanced search parameters.
    -

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/MachineLearning-Nerd/SearchMCP'

If you have feedback or need assistance with the MCP directory API, please join our Discord server