Skip to main content
Glama
deenesjakoruzh
Role1776

mcp-retrieval

by Role1776
OverviewSchemaRelated ServersScoreDiscussions
Go
Hybrid

What it is

mcp-retrieval is a Model Context Protocol server written in Go. It exposes web retrieval capabilities to any MCP-compatible client (Claude Desktop, IDE agents, custom LLM apps) as three read-only tools. Under the hood it uses the retrieval-go library to search the web and fetch pages, returning results as clean Markdown ready to hand to a model.

The library needs no API keys: web search goes through DuckDuckGo Lite, image search through Bing Images, and page fetching runs the HTML through a readability extractor before converting it to Markdown. To stay reliable against bot protection it impersonates real browsers at the TLS level and can rotate both browser fingerprints and proxies — see Retrieval engine.

Both transports the MCP SDK supports are available and expose the identical tool set:

  • stdio — the client launches the binary and talks over stdin/stdout (the default, ideal for desktop clients).

  • http — a long-running streamable HTTP server (useful for remote/shared deployments).

Related MCP server: mcp-web-calc

Tools

Tool

Description

web_search

Runs one or more queries in parallel and returns per-query deduplicated, reranked snippets with links.

web_search_images

Runs one or more image queries in parallel and returns per-query deduplicated image results.

web_scrape

Downloads one or more pages in parallel and returns the main article text as Markdown.

All three are annotated as read-only. Each tool returns a structured JSON payload that matches its output schema; the SDK mirrors the same JSON into the text content block for clients that do not read structuredContent.

web_search

Parameter

Type

Default

Notes

queries

[]string

—

Required. Executed in parallel.

max_results

int

5

Snippets per query, capped at max_results config (20).

timeout_ms

int64

5000

Whole-call timeout; clamped to [min, max] from config.

date

string

—

Freshness filter: d (day), w (week), m (month), y (year).

web_search_images

Parameter

Type

Default

Notes

queries

[]string

—

Required. Executed in parallel.

max_images

int

5

Images per query, capped at max_images config (10).

timeout_ms

int64

5000

Whole-call timeout; clamped to [min, max] from config.

date

string

—

Freshness filter: d / w / m / y.

web_scrape

Parameter

Type

Default

Notes

urls

[]string

—

Required. Downloaded in parallel.

robots_txt

bool

false

Respect the page's robots.txt.

timeout_ms

int64

5000

Whole-call timeout; clamped to [min, max] from config.

remove_links

bool

false

Strip Markdown links from the text.

max_chars

int

20000

Truncate page text to N characters, capped at max_document_chars config (20000).

Both queries/urls lists are capped at max_queries (10) items per call. Queries must be ≤ 512 characters; URLs ≤ 2048 characters and http/https only.

Results and counts

Every call fans out across the input list and returns one entry per query/URL, each with its own status — success, failed, or timeout — so a partial failure still returns the items that did work.

count is the number of items actually returned, and it can be lower than the requested max_results / max_images: duplicates within a single query's results are removed before the limit is applied, and the upstream may simply have fewer items to give. A smaller count is a normal outcome, not an error.

Deduplication is per query, not across queries. Each entry is deduplicated on its own, so a link found by two of the queries in the same call appears in both entries — dedupe the union yourself if you need it.

Errors

Request-level failures are returned as a tool result with isError: true and a plain-text message, not as a JSON-RPC error — the model reads the message and can correct the call itself. Per-item failures never do this; they stay inside the payload as status: "failed" / "timeout".

A call fails outright only when the input is rejected before any work starts, or when every item in it fails:

Message

Meaning

invalid request

The arguments did not pass validation.

too many queries / too many urls

The list exceeds MAX_QUERIES.

query must not be empty

An empty query, or an empty queries list.

query is too long

A query exceeds 512 characters.

invalid url

A URL is malformed, over 2048 characters, or not http/https.

robots.txt denied

robots_txt: true and the page disallows fetching.

upstream service unavailable

The upstream answered with an unexpected status code.

every url failed to be scraped; the pages may be unreachable or hold no extractable text

All URLs failed. Individual causes are logged to stderr, not returned.

every query failed; the search upstream may be unreachable

All queries failed.

internal server error

Anything unclassified.

The all-failed messages deliberately do not distinguish timeouts from other causes: a mixed batch can fail for several reasons at once, and the per-item status already carries that detail whenever at least one item survives.

Known limitations

  • web_scrape handles HTML only. Pages are run through a readability extractor, which needs article markup, so text/plain responses yield nothing and come back as status: "failed". Raw-file hosts are the common case: raw.githubusercontent.com, github.com/.../raw/..., cdn.jsdelivr.net. Scrape the rendered page instead of the raw file.

  • web_search_images relevance is not guaranteed. For some queries Bing Images serves a page that is not a result set, and it is parsed as though it were — the tool then returns unrelated images with status: "success". Treat image results as best-effort and verify them before showing them to a user.

  • No JavaScript. Pages are fetched as-is; content rendered client-side is invisible to the extractor.

Quick start

Install

Pick whichever fits — all of them give the identical server.

Container (no Go toolchain needed):

docker pull ghcr.io/role1776/mcp-retrieval:latest

Prebuilt binary — grab the archive for your platform from the latest release, unpack it, and put mcp-retrieval on your PATH.

MCP Bundle — for clients that install .mcpb files, download mcp-retrieval_<version>_<os>_<arch>.mcpb from the latest release and open it with your client. The bundle carries the compiled binary, so it needs neither Docker nor Go. Pick the file matching your OS and CPU architecture: a bundle holds one native binary.

From source:

go install github.com/Role1776/mcp-retrieval/app/cmd/mcp-retrieval@latest   # needs Go 1.25.5+

Or build the binary in place (the Go module lives in app/):

make build          # -> bin/mcp-retrieval

Run

# defaults: stdio transport, no configuration needed
./bin/mcp-retrieval

# with an explicit env file
./bin/mcp-retrieval -env /absolute/path/to/.env

The one flag is optional:

Flag

Meaning

-env

Path to a .env file. If omitted — or if the file does not exist — the server starts on defaults and whatever is already in the environment. There is no implicit lookup: under stdio the working directory is chosen by the MCP client, so a relative default would be unpredictable.

Connecting an MCP client (stdio)

Point your client at the built binary. Example Claude Desktop config:

{
  "mcpServers": {
    "retrieval": {
      "command": "/absolute/path/to/mcp-retrieval",
      "env": {
        "MAX_RESULTS": "20"
      }
    }
  }
}

The env block is optional — "command" alone is enough.

Connecting an MCP client (container)

Run the image on stdio. Configuration still travels through the env block, but Docker needs each variable named on the command line with -e for it to reach the process:

{
  "mcpServers": {
    "retrieval": {
      "command": "docker",
      "args": [
        "run", "-i", "--rm",
        "-e", "MAX_RESULTS",
        "-e", "DEFAULT_TIMEOUT_MS",
        "ghcr.io/role1776/mcp-retrieval:latest"
      ],
      "env": {
        "MAX_RESULTS": "20",
        "DEFAULT_TIMEOUT_MS": "5000"
      }
    }
  }
}

-i is required — without it the container gets no stdin and the client sees the server die immediately. Clients that install from the MCP Registry build this invocation themselves and prompt for the variables declared in server.json.

Running over HTTP

Set MCP_TRANSPORT=http and the server listens on SERVER_PORT at MCP_PATH (default http://localhost:8080/mcp).

Configuration

Everything is configured through environment variables, and each value is validated before startup: a non-numeric or non-positive value is a startup error. Relationships between limits are not checked at startup — see Limits. Variables already present in the environment win over a .env file, so an MCP client's env block always takes effect. Every field has a sensible default, so the server runs with no configuration at all (stdio transport).

See .env.example for the full list at its default values, ready to copy to .env.

MCP server

Env

Default

Notes

MCP_TRANSPORT

stdio

stdio or http.

MCP_NAME

mcp-retrieval

Server name advertised to clients.

MCP_PATH

/mcp

HTTP route (http transport only).

The version advertised to clients is not configurable: it is stamped into the binary at build time from the git tag.

HTTP server (http transport only)

Env

Default

SERVER_PORT

8080

SERVER_READ_TIMEOUT

60s

SERVER_WRITE_TIMEOUT

60s

HTTP client and proxy

Env

Default

Notes

MAX_IDLE_CONNS_PER_HOST

100

HTTP connection pooling.

PROXY_HOST

—

Optional. If set, requests are routed through a rotating-session proxy.

PROXY_PORT

—

Required when PROXY_HOST is set.

PROXY_SCHEME

—

Required when PROXY_HOST is set.

PROXY_LOGIN

—

Required when PROXY_HOST is set.

PROXY_PASSWORD

—

Required when PROXY_HOST is set.

When a proxy is configured, each outbound request gets a unique session id appended to the login, so the upstream provider rotates the exit IP per request.

Limits

Env

Default

MAX_QUERIES

10

DEFAULT_RESULTS

5

MAX_RESULTS

20

DEFAULT_TIMEOUT_MS

5000

MAX_TIMEOUT_MS

10000

MIN_TIMEOUT_MS

1000

DEFAULT_IMAGES

5

MAX_IMAGES

10

DEFAULT_DOCUMENT_CHARS

20000

MAX_DOCUMENT_CHARS

20000

Each value is checked on its own — it must be greater than zero — but the DEFAULT_*, MIN_* and MAX_* triples are not cross-checked against each other at startup. An inconsistent set does not stop the server; it is reconciled per request instead:

  • a value the caller omits, or passes as zero or negative, falls back to the matching DEFAULT_*;

  • the result is then clamped into [MIN_*, MAX_*], so a DEFAULT_* larger than its MAX_* simply yields MAX_*;

  • if MIN_* exceeds MAX_*, the maximum wins.

The effective limit is therefore always within the configured maximum, and misconfiguration degrades to a working server rather than a failed start. The trade-off is that it degrades silently: a typo such as MAX_RESULTS=2 instead of 20 produces no warning, only quietly smaller responses. Worth double-checking these values when results look truncated.

Logging

Env

Default

Notes

LOG_MODE

local

local → text handler at debug level; prod → JSON handler at info level. Logs go to stderr.

Architecture

The project follows a clean, layered structure. Dependencies point inward toward the domain, and each layer talks to the next through interfaces.

app/                       the Go module: sources plus its build files
                           (Dockerfile, .dockerignore, .goreleaser.yaml)

cmd/mcp-retrieval/main.go  entry point: parse flags, load config, run app

internal/
  app/                     wiring + lifecycle (build server, run, graceful shutdown)
  config/                  config loading (.env → env vars → validate)
  domain/                  core types (Query, Link, Document, Snippet, Image) and errors
  dto/web/                 request/response shapes for the MCP tools
  transport/mcp/           MCP layer
    router/                registers every tool group on the MCP server
    web/                   tool handlers
    utils/                 schema helpers and error → tool-result mapping
  usecase/web/             business logic: validation, parallelism, timeouts, dedupe/limit/rerank
  adapter/web/             retrieval-go client wiring (search, images, scrape, proxy)
  pkg/                     reusable building blocks (mcpserver, server, logger, validator)

Request flow for a tool call:

MCP client → transport/mcp/web (handler) → usecase/web → adapter/web → retrieval-go → the web
                     ↑ maps errors               ↑ validates, fans out, limits results

Search and scrape both fan out across the input list concurrently and aggregate per-item results, each with its own status (success, failed, timeout). A call only fails outright when every item in it fails.

Retrieval engine

All network work is delegated to retrieval-go, configured in app/internal/adapter/web. Worth knowing:

  • Sources. Web search uses DuckDuckGo Lite; image search uses Bing Images; page fetching runs the raw HTML through a readability extractor and converts the main article to Markdown (tables included). No search-engine API keys are required.

  • Browser impersonation. The adapter enables WithBrowserRotation(), so each request is sent from one of ~11 real browser profiles picked at random. Every profile pairs a genuine TLS/JA3 fingerprint (via uTLS) with a matching User-Agent and client-hint headers — Chrome 133/131/120 (Windows/macOS/Linux), Edge 131, Firefox 120 (Windows/macOS), Safari 18.4 (macOS), and iOS 18.4 Safari. This makes the traffic look like ordinary browsers rather than a Go HTTP client, which is what keeps the free sources reachable.

  • Proxy rotation. When PROXY_HOST is configured, the adapter installs a proxy factory that appends a unique session-<id> to the proxy username on every request. With a session-based residential/rotating proxy provider, that yields a fresh exit IP per request, spreading load and avoiding rate limits. Without a proxy, requests go out directly.

  • Response handling. Responses are transparently decompressed (gzip, br, zstd, deflate), and keep-alive is disabled (WithDisableKeepAlive()) so pooled connections don't pin a single fingerprint/IP across requests.

None of this needs configuration to work — the defaults above are applied automatically. Only proxy credentials are optional extras.

Development

Everything Go lives in app/, so either use the makefile from the repository root or pass -C app to the toolchain:

make build          # compile the binary
make test           # run tests

go -C app build ./...      # compile everything
go -C app test ./...       # run tests
go -C app vet ./...        # static checks

See CONTRIBUTING.md for pull-request guidelines.

License

Released under the MIT License.

Available Tools

3 tools
web_scrapeWeb scrapeA
Read-only

Downloads pages by their links and returns the main text as markdown. Links are fetched in parallel; a single call handles up to 10 of them, so batch all links you need into one call instead of calling the tool per link.

Size: the 'max_chars' limit applies to each page separately, not to the whole response — 5 links stay under the limit each and the response holds all 5 texts. The limit counts characters, not tokens. A page cut short ends with '[truncated]' and its 'truncated' field is set; the cut-off part cannot be fetched afterwards, so treat what you got as all there is for that page.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlsYeslinks to pages, downloaded in parallel
max_charsNotruncate each page separately to N characters, default and maximum 20000; only lowers the limit, higher values are ignored
robots_txtNorespect the page robots.txt
timeout_msNotimeout for the whole call in milliseconds
remove_linksNostrip markdown links from the text

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultsYes
metadataYes

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The annotation readOnlyHint: true covers the read-only nature, and the description adds behavioral details like truncation (ending with '[truncated]') and the fact that cut-off content cannot be fetched later. It also mentions respecting robots.txt via the parameter description. This goes beyond the annotation, though it could be more explicit about side effects (or lack thereof).

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness2/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is overly verbose and repetitive, repeating the same pattern for max_chars and truncation ('each page separately... 5 links stay under the limit each...'). This could be condensed into a single statement about per-page behavior without the redundant numeric examples. The structure is not poorly organized, but the wordiness hurts usability.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the presence of an output schema, the description does not need to explain return values, so it is appropriately scoped. It covers key behaviors like batching, per-page limits, robots.txt, timeout, and link removal. Minor gaps exist (e.g., error handling), but overall it equips an agent with the necessary context to use the tool effectively.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Every parameter is thoroughly described in the schema, with examples like 'default and maximum 20000' for max_chars and 'timeout for the whole call' for timeout_ms. The tool description reinforces these semantics, such as the per-page application of max_chars. No parameter lacks context.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's primary function: downloading pages and returning main text as markdown. It also specifies parallel fetching and batching, which is a distinctive capability. However, the repeated phrasing about per-page limits slightly detracts from the core purpose statement.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides a clear batching instruction ('batch all links you need into one call instead of calling the tool per link'), which is a useful usage tip. However, it does not explicitly contrast with sibling tools like web_search or web_search_images, leaving some ambiguity about when to choose this over alternatives. It lacks guidance on when not to use the tool.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_search_imagesWeb image searchA
Read-only

Searches for images by one or more queries, executed in parallel. IMPORTANT: Queries must be in English ONLY.

Freshness: set the 'date' field to restrict results by recency — 'd' (past day), 'w' (past week), 'm' (past month), 'y' (past year); leave it empty for all time.

Query operators can be embedded in the query string: 'site:example.com term' limits to one site, double quotes "exact phrase" force an exact match, and a leading '-' excludes a word.

ParametersJSON Schema
NameRequiredDescriptionDefault
dateNofilter results by freshness: 'd' past day, 'w' past week, 'm' past month, 'y' past year; empty means all time
queriesYesqueries in English for image search
max_imagesNomaximum number of images per query
timeout_msNotimeout for the whole call in milliseconds

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultsYes
metadataYes

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description surfaces behavioral traits beyond the readOnlyHint annotation. It explicitly calls out parallel execution, the all-time default for the date field, and the behavior of query operators ('-'), which is not inferable from the annotations or schema alone. While some rate-limit or pagination details are absent, the description meaningfully enriches the behavioral context for a tool that is already annotated as read-only.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is presented in clearly separated paragraphs with a helpful header and line breaks that break down functionality into easily scannable pieces. Each section—purpose, language constraint, freshness, query operators—adds distinct value without tangents. The formatting makes the tool behavior easily discoverable at a glance.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description covers the main complexities of the tool: parallel queries, the date filter, and query operators. It gracefully relies on the schema for parameters like max_images and timeout_ms, and the presence of an output schema covers return-value expectations. There is just enough added context for correct invocation, and the couple of minor gaps (e.g., total-result cap) are not material to correctness in most scenarios.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage for parameters is 100%, and the description complements this by explaining semantics like the interpretation of date values and how the 'queries' array drives parallel execution. It also adds the English-only constraint that is not obvious from the schema. The description would need to add similar value for all parameters, but it already fills gaps in parameter understanding.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource: 'Searches for images by one or more queries, executed in parallel.' It not only states what the tool does but differentiates it from siblings by highlighting parallel execution and the image-specific scope (vs web_search). The purpose is immediately clear even before reading the parameter details.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description clearly adds contextual usage guidance, such as 'Queries must be in English ONLY,' the date field usage with explicit values ('d', 'w', 'm', 'y'), and the query operator syntax (site:, quotes, '-'). However, it does not explicitly differentiate when to use this tool versus its siblings (e.g., 'use other tools for non-image searches'), so the exclusions could have been spelled out further.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. 3 tool updatesv1.0.0
    • First observedweb_scrape
    • First observedweb_search
    • First observedweb_search_images
Install Server
RayrunBuild, deploy, and operate MCP servers without becoming an infrastructure team.ray.runAd

TDQS

A4.4/5.0
Disambiguation5/5

Each tool has a clearly distinct purpose: scraping page content, searching text, and searching images. No overlap or ambiguity.

Naming Consistency5/5

All tools follow a consistent 'web_' prefix and descriptive verb pattern (scrape, search, search_images), making their functions predictable.

Tool Count5/5

Three tools are well-scoped for a focused web retrieval server, covering the primary operations without unnecessary bloat.

Completeness5/5

The tool set covers the essential retrieval capabilities—fetching pages, searching text, and searching images—providing a complete surface for the server's stated purpose.

Maintenance

ActivityMaintained
ResponsivenessUnresponsive

Resources

Related MCP Connectors

Related MCP Servers

  • ISIS MCP

    alucardeht
    A
    license
    B
    quality
    D
    maintenance
    A local web scraping MCP server with RAG capabilities that provides intelligent web search, content extraction, and screenshot tools without requiring API keys.
    4
    16
    MIT

  • mcp-web-calc

    tickernelz
    A
    license
    A
    quality
    D
    maintenance
    An MCP server that enables web searching, URL content extraction, and summarization without requiring API keys. It also provides advanced mathematical evaluation and multi-language Wikipedia summary retrieval tools.
    5
    319
    6
    MIT

  • web-basics-mcp

    yoloyash
    A
    license
    A
    quality
    A
    maintenance
    A self-hosted MCP server providing web search and URL fetching tools, running locally without external API keys or accounts.
    2
    539
    MIT

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/Role1776/mcp-retrieval'

If you have feedback or need assistance with the MCP directory API, please join our Discord server