Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@smart-webfetch-mcp check page size for https://example.com/docs"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Context-aware web fetching for LLMs. Prevents context window flooding by checking page size before fetching and providing surgical extraction tools.
Standard web fetch tools dump entire pages into the context window, often:
Exceeding token limits
Wasting context on navigation, footers, ads
Flooding the model with irrelevant content
Related MCP server: Fetch MCP Server
Smart WebFetch provides 7 tools for intelligent web fetching:
Tool | Purpose |
| Check page size before fetching |
| Fetch with automatic truncation |
| Extract only code blocks |
| Fetch specific heading/section |
| Paginated fetching for large docs |
| Extract all links from a page |
| Extract tables as markdown |
# Install from PyPI
pip install smart-webfetch-mcp
# Or with uvx (recommended for MCP)
uvx smart-webfetch-mcpclaude mcp add --transport stdio smart-webfetch -- uvx smart-webfetch-mcpAdd to your opencode.json:
{
"mcp": {
"smart-webfetch": {
"type": "local",
"command": ["uvx", "smart-webfetch-mcp"],
"enabled": true
}
}
}Add to claude_desktop_config.json:
{
"mcpServers": {
"smart-webfetch": {
"command": "uvx",
"args": ["smart-webfetch-mcp"]
}
}
}Use web_preflight to check https://docs.python.org/3/library/asyncio.htmlResponse:
{
"url": "https://docs.python.org/3/library/asyncio.html",
"estimated_tokens": 45000,
"safe_for_context": false,
"recommendation": "Very large page (~45,000 tokens). Use web_fetch_section or web_fetch_chunked."
}Use web_smart_fetch on https://example.com/docs with max_tokens=4000Use web_fetch_code on https://docs.python.org/3/library/asyncio-task.htmlUse web_fetch_section on https://docs.python.org/3/library/asyncio.html
with heading="Running an asyncio Program"Use web_fetch_chunked on https://large-docs.com/api with chunk=0, chunk_size=4000Then continue with chunk=1, chunk=2, etc.
Check page metadata before fetching.
Parameters:
url (required): URL to check
Returns:
estimated_tokens: Approximate token count
content_type: MIME type
is_html: Whether content is HTML
title: Page title (if HTML)
safe_for_context: Boolean (true if < 8000 tokens)
recommendation: Human-readable advice
Fetch with automatic truncation for large pages.
Parameters:
url (required): URL to fetch
max_tokens (optional, default 8000): Maximum tokens to return
strategy (optional, default "auto"): "auto" finds natural break points, "truncate" hard cuts
Returns: Markdown content with metadata header
Extract only code blocks from a page.
Parameters:
url (required): URL to extract code from
Returns: Code blocks with language annotations and context
Fetch content under a specific heading.
Parameters:
url (required): URL to fetch from
heading (required): Heading text to find (case-insensitive)
Returns: Section content or list of available sections if not found
Fetch large documents in chunks.
Parameters:
url (required): URL to fetch
chunk (optional, default 0): Chunk index (0-based)
chunk_size (optional, default 4000): Tokens per chunk
Returns: Chunk content with navigation metadata
Extract all links from a page.
Parameters:
url (required): URL to extract links from
filter_pattern (optional): Regex to filter link URLs
external_only (optional, default false): Only return external links
Returns: Markdown list of links with text and URL
Extract tables from a page as markdown.
Parameters:
url (required): URL to extract tables from
table_index (optional): Specific table index (0-based), returns all if not specified
Returns: Markdown formatted tables
# Clone and install dev dependencies
git clone https://github.com/mathisto/smart-webfetch-mcp
cd smart-webfetch-mcp
pip install -e ".[dev]"
# Run tests
pytest
# Format code
ruff format .
ruff check --fix .MIT
Fetch large documents in chunks. Use for paginated reading of large docs. Returns chunk content plus metadata about total chunks available.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to fetch | |
| chunk | No | Chunk index (0-based) | |
| chunk_size | No | Tokens per chunk | |
| timeout | No | Request timeout in seconds (default 30, max 120) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It explicitly states that the tool returns 'chunk content plus metadata about total chunks available', which is a key behavioral trait. It does not mention any destructive actions or side effects, which is appropriate for a read-only fetch tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loaded with the main purpose, and no wasted words. Every sentence adds value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, the description explains the return value (chunk content + metadata). It lacks details on error handling or boundary conditions, but it is sufficient for a straightforward paginated fetch tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3. The description does not add significant semantic information beyond what the schema already provides for the 4 parameters (url, chunk, chunk_size, timeout).
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Fetch large documents in chunks' and 'paginated reading of large docs', which specifies the verb (fetch), resource (large documents), and the chunking mechanism that distinguishes it from sibling tools like web_fetch_code or web_fetch_links.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description says 'Use for paginated reading of large docs', providing clear context. However, it does not explicitly state when not to use this tool or mention alternatives among siblings, though the context implies it for large documents.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Extract only code blocks from a page. Ideal for documentation pages with code examples. Returns code blocks with language annotations.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to extract code from | |
| timeout | No | Request timeout in seconds (default 30, max 120) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description must disclose behavioral traits. It mentions the output ('Returns code blocks with language annotations'), but lacks details on edge cases like pages with no code, error handling, or performance characteristics.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise (two sentences) and front-loads the purpose. Every sentence adds value: the first identifies the tool, the second specifies ideal use and output. No wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the low complexity (2 parameters, 100% schema coverage, no output schema) and clear description, the tool definition is sufficiently complete. The description explains the purpose, use case, and output format, which is enough for an agent to select and use the tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% (both 'url' and 'timeout' are described). The description adds no additional meaning beyond the schema, as the parameter descriptions are already clear. Baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Extract only code blocks'), the resource ('from a page'), and the context ('Ideal for documentation pages with code examples'). It also differentiates from siblings like 'web_fetch_section' and 'web_fetch_tables' by focusing specifically on code extraction.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly states when to use this tool ('Ideal for documentation pages with code examples'), providing clear context. However, it does not mention when not to use it or explicitly compare with alternatives, which would be helpful for agent decision-making.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Extract all links from a page. Returns a markdown list of links with text and URL. Optionally filter by pattern or external links only.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to extract links from | |
| filter_pattern | No | Regex pattern to filter link URLs (e.g., '/docs/') | |
| external_only | No | Only return external links | |
| timeout | No | Request timeout in seconds (default 30, max 120) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden. It discloses the return format and optional filters but omits behavior details like handling of dynamic content, redirects, or errors.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise (two sentences, 19 words) and front-loaded with the core action. Every sentence adds value without redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity and good schema coverage, the description covers core functionality well. However, it lacks details on error handling, edge cases, and output structure beyond the markdown list.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so description adds minimal value beyond paraphrasing parameter descriptions. It does not provide additional context like pattern syntax or definition of 'external'.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action 'Extract all links from a page' and specifies the output format. It distinguishes itself from sibling tools that fetch different content types.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for extracting links but provides no explicit guidance on when to use it versus sibling tools, nor does it mention when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Fetch only content under a specific heading. Case-insensitive heading match. Use when you only need a specific part of a large document.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to fetch from | |
| heading | Yes | Heading text to find (e.g., 'Installation') | |
| timeout | No | Request timeout in seconds (default 30, max 120) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Without annotations, the description carries the full burden. It mentions case-insensitive heading matching, which is useful, but does not disclose whether nested headings are included, what happens if the heading is not found, or the format of returned content.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is composed of three concise sentences, each serving a clear purpose: stating the action, highlighting case sensitivity, and providing usage guidance. No unnecessary words or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the simplicity of the tool (3 parameters, no output schema), the description adequately covers core behavior and usage. Minor gaps exist, such as behavior when no heading is found or handling of nested headings, but overall it is sufficiently complete for an agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with parameter descriptions. The description adds value by specifying 'Case-insensitive heading match' for the heading parameter, which is not in the schema. However, it does not add semantics for url or timeout beyond what is in the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it fetches only content under a specific heading, using a case-insensitive match, which distinguishes it from siblings like web_fetch_chunked or web_fetch_code that fetch entire documents or other sections.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says to use this tool 'when you only need a specific part of a large document,' providing clear usage context. It does not mention when not to use or mention alternatives, but the guidance is sufficient for a focused tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Extract tables from a page and return as markdown tables. Handles thead/tbody, th/td cells, colspan, and captions.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to extract tables from | |
| table_index | No | Specific table index to return (0-based). Returns all tables if not specified. | |
| timeout | No | Request timeout in seconds (default 30, max 120) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden. It discloses handling of thead/tbody, th/td, colspan, and captions, but omits other behavioral traits such as error handling, redirect behavior, or what happens when no tables are found. It is adequate but has gaps.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences with no extraneous content. It is front-loaded with the core action and efficiently communicates key technical details.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with three parameters and no output schema, the description provides a good overview but lacks details on error handling, output structure, and edge cases. It is mostly complete but could be enhanced.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so baseline is 3. The description adds no additional meaning beyond the schema's parameter descriptions. It does not explain or augment the parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Extract tables' and the resource 'from a page', and specifies the output format 'markdown tables'. It distinguishes itself well from sibling tools that fetch other content types (chunked, code, links, section).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives like web_fetch_section or web_smart_fetch. It only describes what it does, lacking any explicit context for selection.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Check page size and metadata before fetching. Returns estimated tokens, content type, and whether fetch is safe for context window. Always call this first to avoid context flooding.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to check | |
| timeout | No | Request timeout in seconds (default 30, max 120) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description explains the return values (estimated tokens, content type, safety check) and the rationale (avoid context flooding). It lacks details on side effects like network activity, but is otherwise transparent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences long, each sentence serving a clear purpose. No wasted words, making it concise and well-structured.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the lack of annotations and output schema, the description covers the essential aspects: purpose, returns, and usage guidance. It is complete enough for a simple preflight check tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already fully describes the parameters (url and timeout). The tool description does not add parameter-specific semantics beyond the schema, so a baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose as checking page size and metadata before fetching, with a specific verb ('Check') and resource ('page size and metadata'). It distinguishes from sibling fetch tools by positioning itself as a preliminary step.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly advises 'Always call this first to avoid context flooding,' providing clear when-to-use guidance. It could mention when not to use it, but the context with fetch tools makes it distinct.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Fetch URL with automatic truncation if content exceeds token limit. Use web_preflight first to check size. Returns markdown content.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to fetch | |
| max_tokens | No | Maximum tokens to return (default 8000) | |
| strategy | No | How to handle large content | auto |
| timeout | No | Request timeout in seconds (default 30, max 120) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Given no annotations, the description effectively discloses key behaviors: automatic truncation upon token limit, and return of markdown content. It does not cover error handling or rate limits, but the main behavioral traits are clear.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences, each essential: action with truncation, prerequisite usage, and output format. No unnecessary words, front-loaded with core purpose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Adequate for a fetch tool with high schema coverage. Covers main functionality and prerequisite, but lacks details on parameters (strategy, timeout) and does not explain return values beyond format.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description adds no new information beyond the schema; 'automatic truncation' hints at max_tokens but does not elaborate on strategy or timeout.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool fetches a URL and returns markdown content, with automatic truncation. It mentions a prerequisite sibling tool (web_preflight) but does not explicitly differentiate from other fetch siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly advises to use web_preflight first to check size, providing clear context for when to use this tool. It does not list exclusions or alternative tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
v0.3.0web_fetch_chunkedweb_fetch_codeweb_fetch_linksweb_fetch_sectionweb_fetch_tablesweb_preflightweb_smart_fetchEach tool targets a specific aspect of web fetching (chunks, code, links, sections, tables, preflight checks, and general fetch) with no overlap in functionality.
Most tools follow 'web_fetch_*' pattern (chunked, code, links, section, tables), but 'web_preflight' and 'web_smart_fetch' deviate slightly. Still readable.
7 tools cover essential web fetching operations without redundancy. The count is well-scoped for a focused utility server.
Common use cases (full page, paginated, sections, tables, links, code) are covered. Missing advanced selectors (CSS, XPath) but core needs are met.
Unclaimed servers have limited discoverability.
If you are the server author, to access and configure the admin panel.
Structured web research tool for AI agents: search, fetch and shape web data into the JSON schema…
Web data tools for AI agents: pages as markdown, search, maps, commerce, jobs, AI answers.
Reliable web access for AI agents: smart HTTP, rotating proxies, and full-browser rendering.
Reliable web fetching for AI agents with retry, circuit breaker, caching, and anti-bot bypass
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/mathisto/smart-webfetch-mcp'
If you have feedback or need assistance with the MCP directory API, please join our Discord server
