Skip to main content
Glama
flutterninja9

Better Fetch

by flutterninja9
OverviewSchemaRelated ServersScoreDiscussions
JavaScript
Remote

Better Fetch Banner

Better Fetch - Advanced Web Content MCP Server

License: MIT TypeScript Node.js npm version

A powerful Model Context Protocol (MCP) server that intelligently fetches and processes web content with nested URL crawling capabilities. Transform any documentation site or web resource into clean, structured markdown files perfect for AI consumption and analysis.

🚀 Key Features

🕸️ Smart Web Crawling

  • Nested URL Fetching: Automatically discovers and crawls linked pages up to configurable depth

  • Single Page Mode: Option for simple single-page content extraction

  • Domain Filtering: Stay within the same domain or allow cross-domain crawling

  • Pattern Matching: Include/exclude URLs based on regex patterns

🧠 Intelligent Content Processing

  • Content Cleaning: Removes ads, navigation, scripts, and other noise automatically

  • Smart Section Detection: Identifies main content areas (<main>, <article>, .content)

  • Automatic Titles: Generates meaningful section headers based on page titles and URL structure

  • Table of Contents: Creates organized TOC with proper nesting

📝 Advanced Markdown Generation

  • Clean Formatting: Converts HTML to well-structured markdown

  • Code Block Preservation: Maintains formatting for code snippets and technical content

  • Link Preservation: Keeps all important links with proper markdown syntax

  • Metadata Integration: Includes source URLs, generation timestamps, and site information

⚙️ Highly Configurable

  • Crawl Depth Control: Set maximum levels to crawl (default: 2)

  • Page Limits: Control maximum pages to process (default: 50)

  • Timeout Settings: Configurable request timeouts

  • Respectful Crawling: Built-in delays between requests

  • Error Handling: Graceful handling of failed requests and invalid URLs

Related MCP server: Inngest MCP Docs Server

📋 Available Tools

1. fetch_website_nested

Comprehensive web crawling with nested URL processing.

Parameters:

  • url (required): Starting URL to crawl

  • maxDepth (optional, default: 2): Maximum crawl depth

  • maxPages (optional, default: 50): Maximum pages to process

  • sameDomainOnly (optional, default: true): Restrict to same domain

  • excludePatterns (optional): Array of regex patterns to exclude

  • includePatterns (optional): Array of regex patterns to include

  • timeout (optional, default: 10000): Request timeout in milliseconds

2. fetch_website_single

Simple single-page content extraction.

Parameters:

  • url (required): URL to fetch

  • timeout (optional, default: 10000): Request timeout in milliseconds

💡 Use Cases

📚 Documentation Processing

  • API Documentation: Convert REST API docs, SDK guides, and technical references

  • Framework Docs: Process React, Vue, Angular, or any framework documentation

  • Library Guides: Extract comprehensive guides from library documentation sites

  • Tutorial Series: Gather multi-part tutorials into single organized documents

🔍 Content Analysis & Research

  • Competitive Analysis: Gather competitor documentation and feature descriptions

  • Market Research: Extract product information from multiple related pages

  • Academic Research: Collect and organize web-based research materials

  • Knowledge Base Creation: Transform scattered web content into structured knowledge bases

🤖 AI Training & Context

  • LLM Context Preparation: Create clean, structured content for AI model training

  • RAG System Input: Generate high-quality documents for Retrieval-Augmented Generation

  • Chatbot Knowledge: Build comprehensive knowledge bases for customer service bots

  • Content Summarization: Prepare web content for automated summarization tasks

🛠️ Installation & Setup

Quick Install via npm (Recommended)

The simplest way to use Better Fetch is via npx. Just add this to your MCP client configuration:

For Claude Desktop - Add to claude_desktop_config.json:

{
  "mcpServers": {
    "better-fetch": {
      "command": "npx",
      "args": ["-y", "@infero.mcp/better-fetch"]
    }
  }
}

For VS Code MCP Extension:

{
  "better-fetch": {
    "command": "npx",
    "args": ["-y", "@infero.mcp/better-fetch"]
  }
}

This automatically downloads and runs the latest version without any manual installation or build steps.

Installing via Smithery

To install Better Fetch for Claude Desktop automatically via Smithery:

npx -y @smithery/cli install @infero.mcp/better-fetch --client claude

Prerequisites

  • Node.js 18+

  • npm or yarn

  • MCP-compatible client (Claude Desktop, VS Code with MCP extension, etc.)

Step 1: Clone and Install

git clone https://github.com/flutterninja9/better-fetch.git
cd better-fetch
npm install

Step 2: Build the Project

npm run build

Step 3: Test the Server (Optional)

# Quick test
npm run dev

# Or run comprehensive tests
node test-mcp.js

Step 4: Configure Your MCP Client

For Claude Desktop (Recommended - via npm):

Add to your claude_desktop_config.json:

{
  "mcpServers": {
    "better-fetch": {
      "command": "npx",
      "args": ["-y", "@infero.mcp/better-fetch"]
    }
  }
}

For VS Code MCP Extension (Recommended - via npm):

{
  "better-fetch": {
    "command": "npx",
    "args": ["-y", "@infero.mcp/better-fetch"]
  }
}

For Custom MCP Client (via npm):

{
  "name": "better-fetch",
  "command": "npx",
  "args": ["-y", "@infero.mcp/better-fetch"]
}

Manual Installation (Alternative)

If you prefer to install and build locally:

For Claude Desktop:

{
  "mcpServers": {
    "better-fetch": {
      "command": "node",
      "args": ["/absolute/path/to/better-fetch/dist/server.js"],
      "env": {
        "NODE_ENV": "production"
      }
    }
  }
}

For VS Code MCP Extension:

{
  "better-fetch": {
    "command": "node",
    "args": ["/absolute/path/to/better-fetch/dist/server.js"]
  }
}

📖 Usage Examples

Basic Documentation Crawling

Fetch all the web contents from this Flutter Shadcn UI documentation site:
https://flutter-shadcn-ui.mariuti.com/

Use nested fetching with a maximum depth of 3 levels and process up to 100 pages.

Advanced Configuration

Fetch content from the React documentation but exclude any URLs containing 'api' or 'reference' and only process pages containing 'tutorial' or 'guide':

URL: https://react.dev
Max Depth: 2
Exclude Patterns: ["/api/", "/reference/"]
Include Patterns: ["/tutorial/", "/guide/"]
Max Pages: 30

Single Page Extraction

Extract the content from this specific page only:
https://nextjs.org/docs/getting-started/installation

Use single page mode to avoid crawling related links.

📄 Sample Output

The server generates comprehensive markdown files with the following structure:

# Site Name Documentation

*Scraped from: https://example.com*
*Generated on: 2024-01-15T10:30:00.000Z*

## Table of Contents

- [Getting Started](#getting-started)
  - [Installation](#installation)
  - [Quick Start](#quick-start)
- [API Reference](#api-reference)
  - [Core Functions](#core-functions)

---

## Getting Started

*Source: [https://example.com/getting-started](https://example.com/getting-started)*

[Clean markdown content here...]

---

## Installation

*Source: [https://example.com/installation](https://example.com/installation)*

[Installation instructions in markdown...]

For a complete example, refer to output.md which demonstrates the server's output when processing a real documentation site.

🔧 Development

Project Structure

better-fetch/
├── src/
│   └── server.ts          # Main server implementation
├── dist/                 # Compiled JavaScript
├── test-mcp.js          # Testing utilities
├── output.md            # Sample output file
├── package.json
├── tsconfig.json
└── README.md

Available Scripts

npm run dev          # Run in development mode with hot reload
npm run build        # Compile TypeScript to JavaScript
npm run start        # Run the compiled server
npm run clean        # Clean dist directory
npm test             # Run test suite

Testing Your Changes

# Interactive testing
node interactive-test.js

# Automated test suite
node test-mcp.js

# Manual JSON-RPC testing
echo '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | node dist/index.js

🚦 Performance & Limits

Default Limits

  • Max Depth: 2 levels (configurable)

  • Max Pages: 50 pages (configurable)

  • Request Timeout: 10 seconds (configurable)

  • Crawl Delay: 500ms between requests (respectful crawling)

Performance Tips

  • Set appropriate maxPages limits for large sites

  • Use includePatterns to focus on relevant content

  • Enable sameDomainOnly to avoid external link crawling

  • Adjust timeout based on target site response times

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

Development Setup

  1. Fork the repository

  2. Create a feature branch: git checkout -b feature/amazing-feature

  3. Make your changes and add tests

  4. Commit your changes: git commit -m 'Add amazing feature'

  5. Push to the branch: git push origin feature/amazing-feature

  6. Open a Pull Request

📜 License

This project is licensed under the MIT License - see the LICENSE file for details.

🆘 Support & Issues

🙏 Acknowledgments

Made with ❤️ for the AI and developer community

Available Tools

2 tools
fetch_website_nestedC

Fetch website content with nested URL crawling and convert to clean markdown

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesThe starting URL to fetch and crawl
maxDepthNoMaximum depth to crawl (default: 2)
maxPagesNoMaximum number of pages to fetch (default: 50)
sameDomainOnlyNoOnly crawl URLs from the same domain (default: true)
excludePatternsNoRegex patterns for URLs to exclude
includePatternsNoRegex patterns for URLs to include (if specified, only matching URLs will be processed)
timeoutNoRequest timeout in milliseconds (default: 10000)

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions crawling and markdown conversion but fails to disclose critical traits like rate limits, authentication needs, error handling, or what happens when limits (maxDepth/maxPages) are reached. The description is too vague about operational behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence that front-loads the core functionality. Every word earns its place, with no redundant or vague phrasing, making it easy to parse quickly.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (7 parameters, crawling behavior) and lack of annotations and output schema, the description is insufficient. It omits details on return format, error cases, performance implications, and practical usage constraints, leaving significant gaps for an AI agent to operate effectively.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema fully documents all 7 parameters. The description adds no additional meaning beyond implying crawling behavior, which is already suggested by parameter names like 'maxDepth' and 'sameDomainOnly.' Baseline 3 is appropriate as the schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('fetch website content with nested URL crawling') and transformation ('convert to clean markdown'), providing a specific verb+resource combination. It distinguishes from the sibling tool 'fetch_website_single' by specifying 'nested URL crawling,' though it could be more explicit about the distinction.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus the sibling 'fetch_website_single' or other alternatives. It lacks context about scenarios where nested crawling is preferred over single-page fetching, such as for multi-page documentation or site-wide content extraction.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

fetch_website_singleB

Fetch content from a single webpage and convert to clean markdown

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesThe URL to fetch
timeoutNoRequest timeout in milliseconds (default: 10000)

TDQS

B3.4/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries full burden. It mentions conversion to markdown, which is a behavioral trait, but lacks details on error handling, rate limits, authentication needs, or what 'clean' entails. This is inadequate for a tool that performs network operations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence with zero waste. It is front-loaded with the core purpose and transformation, making it easy to understand quickly.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no annotations and no output schema, the description is incomplete. It lacks information on return values (e.g., markdown structure, error formats), behavioral constraints, and differentiation from the sibling tool, which is crucial for a tool with network dependencies.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema fully documents both parameters. The description does not add any meaning beyond what the schema provides, such as URL format expectations or timeout implications. Baseline 3 is appropriate when schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the specific action ('Fetch content'), target resource ('from a single webpage'), and transformation ('convert to clean markdown'). It distinguishes from the sibling tool 'fetch_website_nested' by specifying 'single' versus implied nested/multiple pages.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for fetching a single webpage's content, but does not explicitly state when to use this tool versus the sibling 'fetch_website_nested' or other alternatives. No exclusions or prerequisites are mentioned.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. 2 tool updates
    • First observedfetch_website_nested
    • First observedfetch_website_single
Install Server
RayrunBuild, deploy, and operate MCP servers without becoming an infrastructure team.ray.runAd

TDQS

B3.2/5.0
Disambiguation5/5

The two tools have clearly distinct purposes: one handles single-page fetching while the other includes nested URL crawling. The descriptions explicitly differentiate between these scopes, leaving no room for confusion or misselection.

Naming Consistency5/5

Both tools follow a consistent verb_noun pattern with 'fetch_website' as the base, differentiated by descriptive suffixes ('_single' and '_nested'). This predictable naming makes it easy to understand their relationship and functionality.

Tool Count2/5

With only two tools, the server feels under-scoped for a 'Better Fetch' purpose. While the tools cover basic fetching scenarios, there are likely missing operations like handling authentication, adjusting fetch parameters, or error management that would be expected in a robust fetching toolset.

Completeness2/5

For a fetching domain, the toolset is severely incomplete. It lacks essential operations such as configuring fetch options (e.g., headers, timeouts), handling different content types, managing errors, or providing status information. This will likely cause agent failures when dealing with complex fetching tasks.

Maintenance

ActivityInactive
ResponsivenessSyncing

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

  • Inngest MCP Docs Server

    inngest
    -
    license
    Not graded
    quality
    Not graded
    maintenance
    A Model Context Protocol server that provides AI coding agents with access to up-to-date Inngest documentation by fetching and converting web content to Markdown format.
    3
    -

  • Context MCP Server

    LangGPT
    A
    license
    A
    quality
    D
    maintenance
    A Model Context Protocol server that provides web content fetching capabilities with robots.txt checking removed, allowing LLMs to retrieve and convert web content to markdown.
    2
    1
    MIT

Appeared in Searches

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/flutterninja9/better-fetch'

If you have feedback or need assistance with the MCP directory API, please join our Discord server