"Web crawler for sequential URL content retrieval" matching MCP servers:

MCP server that fetches raw HTML content from a given URL to provide web context to LLMs.

Enables agents to crawl URLs and retrieve page content as fixed-shape structured JSON, including title, markdown, links, word count, and error status.

Enables AI agents to fetch, crawl, extract, and search web content through a unified set of tools, with automatic routing to specialized backends.

A Model Context Protocol server that analyzes webpage design structures, providing detailed layout, navigation, content, form, image, and link information from a given URL.

A minimal MCP server that crawls web pages using crawl4ai and returns content as markdown, deployable to Cloud Run.

Enables crawling websites, including JavaScript-rendered pages, and returns extracted page content as JSON with structured error reporting.

Provides lightweight web crawling and scraping capabilities, enabling users to fetch pages as clean Markdown, recursively crawl domains, extract metadata, and perform search-and-crawl operations.

An open-source MCP server providing AI agents with neural web search via Exa and tiered web fetch (Exa, local browser, Firecrawl) as a drop-in replacement for built-in web tools, preserving provenance and guarding against SSRF.

An intelligent web crawling server that uses Cloudflare's headless browser to render dynamic pages and Workers AI to extract relevant links based on natural language queries. It enables AI assistants to search and filter website content while providing secure access through GitHub OAuth authentication.

Web-crawler MCP server that fetches web pages with static parsing or headless browser fallback, enabling Claude to fetch, extract links, crawl sites, and select content via CSS selectors.

Enables fetching and reading content from web URLs through a configurable proxy prefix. Uses curl to retrieve webpage content and return it as text for analysis or processing.

An MCP server providing web search, image search, and page scraping tools to LLMs without requiring API keys.

Enables AI agents to extract clean, structured web content (articles, tables, links, visual layouts) optimized for LLM token efficiency, with fast response times and optional JavaScript support.

A black-box flight recorder for RAG retrieval inside MCP agents. Logs what chunks the model saw, scores, sources, and rankings - so you can audit, replay, and diff retrieval runs after the fact.

Scores AI outputs for faithfulness, relevancy, and hallucination inside any MCP client, with custom metrics, golden sets, and run history with dashboards.

MCP server that gives AI agents access to current web information through web_search and fetch_url tools, using Tavily, Brave, and DuckDuckGo providers with automatic fallback.

Simple sequential thinking MCP in python

Enables AI agents to parse PDFs and search local document context through multimodal extraction and vector-based retrieval, all exposed as MCP tools.

Extract content from URLs, documents, videos, and audio files using intelligent auto-engine selection. Supports web pages, PDFs, Word docs, YouTube transcripts, and more with structured JSON responses.

This server crawls documentation sites and provides search and page-retrieval tools through MCP, enabling users to query indexed documentation content.