"Web crawler for sequential URL content retrieval" matching MCP servers:

This server crawls documentation sites and provides search and page-retrieval tools through MCP, enabling users to query indexed documentation content.

Enables retrieval and cleaning of official documentation content for popular AI/Python libraries (uv, langchain, openai, llama-index) through web scraping and LLM-powered content extraction. Uses Serper API for search and Groq API to clean HTML into readable text with source attribution.

Provides content agents with structured search over editorial guidelines, visual templates, and past post outcomes, enabling auditable proposals that cite rule IDs.

Enables AI agents to search private GitHub Pages by authenticating in a dedicated browser and retrieving indexed sections from a local SQLite database, with support for multiple sites and languages.

Content Registry - MCP server providing AI-powered tools and automation by MEOK AI Labs

Enables comprehensive content management through Markdown processing, HTML rendering, intelligent fuzzy search, and document analysis. Supports frontmatter parsing, tag-based filtering, table of contents generation, and directory statistics for efficient content organization and discovery.

An MCP server that retrieves and cleans official documentation for popular AI/Python libraries via web search and LLM processing, exposing a get_docs tool.

A local-first MCP server for web search and documentation fetching, providing free DuckDuckGo search, URL-to-markdown conversion, and library docs from npm, PyPI, crates.io, Go, and GitHub with caching.

A version-aware cache for web research that stores and serves documentation references, preventing models from re-researching the same topics across sessions.

A BM25-based MCP server that enables document search and retrieval across structured domains of knowledge content, allowing Claude to search and reference documentation when answering questions.

Automatically crawls documentation websites, converts them to organized markdown files, and generates condensed cheat sheets. Intelligently categorizes content into tools/APIs and provides local-first access to downloaded documentation.

Enables web research through multi-provider search, documentation lookup, public code search, and clean Markdown extraction from static or JavaScript-rendered pages via five read-only tools.

A self-hosted MCP server that crawls, indexes, and searches documentation from any website locally, including private sites requiring authentication. It provides hybrid search capabilities and local embedding generation to maintain privacy while keeping AI assistant knowledge up to date.

Automatically crawl and analyze legacy web applications to generate documentation and modernization plans.

The public MCP server for togo — docs, the plugin marketplace, and plugin submission, for any agent.

MCP server for searching web documentation via sitemap.xml, enabling semantic search across multiple documentation sites using Ollama embeddings.

Crawl entire websites, sitemaps, and documentation portals into clean, LLM-ready Markdown without ads, cookie popups, or noise for RAG pipelines and AI agents.

MCP server providing browser compatibility data for the entire Web Platform.

Provides AI coding assistants with on-demand access to component specs, test scenarios, accessibility requirements, and build guides from the Web UI Component Specification.

Provides AI assistants with specialized tools to interact with NIST's Open Security Controls Assessment Language (OSCAL) framework. It enables agents to retrieve schemas, explore models, and generate valid OSCAL documentation for security compliance automation.