"Web crawling techniques and tools" matching MCP servers:

Provides web search and content extraction tools for AI assistants, with zero configuration and swappable backends.

Provides web access capabilities for LLMs including search, fetching, content extraction, PDF reading, image viewing, and screenshots.

Enables AI agents to extract structured content from URLs, search the web via DuckDuckGo, monitor site availability, and convert HTML to markdown without requiring API keys.

Provides local web search and content fetching capabilities for AI assistants, enabling them to search DuckDuckGo and extract clean text from web pages. All requests originate from the user's machine to ensure direct network control and bypass external proxies.

Provides web connectivity tools for searching the web via DuckDuckGo or SerpAPI, fetching URL content, and extracting readable text from web pages.

Multi-engine web search and content extraction MCP server with 10 sources, 4-way parallel queries, and content extraction via Jina Reader, no API keys required.

MCP server exposing web_search, web_fetch, and cluesift_domain_fetch tools with configurable Ollama API keys and HTTP or stdio transport.

A custom MCP server that provides AI agents with tools to fetch web pages, extract readable article text, extract structured data by CSS selector, and check robots.txt permissions.

MCP server providing free, IP-direct tools for web search, crawling, anti-detect browsing, and structured data from Wikipedia, Wikidata, GitHub, and YouTube.

A reconnaissance MCP server that discovers web data sources by monitoring network requests and extracting API structures, enabling AI to understand data location and format without full crawling.

An MCP server for safe web crawling and scraping of public and authenticated sites using Playwright and Crawlee. It provides tools for scraping single pages, crawling entire sites, validating sessions, and listing supported sites, with built-in security protections and adapter-based extensibility.

MCP server that lets AI editors execute JavaScript, read console output, and simulate clicks/input on static web pages. Supports multiple browser pages and can be deployed remotely via HTTP transport.

Enables building and evaluating LangGraph agents over a protocol-level MCP server to measure failure rates across four agent topologies.

Enables a locally-run LLM to search the web, fetch pages as markdown, make arbitrary HTTP requests, and optionally render pages with headless Chromium.

Enables extraction of recipes from PDFs or JPG files and conversion into structured HTML, with interactive region selection and OCR via a web GUI.

Enables live website health checks including TLS, HTTPS, and security headers, returning an A-F grade with specific fixes.

Enables AI agents to extract clean, structured web content (articles, tables, links, visual layouts) optimized for LLM token efficiency, with fast response times and optional JavaScript support.

Provides an LLM client with self-hosted web scraping capabilities including scraping, crawling, mapping, searching, structured data extraction, browser interaction, and deep research, without reliance on paid proxy or CAPTCHA services.

Enables Claude and other LLMs to read and parse web content, with smart fallback strategies to bypass access restrictions (e.g., paywalls, Cloudflare) and output Markdown.

Enables AI agents to drive a logged-in ChatGPT web session over CDP, including managing projects and chats, sending messages with file attachments, and retrieving long Pro replies.