Skip to main content
Glama
524,762 tools. Updated 2026-09-06 18:04

"Web crawling techniques and tools" matching MCP tools:

Matching MCP Servers

  • A
    license
    A
    quality
    B
    maintenance
    Provides web search and content extraction tools for AI assistants, with zero configuration and swappable backends.
    6
    18
    1
    MIT

Matching MCP Connectors

  • Domain/IP intelligence, web page capture and search APIs

  • Web tools for agents: fetch URL as markdown (free MCP) + x402 scrape, links, AI JSON, snapshot.

  • Extract new data from a completed crawl without re-crawling pages. Submit a new prompt and receive a jobId to track progress, paying only for AI tokens.
    MIT
  • Search Apify and Crawlee docs via full-text queries to find relevant pages on platform features, SDKs, CLI, REST API, and web scraping libraries.
    MIT
  • Test web application file upload security by attempting to upload PHP shells using Content-Type bypass techniques and alternative extensions to identify vulnerabilities.
    MIT
  • Retrieve detailed information about embedded TouchDesigner techniques, including optional code snippets and setup/workflow guidance.
    MIT
  • Discover all URLs from a website using sitemaps, CommonCrawl, or in-site crawling. Get a complete list for scraping, analysis, or site mapping.
    MIT
  • Search MITRE ATT&CK techniques by keyword in their name or description. Returns matching techniques with IDs and optional descriptions for enterprise, mobile, or ICS domains.
    Apache 2.0
  • Fetch documentation from a web URL by crawling HTML pages, with configurable depth, pattern, and directory restrictions to add a new source.
    MIT
  • Bulk lookup up to 50 MITRE ATLAS techniques in one request. Returns records with status, parent tactics, and next-step references — ideal after case study lookup.
    MIT
  • Search HackTricks documentation for pentesting techniques and exploits. Returns results grouped by file with page titles, match counts, and relevant sections.
    MIT
  • Retrieve parent techniques (main ATT&CK IDs without subtechniques) for enterprise, mobile, or ICS domains. Filter out revoked or deprecated entries and optionally include descriptions.
    Apache 2.0