Skip to main content
Glama
SMB-Team-Technology

web-crawler-mcp

by SMB-Team-Technology
OverviewSchemaRelated ServersScoreDiscussions
Python
Hybrid

web-crawler-mcp

A minimal crawl4ai crawler exposed as an MCP server, deployable to Cloud Run.

What it does

Exposes a single MCP tool, crawl(url, max_length), which fetches a page with crawl4ai's headless-browser crawler and returns its content as markdown.

Related MCP server: crawl-mcp-server

Local development

pip install -r requirements.txt
playwright install --with-deps chromium
crawl4ai-setup
python src/server.py

The server listens on PORT (default 8080) using the MCP Streamable HTTP transport, reachable at http://localhost:8080/mcp.

Deployment

.github/workflows/deploy.yml builds the Docker image with Cloud Build and deploys it to Cloud Run on every push to main that touches src/, Dockerfile, or requirements.txt.

Required GitHub Actions secrets:

  • GCP_SA_KEY — service account key JSON used to authenticate to Google Cloud.

  • GCP_PROJECT_ID — the target GCP project ID.

The Cloud Run service (web-crawler-mcp, region us-central1) is deployed with --allow-unauthenticated, matching the reference deployment pattern. Restrict access with IAM invoker bindings or a load balancer in front if the endpoint needs to be private.

Connecting an MCP client

Point an MCP client (Streamable HTTP transport) at:

https://<cloud-run-service-url>/mcp

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

No tool schema history has been recorded yet.

This server cannot be installed

RayrunBuild, deploy, and operate MCP servers without becoming an infrastructure team.ray.runAd

Maintenance

ActivityMaintained
ResponsivenessSyncing

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

  • crawl-mcp-server

    Git-Fg
    F
    license
    Not graded
    quality
    Not graded
    maintenance
    An MCP server for web content extraction that converts HTML pages into clean, LLM-optimized Markdown using Mozilla's Readability. It supports batch processing, intelligent multi-page crawling, and configurable caching while respecting robots.txt standards.
    43
    -

  • crawl4ai-mcp-server

    sadiuysal
    A
    license
    Not graded
    quality
    D
    maintenance
    A lightweight MCP server that exposes Crawl4AI web scraping and crawling capabilities as tools for AI agents, enabling single-page scraping and multi-page crawling with adaptive stopping.
    107
    MIT

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/SMB-Team-Technology/web-crawler-mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server