Skip to main content
Glama

Crawlbase

by io.github.pipeworx-io

Crawlbase Scrape

crawlbase_scrape
Read-onlyIdempotent

Scrape any website through Crawlbase rotating residential proxies with server-side anti-bot bypass (Cloudflare, DataDome, hCaptcha solved for you). Returns the page as HTML (default), clean markdown (format:"md" — great for LLM context), or a JSON envelope with metadata (format:"json"). JavaScript-heavy pages (SPAs) render when you pass your Crawlbase JavaScript token as _apiKey; the Normal token does fast static fetches. Example: crawlbase_scrape({ url: "https://example.com", format: "md", _apiKey: "your-crawlbase-token" })

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
urlYesThe absolute URL to scrape including scheme, e.g. "https://example.com/page"
deviceNoDevice profile to emulate: "desktop" (default), "tablet", or "mobile".
formatNoResponse shape: "html" (default, raw page), "md" (GitHub-flavored markdown, LLM-friendly), or "json" (page + metadata in one JSON envelope).
_apiKeyYesYour Crawlbase token — Normal token for static pages, JavaScript token for rendered pages. Free tier (10,000 requests) at https://crawlbase.com
countryNoTwo-letter ISO country code to route the crawl through, e.g. "US", "GB", "DE", "JP". Default: automatic geo selection.
ajax_waitNoWait until the network is idle before capturing (JavaScript token required). Best for SPAs that fetch data after mount.
page_waitNoMilliseconds to wait after page load before capturing (JavaScript token required). Useful for content that animates in.
javascriptNoSet true when the page needs JavaScript rendering (SPAs, lazy-loaded feeds). Crawlbase selects rendering by TOKEN TYPE, so when this is true you must pass your Crawlbase JavaScript token as _apiKey (the Normal token returns the static HTML shell). Default false.

Schema Changelog

Changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. Changed1 schema field changed
    • addedInput schema / examples
      Added value: +[
+  {
+    "_apiKey": "your-crawlbase-token",
+    "format": "md",
+    "url": "https://example.com/articles"
+  },
+  {
+    "_apiKey": "your-crawlbase-token",
+    "ajax_wait": true,
+    "device": "mobile",
+    "javascript": true,
+    "url": "https://example.com/spa-dashboard"
+  }
+]
  2. First observed

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations declare readOnlyHint, idempotentHint, destructiveHint=false. Description adds valuable details: uses rotating residential proxies, bypasses anti-bot, renders SPAs with JS token, returns various formats. No contradictions.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Description is dense but well-structured: starts with purpose, then key features, then example. Every sentence adds value. Could trim some redundancy, but overall efficient.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Covers main behavioral aspects (token types, formats, anti-bypass) and provides example. Missing details like rate limits or error handling, but these are secondary for selection. Suffices for an 8-parameter tool with no output schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema has 100% coverage, and description adds meaning beyond descriptions: explains why markdown is 'great for LLM context', clarifies token types and their effects, and provides example calls. Adds significant context.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly states verb (scrape), resource (any website), and method (Crawlbase proxies with anti-bot bypass). It distinguishes from siblings like crawlbase_screenshot and crawlbase_structured by focusing on full-page scraping with HTML/MD/JSON output.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides clear guidance on when to use Normal vs JavaScript token based on page type (static vs SPA), and when to use each format. Implicitly distinguishes from crawlbase_structured (which returns structured data) but could explicitly state when that alternative is preferred.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All Crawlbase toolsInspect
Try in Browser
RayrunBuild, deploy, and operate MCP servers without becoming an infrastructure team.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

TDQS

A4/5.0
Disambiguation4/5

The toolset is largely distinct: scraping, research, prediction-market, memory, and subscription tools each have clear boundaries. The ask_pipeworx family and the six Polymarket tools are closely related variants, but their descriptions provide explicit usage guidance, so an agent can select correctly with attention.

Naming Consistency3/5

Most tools use snake_case with descriptive names, but conventions are mixed: brand-prefixed noun phrases (crawlbase_scrape, polymarket_arbitrage, pipeworx_trending) sit alongside verb_noun tools (compare_entities, validate_claim) and bare verbs (remember, subscribe). The result is readable but not predictable.

Tool Count2/5

At 34 tools, the server spans several distinct domains (web scraping, structured data research, prediction markets, memory, subscriptions, feedback), making it feel like a kitchen sink rather than a focused toolset. The count is beyond the 'heavy' threshold and would benefit from splitting into separate servers.

Completeness4/5

The research surface is thorough: routing, grounded answers, deep research, entity profiles, comparisons, claim validation, and identifier resolution cover most real-world data needs. Minor gaps exist, such as no explicit tool to fetch pipeworx:// resource URIs and no crawler management for the scraping side.

Resources