web-data-mcp
This server provides quality-gated web scraping and data transformation for AI agents, using Apify actors with automatic retry and validation. You can:
Scrape a URL (
scrape_url): Crawl a public page to clean markdown, automatically scoring quality and retrying with anti-blocking if needed.Run any allowlisted Apify actor (
run_actor): Start an actor with custom input, with options to wait or fire-and-forget.Check run status (
get_run_status): Poll an actor run's status and get dataset info.Fetch dataset items (
fetch_dataset_items): Read items with pagination, field projection, token budgeting, and summary preview.Validate dataset quality (
validate_dataset): Score a dataset against a JSON Schema, measuring schema pass, completeness, duplicates, and bot-wall rates.Retry low-quality runs (
retry_low_quality_run): Re-run with escalating anti-blocking (proxies, browser) until quality threshold is met.Convert to RAG documents (
dataset_to_rag_documents): Turn scraped items into embedding-ready chunks with token limits, source attribution, and content-hash IDs.Observability & security: Integrates OpenTelemetry tracing; includes actor allowlist, SSRF protection, token caps, and HTTP auth options.
Allows scraping Facebook Ad Library data via the facebook-ad-intelligence-pro Apify actor, returning validated, scored, and chunked content suitable for AI agents.
Allows scraping Reddit posts via the reddit-scraper-pro Apify actor, returning validated, scored, and chunked content suitable for AI agents.
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@web-data-mcpscrape https://example.com and return quality-scored markdown"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
web-data-mcp
Quality-gated web data for AI agents. An MCP server that runs Apify scraping actors and — unlike a raw passthrough — validates what came back, scores it, retries with stronger anti-blocking settings when it's bad, and hands your agent embedding-ready chunks instead of a JSON dump.

Scraped data fails silently: the run "succeeds" but the dataset is a wall of Access Denied pages, half-empty records, or duplicates — and an agent that can't see quality will happily reason over garbage. This server makes data quality a first-class, machine-readable part of every tool result.
Quickstart
git clone https://github.com/fctpe/web-data-mcp && cd web-data-mcp
pnpm install && pnpm build
# poke every tool in a UI
APIFY_TOKEN=your-token npx @modelcontextprotocol/inspector node dist/index.js
# Claude Code
claude mcp add web-data --env APIFY_TOKEN=your-token -- node /path/to/web-data-mcp/dist/index.js
# Claude Desktop / Cursor / any stdio client — add to your MCP config:
{
"mcpServers": {
"web-data": {
"command": "node",
"args": ["/path/to/web-data-mcp/dist/index.js"],
"env": { "APIFY_TOKEN": "your-token" }
}
}
}(npx -y web-data-mcp will replace the node path once the first npm release is published — the bin entry is already wired.)
Free Apify accounts include $5/month of platform credit — enough for hundreds of scrape_url calls with the default cheerio crawler.
Related MCP server: mcp-firecrawl
Why not the official Apify MCP server?
Use both — they solve different problems.
web-data-mcp | ||
Scope | The whole Apify store (5,000+ actors), dynamic discovery | 7 curated tools around one workflow |
Output | Raw dataset passthrough | Schema-validated, quality-scored, token-budgeted |
Bad scrapes | Your agent finds out the hard way | Scored ( |
RAG | Bring your own chunking |
|
Guardrails | Platform-level | Actor allowlist, SSRF guard (no private hosts), clamped memory/timeouts, hard response token caps |
flowchart LR
A[AI agent] -->|MCP| S[web-data-mcp]
S --> R[Run actor]
R --> Q{Quality gate<br/>schema · completeness<br/>dupes · bot-wall}
Q -->|score >= threshold| C[Chunked, token-bounded,<br/>hash-addressed documents]
Q -->|score < threshold| E[Escalate: residential proxies,<br/>browser crawler] --> R
C --> ATools
Tool | What it does |
| One call: crawl a URL → wait → score → auto-retry if blocked → return markdown + quality block |
| Start an allowlisted actor with explicit input; returns |
| Poll a run (read-only, free) |
| Paginated reads with field projection, |
| Score a dataset against your JSON Schema: pass rate, completeness, dupes, bot-wall rate |
| Re-run with residential proxies → browser crawler until quality clears your threshold |
| Emit embedding-ready chunks (jsonl/markdown/text) with source attribution + content hashes |
Every tool ships inputSchema and outputSchema (structured output), behavior annotations (readOnlyHint, openWorldHint, …), and returns failures as model-readable isError results with a concrete next step — so the calling agent can self-correct instead of stalling.
What the agent actually gets back
The point is that data quality is structured, not buried in prose. A validate_dataset result (or the quality block on scrape_url) looks like this — the agent can branch on score, and sample_failures tells it exactly what's wrong:
{
"quality": {
"score": 0.42, // composite 0..1 — below threshold, don't trust
"item_count": 50,
"schema_pass_rate": 0.30, // 70% of items fail your JSON Schema
"field_completeness": 0.61,
"duplicate_rate": 0.12,
"suspected_block_rate": 0.24, // ~1 in 4 items look like a bot wall
"sample_failures": [
"item[3]/price: must be number",
"item[7]: 'Attention Required | Cloudflare' in body"
]
}
}And a dataset_to_rag_documents line — token-bounded, source-attributed, content-hashed for idempotent upserts:
{ "id": "a1b2c3d4e5f6-0", "source": "https://example.com/p/12", "chunkIndex": 0,
"chunkCount": 1, "tokenCount": 118, "content": "…clean extracted text…",
"metadata": { "crawledAt": "2026-07-12T…" } }How the quality gate works
Each dataset sample is scored 0..1 from four signals:
Schema pass rate — items validated against your JSON Schema (Ajv), weighted 0.4 when present
Field completeness — non-empty cells across the union of fields
Duplicate rate — hash-based duplicate detection
Bot-wall rate — items containing block markers (
Access Denied,captcha,verify you are human, …)
Blocking is a ceiling, not a deduction. The final score is min(weighted, 1 - bot_wall_rate): a batch cannot score higher than the fraction of it that is actually content. Blocking also gets its own retry trigger (suspected_block_rate > 0.2) independent of the score, because a partly blocked batch caps above the threshold — 25% walls caps at 0.75 — and a quarter of your pages being walls is exactly what a residential proxy is for. Why it is a ceiling and not a weighted term, and the test that attacks it: ADR 0005 and test/blocked-content.test.ts.
These are heuristics. The bot-wall regex catches common block pages, not all of them, and field completeness treats every field as equally important. Schema pass rate is the signal to rely on when you can supply a schema.
Below threshold, retries escalate: original input → + residential proxies → + playwright:firefox with dynamic-content waits. Attempts and both scores are reported in the structured result, and exhausted retries return an isError result that tells the agent why and what to try next — naming a bot wall as a bot wall rather than as "low quality", since a model told the content is thin will summarise the wall as if it were the page.
Results
Reproducible from this repository: 106 offline tests (pnpm test, no network, no token) plus a real stdio protocol round trip through the client library the repo already depends on — node scripts/stdio-smoke.mjs prints smoke ok — 7 tools, all with output schemas, guard + tool call live.
Beyond that, the full MCP flow (run → status → fetch → validate → RAG) was pressure-tested against three production Apify actors with real workloads. Those runs committed no artifact and are not reproducible here — reproducing them needs an APIFY_TOKEN and named actor slugs — so no quality score or item count from them is quoted. scripts/live-smoke.mjs is the documented path for anyone with a token to run the same flow against their own actor and read the numbers off their own output.
What the live runs did leave behind is checkable: two bugs the mocked tests could not catch, each now pinned by a regression test in test/review-regressions.test.ts:
Apify's dataset
itemCountis eventually consistent — it reads 0 for a few seconds after a run finishes. Pagination now floors the total at what was actually fetched.A 245-token CDN image URL out-lengthed the ad copy in the RAG auto-detect fallback, producing well-formed garbage embeddings. Content detection now requires prose shape (whitespace ratio, non-URL) and skips items honestly instead.
HTTP mode
WEB_DATA_MCP_HTTP_TOKEN=$(openssl rand -hex 24) node dist/index.js --transport http --port 3000Binds to 127.0.0.1 with Host/Origin validation (DNS-rebinding protection) and constant-time bearer auth. Built on the MCP TypeScript SDK v2 (spec 2026-07-28); 2025-era clients are served through the SDK's built-in legacy fallback.
Tracing (optional)
pnpm add @opentelemetry/api @opentelemetry/sdk-trace-base @opentelemetry/exporter-trace-otlp-http
OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318 node dist/index.jsOne span per tool call, named execute_tool <tool>, carrying gen_ai.operation.name, gen_ai.tool.name, and — for the tools that score their data — web_data_mcp.quality.score, so a bad scrape is visible in the trace and not just in the tool result. Failed calls get span status ERROR. OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is honoured too and, as the standard requires, wins over the base endpoint.
Export is OTLP/HTTP only, with no console exporter — stdout belongs to the MCP protocol stream. An endpoint that is not an http(s) URL turns tracing off with a message naming the variable rather than dropping every span in silence, and with neither variable set the SDK is never imported at all, so the packages above are genuinely optional peers. The stdout/stderr rules and what the spans may carry are in SECURITY.md.
Example: LangGraph agent
examples/langgraph-agent wires the server into a LangGraph agent whose system prompt enforces the quality contract ("if score < 0.7, say so instead of trusting the content"):
cd examples/langgraph-agent
npm install
OPENAI_API_KEY=... APIFY_TOKEN=... npm start -- "https://apify.com/pricing"Limitations
Quality scoring is heuristic (see above). Schema pass rate is the signal to rely on when you can provide a schema.
Escalation strategies (
crawlerType,dynamicContentWaitSecs) targetapify/website-content-crawler-style inputs; other actors get proxy escalation only, and unknown input keys are passed through untouched.retry_low_quality_runre-runs the whole actor input — it does not retry only failed URLs within a run.No streaming: long crawls block up to the wait budget (300s default). Fire-and-forget via
run_actor+ polling is the workaround for bigger jobs.Tested against
apify-client2.x and MCP SDK2.0.0-beta.3(pinned); the pin will move to v2 stable when it ships.
Design notes
Architecture decisions are recorded in docs/adr/: curated tools vs. dynamic discovery, SDK v2 beta, the dependency-injected gateway that keeps the whole test suite offline and sub-second, hard token budgets with explicit handles, and why blocking caps the quality score. Security posture (token handling, URL guard, transport auth) is in SECURITY.md.
Development
pnpm test # offline test suite incl. full client<->server integration
pnpm lint && pnpm typecheck
pnpm inspect # build + MCP Inspector
node scripts/stdio-smoke.mjs # protocol round trip against dist/
OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318 node scripts/stdio-smoke.mjs # same, tracing on
APIFY_TOKEN=... SMOKE_ACTOR=... node scripts/live-smoke.mjs # pre-release live smokeBuilt with AI-assisted scaffolding; architecture, quality heuristics, tool contracts, and tests are hand-designed — see the ADRs for the reasoning.
License
Available Tools
7 toolsdataset_to_rag_documentsConvert dataset to RAG documentsARead-onlyIdempotent
Turn scraped items into embedding-ready documents: token-bounded chunks with overlap, source attribution, stable content-hash ids for idempotent vector upserts, and selected metadata. Paginate with offset/limit for large datasets.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | ||
| format | No | "json" emits one JSON document per line (jsonl), ready for embedding pipelines | json |
| offset | No | ||
| dataset_id | Yes | ||
| content_fields | No | Item fields to use as document text; auto-detects markdown/text/content when omitted | |
| overlap_tokens | No | ||
| metadata_fields | No | Item fields to carry into each document’s metadata | |
| max_response_tokens | No | ||
| max_tokens_per_chunk | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| documents | Yes | |
| truncated | Yes | |
| dataset_id | Yes | |
| next_offset | Yes | |
| total_tokens | Yes | |
| skipped_items | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The annotations already declare readOnlyHint and idempotentHint, and the description adds useful behavioral context by explaining the mechanism for idempotency ('stable content-hash ids') and the chunking/overlap behavior. It does not go into edge cases or failure modes, but the combination of annotations and description is solid.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, highly informative, front-loaded with the core transformation, and no filler. Every clause adds meaning: input, output, chunking, idempotency, metadata, pagination.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (9 parameters, transformation, output schema exists), the description covers the main operational aspects: chunking, overlap, idempotency, metadata, and pagination. It leaves some ambiguity about 'selected metadata' and exact chunking behavior, but the output schema and annotations fill in enough gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is only 33%, so the description must compensate. It does explain the purpose of chunking, overlap, metadata selection, and pagination, but it does not explicitly map these to parameter names, and parameters like max_response_tokens and format are left to the reader's inference. This is useful but not complete compensation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states a specific verb+resource transformation: 'Turn scraped items into embedding-ready documents'. It enumerates distinct output characteristics (token-bounded chunks, overlap, source attribution, content-hash ids, metadata) that clearly differentiate it from sibling tools like fetch_dataset_items or validate_dataset.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context for when to use the tool: after scraping, before embedding, and it advises pagination for large datasets. However, it does not explicitly name alternatives or state when not to use it, so it misses the explicit exclusion guidance that would earn a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
fetch_dataset_itemsFetch dataset itemsARead-onlyIdempotent
Read items from an actor run dataset with pagination, field projection, and a hard token budget. Start with response_format "summary" to see the shape cheaply, then fetch "items" with a fields projection. Items arrive as JSON in the text content; structured content carries pagination metadata (total, next_offset, truncated).
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | ||
| fields | No | Project only these top-level fields (cuts token cost drastically) | |
| offset | No | ||
| dataset_id | Yes | Dataset id from run_actor / scrape_url / get_run_status | |
| max_tokens | No | ||
| response_format | No | "summary" describes the data cheaply; "items" returns raw JSON items | summary |
Output Schema
| Name | Required | Description |
|---|---|---|
| total | Yes | |
| offset | Yes | |
| returned | Yes | |
| truncated | Yes | |
| dataset_id | Yes | |
| fields_seen | Yes | |
| next_offset | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already provide readOnlyHint, openWorldHint, idempotentHint. Description adds valuable behavioral details: response format (JSON in text content, pagination metadata in structured content), hard token budget, and pagination behavior. No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three concise sentences, front-loaded with main purpose, followed by actionable tips and return format. Every sentence earns its place; no fluff.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Tool has an output schema and the description covers the workflow, pagination metadata, and return format. It is complete for the complexity level, though it could briefly mention the dataset_id requirement. Still, the description is adequate and well-rounded.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 50%, and description compensates by explaining key parameter usage: response_format ('summary' vs 'items'), fields projection, token budget, and pagination. It adds strategic meaning beyond schema definitions, though limit/offset/max_tokens lack explicit descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description uses specific verb 'Read' + resource 'items from an actor run dataset', clearly distinguishing it from siblings like run_actor (executes) and get_run_status (status). The scope is unambiguous and action-oriented.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides explicit workflow guidance: 'Start with response_format summary... then fetch items with a fields projection.' This is clear practical usage context. However, it lacks explicit when-not-to-use or alternative tool comparisons, so not a full 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_run_statusGet actor run statusARead-onlyIdempotent
Check whether an actor run has finished and where its dataset is. Free and safe to poll.
| Name | Required | Description | Default |
|---|---|---|---|
| run_id | Yes | Run id returned by run_actor or scrape_url |
Output Schema
| Name | Required | Description |
|---|---|---|
| run_id | Yes | |
| status | Yes | |
| finished | Yes | |
| dataset_id | Yes | |
| started_at | Yes | |
| finished_at | Yes | |
| status_message | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly, openWorld, and idempotent hints. The description adds value by specifying 'Free and safe to poll' and disclosing that the response includes dataset location. This goes beyond the structured annotations without contradicting them.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two short sentences, front-loaded with the core purpose, and contains no filler. Every word contributes to understanding the tool's function and safety.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple status-checking tool with one parameter, rich annotations, and an output schema, the description is complete. It states what the tool does, what information it provides, and that it is safe to poll, fully covering the essential context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already provides 100% coverage for the single parameter (run_id) with a helpful description. The tool description adds no additional parameter meaning, so the baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's function: 'Check whether an actor run has finished and where its dataset is.' This is a specific verb+resource statement that distinguishes it from siblings like run_actor or fetch_dataset_items, which handle running or retrieving data.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context: it is for checking run completion and locating the dataset, implying use after triggering a run. It also notes it is 'free and safe to poll,' indicating suitability for repeated use. However, it does not explicitly mention when not to use it or name alternative tools for other purposes.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
retry_low_quality_runRetry a low-quality runA
Re-run an actor with progressively stronger anti-blocking settings (residential proxies, then a browser crawler) until the dataset quality score reaches the threshold or attempts are exhausted. Each attempt costs Apify credits.
| Name | Required | Description | Default |
|---|---|---|---|
| run_id | Yes | Run whose output quality was too low | |
| threshold | No | Stop retrying once the quality score reaches this value | |
| json_schema | No | JSON Schema items should satisfy; also drives the quality score | |
| max_attempts | No | Maximum number of re-runs (each escalates proxy/browser settings) |
Output Schema
| Name | Required | Description |
|---|---|---|
| attempts | Yes | |
| final_run_id | Yes | |
| final_quality | Yes | |
| initial_quality | Yes | |
| final_dataset_id | Yes | |
| reached_threshold | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description discloses significant behavioral details beyond annotations: progressive anti-blocking escalation (residential proxies, then browser crawler), termination after threshold or exhausting attempts, and that each attempt costs credits. Annotations only note non-read-only and non-destructive, lacking these important operational details.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loaded with the action and key mechanism, and ends with a critical cost warning. Every word earns its place with no unnecessary elaboration.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's moderate complexity, the description covers the algorithm, termination conditions, and cost. Output schema exists, so return values are not required in the description. The only minor gap is not specifying what happens after attempts are exhausted, but the description implies a stop condition.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, providing baseline 3. The description adds context for threshold and max_attempts by explaining the escalation and stopping criteria, and implies json_schema drives quality scoring. This adds meaningful semantics beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool re-runs an actor with progressively stronger anti-blocking settings until a quality threshold is met, distinguishing it from related tools like run_actor or scrape_url. The specific verb 're-run' and resource 'actor' make the purpose unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The title and description clearly imply use for low-quality runs that need retrying with escalation, but it does not explicitly state when not to use or mention alternatives. The context is clear enough for an agent to infer appropriate usage.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
run_actorRun an Apify actorA
Start an allowlisted Apify actor with an explicit input object. Returns run_id and dataset_id handles for get_run_status / fetch_dataset_items / validate_dataset. Costs Apify credits. Prefer scrape_url for simple page scrapes.
| Name | Required | Description | Default |
|---|---|---|---|
| input | No | Actor input object, passed through as-is | |
| actor_id | Yes | Actor to run, e.g. "apify/website-content-crawler" | |
| memory_mb | No | ||
| wait_secs | No | 0 = return immediately with a run handle; >0 = wait up to this long for completion | |
| timeout_secs | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| run_id | Yes | |
| status | Yes | |
| dataset_id | Yes | |
| status_message | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations indicate readOnlyHint=false and openWorldHint=true, so the description does not repeat those. It adds meaningful behaviors beyond annotations: 'Costs Apify credits' reveals a side effect not captured in annotations, and 'allowlisted' imposes a constraint on which actors can be run. No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is four concise sentences, each conveying essential information: action, return handles, cost, and alternative. It is front-loaded with the primary verb and avoids any redundancies or extraneous details. This is a model of efficient structure.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is a mutating, external operation with 5 parameters and an output schema. The description covers purpose, return handles, costs, and a usage alternative. Combined with schema descriptions for wait_secs and input, it gives sufficient context for selection and invocation. Gaps remain for memory_mb and timeout_secs semantics, but these are optional and have schema-specified ranges, making them less critical.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 60%; input, actor_id, and wait_secs have descriptions, but memory_mb and timeout_secs lack them. The tool description adds only minor parameter context (e.g., 'explicit input object' already in schema), and does not explain the purpose or effect of memory_mb or timeout_secs. It doesn't adequately compensate for the schema gaps.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states the action: 'Start an allowlisted Apify actor with an explicit input object.' It identifies the specific verb (start), resource (Apify actor), and scope (with input object). It also distinguishes from siblings by noting it returns run_id and dataset_id handles for follow-up tools, and explicitly prefers scrape_url for simple page scrapes.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides an explicit alternative: 'Prefer scrape_url for simple page scrapes.' This tells the agent when not to use this tool. It also implies when to use it: when a more complex custom actor run is needed. Mentions of get_run_status/fetch_dataset_items/validate_dataset indicate the workflow where this tool is the entry point, offering clear context on usage.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scrape_urlScrape a URL (quality-gated)A
Scrape a public web page into clean markdown in one call: runs a crawler, waits, scores the result (completeness, bot-wall detection), and automatically retries with stronger settings when quality is low. Costs Apify credits per run. For arbitrary actors or fire-and-forget runs use run_actor instead.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Public http(s) URL to scrape | |
| max_pages | No | How many pages to crawl starting from the URL (same site) | |
| max_tokens | No | Token budget for the returned page content | |
| quality_retry | No | Re-run with residential proxies / a browser crawler when quality is low |
Output Schema
| Name | Required | Description |
|---|---|---|
| pages | Yes | |
| run_id | Yes | |
| quality | Yes | |
| attempts | Yes | |
| truncated | Yes | |
| dataset_id | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description significantly enriches the sparse annotations. It discloses cost ('Costs Apify credits per run'), the multi-step behavior ('runs a crawler, waits, scores the result... automatically retries with stronger settings'), and quality criteria ('completeness, bot-wall detection'). This contextualizes the non-read-only and open-world hints beyond what annotations provide.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two concise sentences: the first states the core function and mechanism, the second covers cost and the sibling alternative. Every clause adds value without redundancy or fluff, making it well-structured and front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite the tool's moderate complexity (4 params, output schema, annotations), the description covers the purpose, workflow, retry logic, cost, and a key alternative. The output schema exists to define return values, so the description doesn't need to explain them. It is complete for an agent to select and invoke the tool appropriately.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already explains all four parameters clearly. The description adds useful pipeline context (e.g., 'automatically retries with stronger settings' clarifies quality_retry) but does not individually elaborate on parameter syntax or defaults beyond schema, so the baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's specific functionality: 'Scrape a public web page into clean markdown in one call.' It distinguishes from sibling tools by explicitly naming run_actor as the alternative for 'arbitrary actors or fire-and-forget runs.' This is a specific verb+resource+result with clear differentiation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit when-to-use guidance, noting it is for quality-gated scraping with automatic retries. It also gives an alternative: 'For arbitrary actors or fire-and-forget runs use run_actor instead.' While it doesn't address all siblings, the key distinction from the closest alternative is clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
validate_datasetValidate dataset qualityARead-onlyIdempotent
Score a dataset before trusting it: schema pass rate (if a JSON Schema is given), field completeness, duplicate rate, and bot-wall detection. Use the score to decide between consuming the data and retry_low_quality_run.
| Name | Required | Description | Default |
|---|---|---|---|
| dataset_id | Yes | ||
| json_schema | No | JSON Schema each item should satisfy; omit for schema-free quality metrics | |
| sample_size | No | How many items to inspect (from the start of the dataset) |
Output Schema
| Name | Required | Description |
|---|---|---|
| total | Yes | |
| quality | Yes | |
| sampled | Yes | |
| dataset_id | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly, idempotent, and openWorld hints. The description adds meaningful behavioral context by explaining what quality metrics are computed and how the score should influence the decision to consume or retry, without contradicting the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with the core purpose and followed by actionable advice. No filler or redundancy; every word contributes to clarity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With an output schema available, the description does not need to explain return values. It covers the tool's purpose, key inputs (optional schema), and decision process, which is sufficient for an agent to select and invoke the tool appropriately in most contexts.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 67%, with json_schema and sample_size described. The description adds value by clarifying that json_schema is optional and what role it plays in the score, but it does not explain dataset_id or sample_size beyond the schema. This is adequate but not exceptional.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb 'Score' with a clear resource ('a dataset') and lists concrete quality dimensions (schema pass rate, field completeness, duplicate rate, bot-wall detection). It distinguishes itself from siblings by tying directly to retry_low_quality_run, making its role unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear usage context: use it before trusting data, and use the resulting score to decide between consuming or retrying. It names a specific alternative (retry_low_quality_run), but does not explicitly state when not to use the tool versus other siblings like fetch_dataset_items.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
7 tool updates
v0.1.0- First observed
dataset_to_rag_documents - First observed
fetch_dataset_items - First observed
get_run_status - First observed
retry_low_quality_run - First observed
run_actor - First observed
scrape_url - First observed
validate_dataset
TDQS
Each tool has a clearly distinct role: run management, scraping, actor execution, data retrieval, quality validation, retry logic, and RAG conversion. Overlapping tools like scrape_url and run_actor are explicitly differentiated in their descriptions, so an agent can reliably select the correct one.
Most tools follow a consistent verb_noun snake_case pattern (get_run_status, scrape_url, run_actor, fetch_dataset_items, validate_dataset, retry_low_quality_run). The exception is dataset_to_rag_documents, which lacks a leading verb and breaks the pattern, though it remains readable and unambiguous.
Seven tools is a well-scoped count for a web data extraction and processing server. Each tool addresses a distinct step in the pipeline without redundancy, making the set feel complete yet not overwhelming.
The tool surface covers the full lifecycle: initiating extractions (scrape_url/run_actor), monitoring runs (get_run_status), fetching results (fetch_dataset_items), validating quality (validate_dataset), improving results (retry_low_quality_run), and preparing output for downstream use (dataset_to_rag_documents). No critical gaps are apparent for the stated purpose.
Maintenance
Resources
Unclaimed servers have limited discoverability.
Looking for Admin?
If you are the server author, to access and configure the admin panel.
Related MCP Connectors
Scrape, crawl and search the web for AI agents via MCP.
- UnifAPIOAuthcom.unifapi
Hosted MCP server for live public-data APIs and Skills for AI agents.
MCP server for web extraction and rendering via AceDataCloud WebExtrator
MCP server for building and testing AI agents with multi-model experimentation and insights.
Related MCP Servers
- FlicenseBqualityDmaintenanceAn MCP Server for Web scraping and Crawling, built using Crawl4AI224-
- AlicenseNot gradedqualityCmaintenanceWeb scraping and search MCP server that wraps Firecrawl API for URL discovery and web search with optional content retrieval.161MIT
- -licenseNot gradedqualityCmaintenanceSelf-hosted MCP server that provides web scraping and crawling tools, integrating seamlessly with AI frameworks like OpenAI Agents SDK, Cursor, and Claude Code.4-
- AlicenseBqualityDmaintenanceMCP server for the Spider web crawling and scraping API, enabling AI agents to crawl, scrape, search, and extract web data.13502MIT
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/fctpe/web-data-mcp'
If you have feedback or need assistance with the MCP directory API, please join our Discord server