Skip to main content
Glama
deenesjakoruzh
ScrapeGraphAI

ScrapeGraph MCP Server

Official
by ScrapeGraphAI
OverviewSchemaRelated ServersScoreDiscussions
Python
Remote

ScrapeGraph MCP Server

License: MIT Python 3.13+

A production-ready Model Context Protocol (MCP) server that provides seamless integration with the ScrapeGraph AI API. This server enables language models to leverage advanced AI-powered web scraping capabilities with enterprise-grade reliability.

Table of Contents

Related MCP server: OneSearch MCP Server

API v2

This MCP server targets ScrapeGraph API v2 (https://v2-api.scrapegraphai.com/api), aligned 1:1 with scrapegraph-py PR #84. Auth uses the SGAI-APIKEY header. Environment variables mirror the Python SDK:

  • SGAI_API_URL — override the base URL (default https://v2-api.scrapegraphai.com/api)

  • SGAI_TIMEOUT — request timeout in seconds (default 120)

  • SGAI_API_KEY — API key (can also be passed via MCP scrapegraphApiKey or X-API-Key header)

Legacy aliases (still honored): SCRAPEGRAPH_API_BASE_URL for SGAI_API_URL, SGAI_TIMEOUT_S for SGAI_TIMEOUT.

Key Features

  • Scrape & extract: scrape (POST /scrape, multi-format), extract (POST /extract, URL + prompt)

  • Search: search (POST /search; num_results clamped 3–20)

  • Crawl: Async multi-page crawl with crawl_start / crawl_get_status / crawl_stop / crawl_resume

  • Schema: schema (POST /schema) — generate or augment a JSON Schema from a prompt

  • Monitors: Scheduled jobs via monitor_create, monitor_list, monitor_get, pause/resume/delete, monitor_activity (paginated tick history)

  • Account: credits, history

  • Easy integration: Claude Desktop, Cursor, Smithery, HTTP transport

  • Developer docs: .agent/ folder

Migration: v2 → v3

v3 renames every MCP tool that diverged from the v2 API docs. Hard rename, no aliases.

v2 (old)

v3 (new)

smartscraper

extract

searchscraper

search

smartcrawler_initiate

crawl_start

smartcrawler_fetch_results

crawl_get_status

sgai_history

history

generate_schema

schema

markdownify

removed — use scrape with output_format="markdown"

Quick Start

1. Get Your API Key

Sign up and get your API key from the ScrapeGraph Dashboard

2. Install with Smithery (Recommended)

npx -y @smithery/cli install @ScrapeGraphAI/scrapegraph-mcp --client claude

3. Start Using

Ask Claude or Cursor:

  • "Convert https://scrapegraphai.com to markdown"

  • "Extract all product prices from this e-commerce page"

  • "Research the latest AI developments and summarize findings"

That's it! The server is now available to your AI assistant.

Available Tools

Tool

Role

scrape

POST /scrape (output_format: markdown, html, screenshot, branding, links, images, summary)

extract

POST /extract (requires website_url + user_prompt; optional output_schema)

search

POST /search (num_results 1–20; supports country_search, time_range, output_schema)

crawl_start

POST /crawl — extraction_mode markdown / html / links / images / summary / branding / screenshot

crawl_get_status

GET /crawl/:id (poll until status: completed)

crawl_stop, crawl_resume

POST /crawl/:id/stop | resume

schema

POST /schema (generate or augment a JSON Schema from a prompt)

credits

GET /credits

history

GET /history (paginated, service filter)

monitor_create, monitor_list, monitor_get, monitor_pause, monitor_resume, monitor_delete

/monitor API

monitor_activity

GET /monitor/:id/activity (paginated tick history: id, createdAt, status, changed, elapsedMs, diffs)

Removed: sitemap, agentic_scrapper, async-status polling, and (in v3) markdownify — use scrape with output_format="markdown".

Setup Instructions

To utilize this server, you'll need a ScrapeGraph API key. Follow these steps to obtain one:

  1. Navigate to the ScrapeGraph Dashboard

  2. Create an account and generate your API key

Automated Installation via Smithery

For automated installation of the ScrapeGraph API Integration Server using Smithery:

npx -y @smithery/cli install @ScrapeGraphAI/scrapegraph-mcp --client claude

Claude Desktop Configuration

Update your Claude Desktop configuration file with the following settings (located on the top rigth of the Cursor page):

(remember to add your API key inside the config)

{
    "mcpServers": {
        "@ScrapeGraphAI-scrapegraph-mcp": {
            "command": "npx",
            "args": [
                "-y",
                "@smithery/cli@latest",
                "run",
                "@ScrapeGraphAI/scrapegraph-mcp",
                "--config",
                "\"{\\\"scrapegraphApiKey\\\":\\\"YOUR-SGAI-API-KEY\\\"}\""
            ]
        }
    }
}

The configuration file is located at:

  • Windows: %APPDATA%/Claude/claude_desktop_config.json

  • macOS: ~/Library/Application\ Support/Claude/claude_desktop_config.json

Cursor Integration

Add the ScrapeGraphAI MCP server on the settings:

Cursor MCP Integration

Remote Server Usage

Connect to our hosted MCP server - no local installation required!

WARNING

The legacy MCP endpoint athttps://mcp.scrapegraphai.com/mcp will be deprecated soon. New integrations should use the replacement MCP endpoint: https://sgai-mcp-main.onrender.com.

Claude Desktop Configuration (Remote)

Add this to your Claude Desktop config (~/Library/Application Support/Claude/claude_desktop_config.json on macOS):

{
  "mcpServers": {
    "scrapegraph-mcp": {
      "command": "npx",
      "args": [
        "mcp-remote@0.1.25",
        "https://sgai-mcp-main.onrender.com",
        "--header",
        "X-API-Key:YOUR_API_KEY"
      ]
    }
  }
}

Cursor Configuration (Remote)

Cursor supports native HTTP MCP connections. Add to your Cursor MCP settings (~/.cursor/mcp.json):

{
  "mcpServers": {
    "scrapegraph-mcp": {
      "url": "https://sgai-mcp-main.onrender.com",
      "headers": {
        "X-API-Key": "YOUR_API_KEY"
      }
    }
  }
}

Benefits of Remote Server

  • No local setup - Just configure and start using

  • Always up-to-date - Automatically receives latest updates

  • Cross-platform - Works on any OS with Node.js

Local Usage

To run the MCP server locally for development or testing, follow these steps:

Prerequisites

  • Python 3.13 or higher

  • pip or uv package manager

  • ScrapeGraph API key

Installation

  1. Clone the repository (if you haven't already):

git clone https://github.com/ScrapeGraphAI/scrapegraph-mcp
cd scrapegraph-mcp

  1. Install the package:

# Using pip
pip install -e .

# Or using uv (faster)
uv pip install -e .

  1. Set your API key:

# macOS/Linux
export SGAI_API_KEY=your-api-key-here

# Windows (PowerShell)
$env:SGAI_API_KEY="your-api-key-here"

# Windows (CMD)
set SGAI_API_KEY=your-api-key-here

Running the Server Locally

You can run the server directly:

# Using the installed command
scrapegraph-mcp

# Or using Python module
python -m scrapegraph_mcp.server

The server will start and communicate via stdio (standard input/output), which is the standard MCP transport method.

Testing with MCP Inspector

Test your local server using the MCP Inspector tool:

npx @modelcontextprotocol/inspector python -m scrapegraph_mcp.server

This provides a web interface to test all available tools interactively.

Configuring Claude Desktop for Local Server

To use your locally running server with Claude Desktop, update your configuration file:

macOS/Linux (~/Library/Application Support/Claude/claude_desktop_config.json):

{
    "mcpServers": {
        "scrapegraph-mcp-local": {
            "command": "python",
            "args": [
                "-m",
                "scrapegraph_mcp.server"
            ],
            "env": {
                "SGAI_API_KEY": "your-api-key-here"
            }
        }
    }
}

Windows (%APPDATA%\Claude\claude_desktop_config.json):

{
    "mcpServers": {
        "scrapegraph-mcp-local": {
            "command": "python",
            "args": [
                "-m",
                "scrapegraph_mcp.server"
            ],
            "env": {
                "SGAI_API_KEY": "your-api-key-here"
            }
        }
    }
}

Note: Make sure Python is in your PATH. You can verify by running python --version in your terminal.

Configuring Cursor for Local Server

In Cursor's MCP settings, add a new server with:

  • Command: python

  • Args: ["-m", "scrapegraph_mcp.server"]

  • Environment Variables: {"SGAI_API_KEY": "your-api-key-here"}

Troubleshooting Local Setup

Server not starting:

  • Verify Python is installed: python --version

  • Check that the package is installed: pip list | grep scrapegraph-mcp

  • Ensure API key is set: echo $SGAI_API_KEY (macOS/Linux) or echo %SGAI_API_KEY% (Windows)

Tools not appearing:

  • Check Claude Desktop logs:

    • macOS: ~/Library/Logs/Claude/

    • Windows: %APPDATA%\Claude\Logs\

  • Verify the server starts without errors when run directly

  • Check that the configuration JSON is valid

Import errors:

  • Reinstall the package: pip install -e . --force-reinstall

  • Verify dependencies: pip install -r requirements.txt (if available)

Google ADK Integration

The ScrapeGraph MCP server can be integrated with Google ADK (Agent Development Kit) to create AI agents with web scraping capabilities.

Prerequisites

  • Python 3.13 or higher

  • Google ADK installed

  • ScrapeGraph API key

Installation

  1. Install Google ADK (if not already installed):

pip install google-adk

  1. Set your API key:

export SGAI_API_KEY=your-api-key-here

Basic Integration Example

Create an agent file (e.g., agent.py) with the following configuration:

import os
from google.adk.agents import LlmAgent
from google.adk.tools.mcp_tool.mcp_toolset import MCPToolset
from google.adk.tools.mcp_tool.mcp_session_manager import StdioConnectionParams
from mcp import StdioServerParameters

# Path to the scrapegraph-mcp server directory
SCRAPEGRAPH_MCP_PATH = "/path/to/scrapegraph-mcp"

# Path to the server.py file
SERVER_SCRIPT_PATH = os.path.join(
    SCRAPEGRAPH_MCP_PATH, 
    "src", 
    "scrapegraph_mcp", 
    "server.py"
)

root_agent = LlmAgent(
    model='gemini-2.0-flash',
    name='scrapegraph_assistant_agent',
    instruction='Help the user with web scraping and data extraction using ScrapeGraph AI. '
                'You can convert webpages to markdown, extract structured data using AI, '
                'perform web searches, crawl multiple pages, and automate complex scraping workflows.',
    tools=[
        MCPToolset(
            connection_params=StdioConnectionParams(
                server_params=StdioServerParameters(
                    command='python3',
                    args=[
                        SERVER_SCRIPT_PATH,
                    ],
                    env={
                        'SGAI_API_KEY': os.getenv('SGAI_API_KEY'),
                    },
                ),
                timeout=300.0,)
            ),
            # Optional: Filter which tools from the MCP server are exposed
            # tool_filter=['scrape', 'extract', 'search']
        )
    ],
)

Configuration Options

Timeout Settings:

  • Default timeout is 5 seconds, which may be too short for web scraping operations

  • Recommended: Set `timeout=300.0

  • Adjust based on your use case (crawling operations may need even longer timeouts)

Tool Filtering:

  • By default, all registered MCP tools are exposed to the agent (see Available Tools)

  • Use tool_filter to limit which tools are available:

    tool_filter=['scrape', 'extract', 'search']

API Key Configuration:

  • Set via environment variable: export SGAI_API_KEY=your-key

  • Or pass directly in env dict: 'SGAI_API_KEY': 'your-key-here'

  • Environment variable approach is recommended for security

Usage Example

Once configured, your agent can use natural language to interact with web scraping tools:

# The agent can now handle queries like:
# - "Convert https://example.com to markdown"
# - "Extract all product prices from this e-commerce page"
# - "Search for recent AI research papers and summarize them"
# - "Crawl this documentation site and extract all API endpoints"

For more information about Google ADK, visit the official documentation.

Example Use Cases

The server enables sophisticated queries across various scraping scenarios:

Single Page Scraping

  • Markdownify: "Convert the ScrapeGraph documentation page to markdown"

  • Extract: "Extract all product names, prices, and ratings from this e-commerce page"

  • Extract with scrolling: "Scrape this infinite scroll page with 5 scrolls and extract all items"

  • Basic Scrape: "Fetch the HTML content of this JavaScript-heavy page with full rendering"

Search and Research

  • Search: "Research and summarize recent developments in AI-powered web scraping"

  • Search: "Search for the top 5 articles about machine learning frameworks and extract key insights"

  • Search: "Find recent news about GPT-4 and provide a structured summary"

  • Search: v2 does not apply time_range; phrase queries to bias recency in natural language instead

Website analysis

  • Use crawl_start plus crawl_get_status to map and capture multi-page content; there is no separate sitemap tool on v2.

Multi-page crawling

  • Crawl: "Crawl the blog in markdown mode and poll until complete"

  • For structured fields per page, run extract on individual URLs (or monitor_create on a schedule)

Monitors and account

  • Monitor: "Run this extract prompt on https://example.com every day at 9am" (monitor_create with interval)

  • Credits / history: credits, history

  • Agentic Scraper: "Execute a complex workflow: login, navigate to reports, download data, and extract summary statistics"

Error Handling

The server implements robust error handling with detailed, actionable error messages for:

  • API authentication issues

  • Malformed URL structures

  • Network connectivity failures

  • Rate limiting and quota management

Common Issues

Windows-Specific Connection

When running on Windows systems, you may need to use the following command to connect to the MCP server:

C:\Windows\System32\cmd.exe /c npx -y @smithery/cli@latest run @ScrapeGraphAI/scrapegraph-mcp --config "{\"scrapegraphApiKey\":\"YOUR-SGAI-API-KEY\"}"

This ensures proper execution in the Windows environment.

Other Common Issues

"ScrapeGraph client not initialized"

  • Cause: Missing API key

  • Solution: Set SGAI_API_KEY environment variable or provide via --config

"Error 401: Unauthorized"

"Error 402: Payment Required"

  • Cause: Insufficient credits

  • Solution: Add credits to your ScrapeGraph account

Crawl not returning results

  • Cause: Still processing (asynchronous operation)

  • Solution: Keep polling crawl_get_status() until status is "completed"

Tools not appearing in Claude Desktop

  • Cause: Server not starting or configuration error

  • Solution: Check Claude logs at ~/Library/Logs/Claude/ (macOS) or %APPDATA%\Claude\Logs\ (Windows)

For detailed troubleshooting, see the .agent documentation.

Development

Prerequisites

  • Python 3.13 or higher

  • pip or uv package manager

  • ScrapeGraph API key

Installation from Source

# Clone the repository
git clone https://github.com/ScrapeGraphAI/scrapegraph-mcp
cd scrapegraph-mcp

# Install dependencies
pip install -e ".[dev]"

# Set your API key
export SGAI_API_KEY=your-api-key

# Run the server
scrapegraph-mcp
# or
python -m scrapegraph_mcp.server

Testing with MCP Inspector

Test your server locally using the MCP Inspector tool:

npx @modelcontextprotocol/inspector scrapegraph-mcp

This provides a web interface to test all available tools.

Code Quality

Linting:

ruff check src/

Type Checking:

mypy src/

Format Checking:

ruff format --check src/

Project Structure

scrapegraph-mcp/
├── src/
│   └── scrapegraph_mcp/
│       ├── __init__.py      # Package initialization
│       └── server.py        # Main MCP server (all code in one file)
├── .agent/                  # Developer documentation
│   ├── README.md           # Documentation index
│   └── system/             # System architecture docs
├── assets/                  # Images and badges
├── pyproject.toml          # Project metadata & dependencies
├── smithery.yaml           # Smithery deployment config
└── README.md               # This file

Contributing

We welcome contributions! Here's how you can help:

Adding a New Tool

  1. Add method to ScapeGraphClient class in server.py:

def new_tool(self, param: str) -> Dict[str, Any]:
    """Tool description."""
    url = f"{self.BASE_URL}/new-endpoint"
    data = {"param": param}
    response = self.client.post(url, headers=self.headers, json=data)
    if response.status_code != 200:
        raise Exception(f"Error {response.status_code}: {response.text}")
    return response.json()

  1. Add MCP tool decorator:

@mcp.tool()
def new_tool(param: str) -> Dict[str, Any]:
    """
    Tool description for AI assistants.

    Args:
        param: Parameter description

    Returns:
        Dictionary containing results
    """
    if scrapegraph_client is None:
        return {"error": "ScrapeGraph client not initialized. Please provide an API key."}

    try:
        return scrapegraph_client.new_tool(param)
    except Exception as e:
        return {"error": str(e)}

  1. Test with MCP Inspector:

npx @modelcontextprotocol/inspector scrapegraph-mcp

  1. Update documentation:

  2. Submit a pull request

Development Workflow

  1. Fork the repository

  2. Create a feature branch (git checkout -b feature/amazing-feature)

  3. Make your changes

  4. Run linting and type checking

  5. Test with MCP Inspector and Claude Desktop

  6. Update documentation

  7. Commit your changes (git commit -m 'Add amazing feature')

  8. Push to the branch (git push origin feature/amazing-feature)

  9. Open a Pull Request

Code Style

  • Line length: 100 characters

  • Type hints: Required for all functions

  • Docstrings: Google-style docstrings

  • Error handling: Return error dicts, don't raise exceptions in tools

  • Python version: Target 3.13+

For detailed development guidelines, see the .agent documentation.

Documentation

For comprehensive developer documentation, see:

Technology Stack

Core Framework

  • Python 3.13+ - Modern Python with type hints

  • FastMCP - Lightweight MCP server framework

  • httpx 0.24.0+ - Modern async HTTP client

Development Tools

  • Ruff - Fast Python linter and formatter

  • mypy - Static type checker

  • Hatchling - Modern build backend

Deployment

  • Smithery - Automated MCP server deployment

  • Docker - Container support with Alpine Linux

  • stdio transport - Standard MCP communication

API Integration

  • ScrapeGraph AI API - Enterprise web scraping service

  • Base URL: https://v2-api.scrapegraphai.com/api

  • Authentication: API key-based

License

This project is distributed under the MIT License. For detailed terms and conditions, please refer to the LICENSE file.

Acknowledgments

Special thanks to tomekkorbak for his implementation of oura-mcp-server, which served as starting point for this repo.

Resources

Official Links

MCP Resources

AI Assistant Integration

Support

Made with ❤️ by ScrapeGraphAI Team

Available Tools

8 tools
agentic_scrapperAInspect

Execute complex multi-step web scraping workflows with AI-powered automation.

This tool runs an intelligent agent that can navigate websites, interact with forms and buttons, follow multi-step workflows, and extract structured data. Ideal for complex scraping scenarios requiring user interaction simulation, form submissions, or multi-page navigation flows. Supports custom output schemas and step-by-step instructions. Variable credit cost based on complexity. Can perform actions on the website (non-read-only, non-idempotent).

The agent accepts flexible input formats for steps (list or JSON string) and output_schema (dict or JSON string) to accommodate different client implementations.

Args: url (str): The target website URL where the agentic scraping workflow should start. - Must include protocol (http:// or https://) - Should be the starting page for your automation workflow - The agent will begin its actions from this URL - Examples: * https://example.com/search (start at search page) * https://shop.example.com/login (begin with login flow) * https://app.example.com/dashboard (start at main interface) * https://forms.example.com/contact (begin at form page) - Considerations: * Choose a starting point that makes sense for your workflow * Ensure the page is publicly accessible or handle authentication * Consider the logical flow of actions from this starting point

user_prompt (Optional[str]): High-level instructions for what the agent should accomplish.
    - Describes the overall goal and desired outcome of the automation
    - Should be clear and specific about what you want to achieve
    - Works in conjunction with the steps parameter for detailed guidance
    - Examples:
      * "Navigate to the search page, search for laptops, and extract the top 5 results with prices"
      * "Fill out the contact form with sample data and submit it"
      * "Login to the dashboard and extract all recent notifications"
      * "Browse the product catalog and collect information about all items"
      * "Navigate through the multi-step checkout process and capture each step"
    - Tips for better results:
      * Be specific about the end goal
      * Mention what data you want extracted
      * Include context about the expected workflow
      * Specify any particular elements or sections to focus on

output_schema (Optional[Union[str, Dict]]): Desired output structure for extracted data.
    - Can be provided as a dictionary or JSON string
    - Defines the format and structure of the final extracted data
    - Helps ensure consistent, predictable output format
    - IMPORTANT: Must include a "required" field (can be empty array [] if no fields are required)
    - Examples:
      * Simple object: {'type': 'object', 'properties': {'title': {'type': 'string'}, 'price': {'type': 'number'}}, 'required': []}
      * Array of objects: {'type': 'array', 'items': {'type': 'object', 'properties': {'name': {'type': 'string'}, 'value': {'type': 'string'}}, 'required': []}, 'required': []}
      * Complex nested: {'type': 'object', 'properties': {'products': {'type': 'array', 'items': {...}}, 'total_count': {'type': 'number'}}, 'required': []}
      * As JSON string: '{"type": "object", "properties": {"results": {"type": "array"}}, "required": []}'
      * With required fields: {'type': 'object', 'properties': {'id': {'type': 'string'}, 'name': {'type': 'string'}}, 'required': ['id']}
    - Note: If "required" field is missing, it will be automatically added as an empty array []
    - Default: None (agent will infer structure from prompt and steps)

steps (Optional[Union[str, List[str]]]): Step-by-step instructions for the agent.
    - Can be provided as a list of strings or JSON array string
    - Provides detailed, sequential instructions for the automation workflow
    - Each step should be a clear, actionable instruction
    - Examples as list:
      * ['Click the search button', 'Enter "laptops" in the search box', 'Press Enter', 'Wait for results to load', 'Extract product information']
      * ['Fill in email field with test@example.com', 'Fill in password field', 'Click login button', 'Navigate to profile page']
    - Examples as JSON string:
      * '["Open navigation menu", "Click on Products", "Select category filters", "Extract all product data"]'
    - Best practices:
      * Break complex actions into simple steps
      * Be specific about UI elements (button text, field names, etc.)
      * Include waiting/loading steps when necessary
      * Specify extraction points clearly
      * Order steps logically for the workflow

ai_extraction (Optional[bool]): Enable AI-powered extraction mode for intelligent data parsing.
    - Default: true (recommended for most use cases)
    - Options:
      * true: Uses advanced AI to intelligently extract and structure data
        - Better at handling complex page layouts
        - Can adapt to different content structures
        - Provides more accurate data extraction
        - Recommended for most scenarios
      * false: Uses simpler extraction methods
        - Faster processing but less intelligent
        - May miss complex or nested data
        - Use when speed is more important than accuracy
    - Performance impact:
      * true: Higher processing time but better results
      * false: Faster execution but potentially less accurate extraction

persistent_session (Optional[bool]): Maintain session state between steps.
    - Default: false (each step starts fresh)
    - Options:
      * true: Keeps cookies, login state, and session data between steps
        - Essential for authenticated workflows
        - Maintains shopping cart contents, user preferences, etc.
        - Required for multi-step processes that depend on previous actions
        - Use for: Login flows, shopping processes, form wizards
      * false: Each step starts with a clean session
        - Faster and simpler for independent actions
        - No state carried between steps
        - Use for: Simple data extraction, public content scraping
    - Examples when to use true:
      * Login → Navigate to protected area → Extract data
      * Add items to cart → Proceed to checkout → Extract order details
      * Multi-step form completion with session dependencies

timeout_seconds (Optional[float]): Maximum time to wait for the entire workflow.
    - Default: 120 seconds (2 minutes)
    - Recommended ranges:
      * 60-120: Simple workflows (2-5 steps)
      * 180-300: Medium complexity (5-10 steps)
      * 300-600: Complex workflows (10+ steps or slow sites)
      * 600+: Very complex or slow-loading workflows
    - Considerations:
      * Include time for page loads, form submissions, and processing
      * Factor in network latency and site response times
      * Allow extra time for AI processing and extraction
      * Balance between thoroughness and efficiency
    - Examples:
      * 60.0: Quick single-page data extraction
      * 180.0: Multi-step form filling and submission
      * 300.0: Complex navigation and comprehensive data extraction
      * 600.0: Extensive workflows with multiple page interactions

Returns: Dictionary containing: - extracted_data: The structured data matching your prompt and optional schema - workflow_log: Detailed log of all actions performed by the agent - pages_visited: List of URLs visited during the workflow - actions_performed: Summary of interactions (clicks, form fills, navigations) - execution_time: Total time taken for the workflow - steps_completed: Number of steps successfully executed - final_page_url: The URL where the workflow ended - session_data: Session information if persistent_session was enabled - credits_used: Number of credits consumed (varies by complexity) - status: Success/failure status with any error details

Raises: ValueError: If URL is malformed or required parameters are missing TimeoutError: If the workflow exceeds the specified timeout NavigationError: If the agent cannot navigate to required pages InteractionError: If the agent cannot interact with specified elements ExtractionError: If data extraction fails or returns invalid results

Use Cases: - Automated form filling and submission - Multi-step checkout processes - Login-protected content extraction - Interactive search and filtering workflows - Complex navigation scenarios requiring user simulation - Data collection from dynamic, JavaScript-heavy applications

Best Practices: - Start with simple workflows and gradually increase complexity - Use specific element identifiers in steps (button text, field labels) - Include appropriate wait times for page loads and dynamic content - Test with persistent_session=true for authentication-dependent workflows - Set realistic timeouts based on workflow complexity - Provide clear, sequential steps that build on each other - Use output_schema to ensure consistent data structure

Note: - This tool can perform actions on websites (non-read-only) - Results may vary between runs due to dynamic content (non-idempotent) - Credit cost varies based on workflow complexity and execution time - Some websites may have anti-automation measures that could affect success - Consider using simpler tools (smartscraper, markdownify) for basic extraction needs

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
user_promptNo
output_schemaNo
stepsNo
ai_extractionNo
persistent_sessionNo
timeout_secondsNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description adds significant behavioral context beyond annotations. While annotations indicate non-readOnly and non-idempotent, the description elaborates on credit costs, anti-automation measures, result variability, and specific capabilities like session persistence and AI extraction modes. It also details return structure and potential errors, providing comprehensive operational insight.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured with clear sections (Args, Returns, Raises, Use Cases, Best Practices, Note) and front-loads key information. However, it is lengthy due to comprehensive parameter details; some redundancy exists (e.g., repeating examples), but most content earns its place by adding value.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (7 parameters, no schema descriptions, non-readOnly/non-idempotent annotations), the description is exceptionally complete. It covers purpose, usage, parameters, returns, errors, use cases, best practices, and limitations. The presence of an output schema reduces need to explain returns, and the description fills all other gaps thoroughly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 0% schema description coverage, the description fully compensates by providing detailed semantics for all 7 parameters. Each parameter includes explanations, examples, tips, and default behaviors (e.g., ai_extraction default true, timeout_seconds default 120). This adds substantial meaning beyond the bare schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Execute complex multi-step web scraping workflows with AI-powered automation.' It specifies the verb ('execute'), resource ('web scraping workflows'), and distinguishes from siblings by emphasizing multi-step, interactive capabilities versus simpler extraction tools like smartscraper or markdownify.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly provides when to use this tool ('Ideal for complex scraping scenarios requiring user interaction simulation, form submissions, or multi-page navigation flows') and when not to use it ('Consider using simpler tools (smartscraper, markdownify) for basic extraction needs'). It also lists specific use cases and best practices for guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

markdownifyA
Read-onlyIdempotent
Inspect

Convert a webpage into clean, formatted markdown.

This tool fetches any webpage and converts its content into clean, readable markdown format. Useful for extracting content from documentation, articles, and web pages for further processing. Costs 2 credits per page. Read-only operation with no side effects.

Args: website_url (str): The complete URL of the webpage to convert to markdown format. - Must include protocol (http:// or https://) - Supports most web content types (HTML, articles, documentation) - Works with both static and dynamic content - Examples: * https://example.com/page * https://docs.python.org/3/tutorial/ * https://github.com/user/repo/README.md - Invalid examples: * example.com (missing protocol) * ftp://example.com (unsupported protocol) * localhost:3000 (missing protocol)

Returns: Dictionary containing: - markdown: The converted markdown content as a string - metadata: Additional information about the conversion (title, description, etc.) - status: Success/error status of the operation - credits_used: Number of credits consumed (always 2 for this operation)

Raises: ValueError: If website_url is malformed or missing protocol HTTPError: If the webpage cannot be accessed or returns an error TimeoutError: If the webpage takes too long to load (>120 seconds)

ParametersJSON Schema
NameRequiredDescriptionDefault
website_urlYes

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description adds significant behavioral context beyond what annotations provide. While annotations indicate read-only, idempotent, and non-destructive operations, the description adds: cost information ('Costs 2 credits per page'), timeout behavior ('>120 seconds'), error conditions (ValueError, HTTPError, TimeoutError), and specific constraints about protocol requirements. This provides rich operational context not captured in annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured with clear sections (purpose, usage, cost, args, returns, raises) and front-loaded key information. While comprehensive, it's appropriately sized for a tool with complex behavior and parameter requirements. Some sentences could potentially be more concise, but overall structure is excellent.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity, the description provides complete context. It covers purpose, usage guidelines, behavioral traits, parameter semantics, return values, and error conditions. With an output schema present, the return value documentation is appropriately detailed but not redundant. The description addresses all aspects needed for effective tool use.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 0% schema description coverage, the description fully compensates by providing extensive parameter documentation. It explains the website_url parameter's format requirements, provides valid and invalid examples, and details content type support. This adds substantial meaning beyond the bare schema definition.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose with specific verb ('convert') and resource ('webpage into clean, formatted markdown'). It distinguishes from sibling tools by specifying this is specifically for markdown conversion rather than general scraping or crawling operations.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context for when to use this tool ('Useful for extracting content from documentation, articles, and web pages for further processing'). However, it doesn't explicitly state when NOT to use it or name specific alternatives among the sibling tools, though the purpose differentiation implies alternatives exist.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

scrapeA
Read-onlyIdempotent
Inspect

Fetch raw page content from any URL with optional JavaScript rendering.

This tool performs basic web scraping to retrieve the raw HTML content of a webpage. Optionally enable JavaScript rendering for Single Page Applications (SPAs) and sites with heavy client-side rendering. Lower cost than AI extraction (1 credit/page). Read-only operation with no side effects.

Args: website_url (str): The complete URL of the webpage to scrape. - Must include protocol (http:// or https://) - Returns raw HTML content of the page - Works with both static and dynamic websites - Examples: * https://example.com/page * https://api.example.com/docs * https://news.site.com/article/123 * https://app.example.com/dashboard (may need render_heavy_js=true) - Supported protocols: HTTP, HTTPS - Invalid examples: * example.com (missing protocol) * ftp://example.com (unsupported protocol)

render_heavy_js (Optional[bool]): Enable full JavaScript rendering for dynamic content.
    - Default: false (faster, lower cost, works for most static sites)
    - Set to true for sites that require JavaScript execution to display content
    - When to use true:
      * Single Page Applications (React, Angular, Vue.js)
      * Sites with dynamic content loading via AJAX
      * Content that appears only after JavaScript execution
      * Interactive web applications
      * Sites where initial HTML is mostly empty
    - When to use false (default):
      * Static websites and blogs
      * Server-side rendered content
      * Traditional HTML pages
      * News articles and documentation
      * When you need faster processing
    - Performance impact:
      * false: 2-5 seconds processing time
      * true: 15-30 seconds processing time (waits for JS execution)
    - Cost: Same (1 credit) regardless of render_heavy_js setting

Returns: Dictionary containing: - html_content: The raw HTML content of the page as a string - page_title: Extracted page title if available - status_code: HTTP response status code (200 for success) - final_url: Final URL after any redirects - content_length: Size of the HTML content in bytes - processing_time: Time taken to fetch and process the page - javascript_rendered: Whether JavaScript rendering was used - credits_used: Number of credits consumed (always 1)

Raises: ValueError: If website_url is malformed or missing protocol HTTPError: If the webpage returns an error status (404, 500, etc.) TimeoutError: If the page takes too long to load ConnectionError: If the website cannot be reached

Use Cases: - Getting raw HTML for custom parsing - Checking page structure before using other tools - Fetching content for offline processing - Debugging website content issues - Pre-processing before AI extraction

Note: - This tool returns raw HTML without any AI processing - Use smartscraper for structured data extraction - Use markdownify for clean, readable content - Consider render_heavy_js=true if initial results seem incomplete

ParametersJSON Schema
NameRequiredDescriptionDefault
website_urlYes
render_heavy_jsNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, covering safety and idempotency. The description adds valuable context beyond this: it specifies cost (1 credit/page), performance impact (2-5 sec vs. 15-30 sec), and that it's 'lower cost than AI extraction,' which helps the agent make informed decisions without contradicting annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured with sections (Args, Returns, Raises, Use Cases, Note) and front-loaded key information. However, it is lengthy with detailed examples and lists; while informative, some redundancy (e.g., repeating cost info) slightly reduces efficiency, though every sentence adds value.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (web scraping with JS rendering), the description is highly complete. It covers purpose, usage, parameters, returns (though output schema exists), error handling, and sibling differentiation. With annotations and output schema provided, the description adds comprehensive context without gaps, making it fully adequate for agent use.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 0% schema description coverage, the description fully compensates by detailing both parameters. For website_url, it explains requirements (protocol inclusion), examples, supported/unsupported protocols, and behavior. For render_heavy_js, it provides default values, usage scenarios, performance impacts, and cost implications, adding significant meaning beyond the bare schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Fetch raw page content from any URL with optional JavaScript rendering' and 'performs basic web scraping to retrieve the raw HTML content of a webpage.' It distinguishes from siblings by mentioning alternatives like smartscraper for structured data extraction and markdownify for clean content, making the scope specific.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit guidance on when to use this tool vs. alternatives: 'Use smartscraper for structured data extraction' and 'Use markdownify for clean, readable content.' It also details when to enable JavaScript rendering for SPAs vs. static sites, offering clear context and exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

searchscraperA
Read-only
Inspect

Perform AI-powered web searches with structured data extraction.

This tool searches the web based on your query and uses AI to extract structured information from the search results. Ideal for research, competitive analysis, and gathering information from multiple sources. Each website searched costs 10 credits (default 3 websites = 30 credits). Read-only operation but results may vary over time (non-idempotent).

Args: user_prompt (str): Search query or natural language instructions for information to find. - Can be a simple search query or detailed extraction instructions - The AI will search the web and extract relevant data from found pages - Be specific about what information you want extracted - Examples: * "Find latest AI research papers published in 2024 with author names and abstracts" * "Search for Python web scraping tutorials with ratings and difficulty levels" * "Get current cryptocurrency prices and market caps for top 10 coins" * "Find contact information for tech startups in San Francisco" * "Search for job openings for data scientists with salary information" - Tips for better results: * Include specific fields you want extracted * Mention timeframes or filters (e.g., "latest", "2024", "top 10") * Specify data types needed (prices, dates, ratings, etc.)

num_results (Optional[int]): Number of websites to search and extract data from.
    - Default: 3 websites (costs 30 credits total)
    - Range: 1-20 websites (recommended to stay under 10 for cost efficiency)
    - Each website costs 10 credits, so total cost = num_results × 10
    - Examples:
      * 1: Quick single-source lookup (10 credits)
      * 3: Standard research (30 credits) - good balance of coverage and cost
      * 5: Comprehensive research (50 credits)
      * 10: Extensive analysis (100 credits)
    - Note: More results provide broader coverage but increase costs and processing time

number_of_scrolls (Optional[int]): Number of infinite scrolls per searched webpage.
    - Default: 0 (no scrolling on search result pages)
    - Range: 0-10 scrolls per page
    - Useful when search results point to pages with dynamic content loading
    - Each scroll waits for content to load before continuing
    - Examples:
      * 0: Static content pages, news articles, documentation
      * 2: Social media pages, product listings with lazy loading
      * 5: Extensive feeds, long-form content with infinite scroll
    - Note: Increases processing time significantly (adds 5-10 seconds per scroll per page)

Returns: Dictionary containing: - search_results: Array of extracted data from each website found - sources: List of URLs that were searched and processed - total_websites_processed: Number of websites successfully analyzed - credits_used: Total credits consumed (num_results × 10) - processing_time: Total time taken for search and extraction - search_query_used: The actual search query sent to search engines - metadata: Additional information about the search process

Raises: ValueError: If user_prompt is empty or num_results is out of range HTTPError: If search engines are unavailable or return errors TimeoutError: If search or extraction process exceeds timeout limits RateLimitError: If too many requests are made in a short time period

Note: - Results may vary between calls due to changing web content (non-idempotent) - Search engines may return different results over time - Some websites may be inaccessible or block automated access - Processing time increases with num_results and number_of_scrolls - Consider using smartscraper on specific URLs if you know the target sites

ParametersJSON Schema
NameRequiredDescriptionDefault
user_promptYes
num_resultsNo
number_of_scrollsNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description adds substantial behavioral context beyond annotations: cost structure (10 credits per website), non-idempotence details ('results may vary over time'), processing time implications, accessibility constraints ('some websites may be inaccessible'), and specific error conditions. While annotations cover read-only and non-idempotent hints, the description provides operational details that help the agent make informed decisions.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured with clear sections (Args, Returns, Raises, Note) and front-loaded key information. While comprehensive, some sections could be more concise (e.g., multiple similar examples). Every sentence adds value, but the overall length is substantial for a tool description.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (web search with AI extraction, cost structure, multiple parameters) and the presence of output schema, the description is exceptionally complete. It covers purpose, usage, parameters, returns, errors, cost implications, performance considerations, and sibling tool differentiation - providing everything an agent needs to use this tool effectively.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 0% schema description coverage, the description fully compensates by providing comprehensive parameter documentation: detailed explanations, examples, tips, ranges, defaults, and cost implications for all three parameters. The description adds significant meaning beyond the bare schema, including practical guidance for better results and trade-offs between parameters.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool performs 'AI-powered web searches with structured data extraction' - a specific verb (search/extract) and resource (web). It distinguishes from siblings like 'smartscraper' (for specific URLs) and 'scrape' (generic scraping) by emphasizing AI-powered search and extraction from search results.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicit guidance is provided: 'Ideal for research, competitive analysis, and gathering information from multiple sources' and 'Consider using smartscraper on specific URLs if you know the target sites.' The description clearly distinguishes when to use this tool (broad web searches) vs. alternatives (targeted scraping with smartscraper).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

sitemapA
Read-onlyIdempotent
Inspect

Extract and discover the complete sitemap structure of any website.

This tool automatically discovers all accessible URLs and pages within a website, providing a comprehensive map of the site's structure. Useful for understanding site architecture before crawling or for discovering all available content. Very cost-effective at 1 credit per request. Read-only operation with no side effects.

Args: website_url (str): The base URL of the website to extract sitemap from. - Must include protocol (http:// or https://) - Should be the root domain or main section you want to map - The tool will discover all accessible pages from this starting point - Examples: * https://example.com (discover entire website structure) * https://docs.example.com (map documentation site) * https://blog.company.com (discover all blog pages) * https://shop.example.com (map e-commerce structure) - Best practices: * Use root domain (https://example.com) for complete site mapping * Use subdomain (https://docs.example.com) for focused mapping * Ensure the URL is accessible and doesn't require authentication - Discovery methods: * Checks for robots.txt and sitemap.xml files * Crawls navigation links and menus * Discovers pages through internal link analysis * Identifies common URL patterns and structures

Returns: Dictionary containing: - discovered_urls: List of all URLs found on the website - site_structure: Hierarchical organization of pages and sections - url_categories: URLs grouped by type (pages, images, documents, etc.) - total_pages: Total number of pages discovered - subdomains: List of subdomains found (if any) - sitemap_sources: Sources used for discovery (sitemap.xml, robots.txt, crawling) - page_types: Breakdown of different content types found - depth_analysis: URL organization by depth from root - external_links: Links pointing to external domains (if found) - processing_time: Time taken to complete the discovery - credits_used: Number of credits consumed (always 1)

Raises: ValueError: If website_url is malformed or missing protocol HTTPError: If the website cannot be accessed or returns errors TimeoutError: If the discovery process takes too long ConnectionError: If the website cannot be reached

Use Cases: - Planning comprehensive crawling operations - Understanding website architecture and organization - Discovering all available content before targeted scraping - SEO analysis and site structure optimization - Content inventory and audit preparation - Identifying pages for bulk processing operations

Best Practices: - Run sitemap before using smartcrawler_initiate for better planning - Use results to set appropriate max_pages and depth parameters - Check discovered URLs to understand site organization - Identify high-value pages for targeted extraction - Use for cost estimation before large crawling operations

Note: - Very cost-effective at only 1 credit per request - Results may vary based on site structure and accessibility - Some pages may require authentication and won't be discovered - Large sites may have thousands of URLs - consider filtering results - Use discovered URLs as input for other scraping tools

ParametersJSON Schema
NameRequiredDescriptionDefault
website_urlYes

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description adds valuable behavioral context beyond annotations. Annotations indicate read-only, idempotent, and non-destructive operations, but the description elaborates on cost ('1 credit per request'), discovery methods (e.g., checking robots.txt, crawling links), limitations (pages requiring authentication won't be discovered), and performance considerations (large sites may have thousands of URLs). No contradictions with annotations exist.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured with clear sections (Args, Returns, Raises, Use Cases, Best Practices, Note), but it is lengthy. While most sentences add value (e.g., explaining cost-effectiveness, discovery methods, limitations), some redundancy exists (e.g., repeating 'cost-effective' in multiple sections), slightly reducing efficiency.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (website mapping), the description is highly complete. It covers purpose, usage, parameters, return values (detailed in Returns section), error handling (Raises), practical applications (Use Cases), and operational notes. With annotations and an output schema present, the description provides all necessary contextual information without gaps.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 0% schema description coverage for the single parameter 'website_url', the description fully compensates by providing extensive semantic details. It explains the parameter's purpose, format requirements (must include protocol), usage examples (e.g., root domain vs. subdomain), best practices, and discovery methods, adding significant value beyond the bare schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Extract and discover the complete sitemap structure of any website.' It specifies the verb ('extract and discover'), resource ('sitemap structure'), and scope ('any website'), and distinguishes it from siblings like 'smartcrawler_initiate' by focusing on comprehensive mapping rather than targeted crawling.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit guidance on when to use this tool versus alternatives. It states: 'Useful for understanding site architecture before crawling or for discovering all available content,' and under 'Best Practices' advises: 'Run sitemap before using smartcrawler_initiate for better planning.' This clearly positions it as a preparatory tool for other scraping operations.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

smartcrawler_fetch_resultsA
Read-onlyIdempotent
Inspect

Retrieve the results of an asynchronous SmartCrawler operation.

This tool fetches the results from a previously initiated crawling operation using the request_id. The crawl request processes asynchronously in the background. Keep polling this endpoint until the status field indicates 'completed'. While processing, you'll receive status updates. Read-only operation that safely retrieves results without side effects.

Args: request_id: The unique request ID returned by smartcrawler_initiate. Use this to retrieve the crawling results. Keep polling until status is 'completed'. Example: 'req_abc123xyz'

Returns: Dictionary containing: - status: Current status of the crawl operation ('processing', 'completed', 'failed') - results: Crawled data (structured extraction or markdown) when completed - metadata: Information about processed pages, URLs visited, and processing statistics Keep polling until status is 'completed' to get final results

ParametersJSON Schema
NameRequiredDescriptionDefault
request_idYes

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

While annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, the description adds valuable behavioral context beyond annotations: it explains the asynchronous nature of the operation, polling requirements, status tracking, and safe retrieval without side effects. The description doesn't contradict annotations and provides operational details not captured in structured fields.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Well-structured with clear sections (purpose, usage instructions, Args, Returns) and front-loaded key information. Some repetition of 'Keep polling until status is 'completed'' could be reduced, but overall each sentence adds value. The description is appropriately sized for the tool's complexity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool has annotations covering safety profile, an output schema exists (so return values don't need explanation), and the description provides comprehensive parameter semantics and usage guidelines, this description is complete. It covers the asynchronous nature, polling behavior, parameter meaning, and distinguishes from sibling tools - all essential context for proper tool invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 0% schema description coverage for the single parameter, the description fully compensates by providing comprehensive semantic information: it explains what request_id is ('unique request ID returned by smartcrawler_initiate'), its purpose ('to retrieve the crawling results'), usage guidance ('Keep polling until status is 'completed''), and provides an example ('req_abc123xyz'). This adds significant value beyond the bare schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the specific action ('retrieve results'), resource ('asynchronous SmartCrawler operation'), and distinguishes it from sibling tools by explicitly mentioning it fetches results from a previously initiated operation using request_id. It differentiates from smartcrawler_initiate by focusing on result retrieval rather than initiation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly states when to use this tool ('to retrieve the results from a previously initiated crawling operation using the request_id') and provides clear alternatives by naming the initiation tool ('smartcrawler_initiate'). It also provides specific usage instructions about polling behavior and when to stop ('Keep polling until status is 'completed'').

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

smartcrawler_initiateAInspect

Initiate an asynchronous multi-page web crawling operation with AI extraction or markdown conversion.

This tool starts an intelligent crawler that discovers and processes multiple pages from a starting URL. Choose between AI Extraction Mode (10 credits/page) for structured data or Markdown Mode (2 credits/page) for content conversion. The operation is asynchronous - use smartcrawler_fetch_results to retrieve results. Creates a new crawl request (non-idempotent, non-read-only).

SmartCrawler supports two modes:

  • AI Extraction Mode: Extracts structured data based on your prompt from every crawled page

  • Markdown Conversion Mode: Converts each page to clean markdown format

Args: url (str): The starting URL to begin crawling from. - Must include protocol (http:// or https://) - The crawler will discover and process linked pages from this starting point - Should be a page with links to other pages you want to crawl - Examples: * https://docs.example.com (documentation site root) * https://blog.company.com (blog homepage) * https://example.com/products (product category page) * https://news.site.com/category/tech (news section) - Best practices: * Use homepage or main category pages as starting points * Ensure the starting page has links to content you want to crawl * Consider site structure when choosing the starting URL

prompt (Optional[str]): AI prompt for data extraction.
    - REQUIRED when extraction_mode is 'ai'
    - Ignored when extraction_mode is 'markdown'
    - Describes what data to extract from each crawled page
    - Applied consistently across all discovered pages
    - Examples:
      * "Extract API endpoint name, method, parameters, and description"
      * "Get article title, author, publication date, and summary"
      * "Find product name, price, description, and availability"
      * "Extract job title, company, location, salary, and requirements"
    - Tips for better results:
      * Be specific about fields you want from each page
      * Consider that different pages may have different content structures
      * Use general terms that apply across multiple page types

extraction_mode (str): Extraction mode for processing crawled pages.
    - Default: "ai"
    - Options:
      * "ai": AI-powered structured data extraction (10 credits per page)
        - Uses the prompt to extract specific data from each page
        - Returns structured JSON data
        - More expensive but provides targeted information
        - Best for: Data collection, research, structured analysis
      * "markdown": Simple markdown conversion (2 credits per page)
        - Converts each page to clean markdown format
        - No AI processing, just content conversion
        - More cost-effective for content archival
        - Best for: Documentation backup, content migration, reading
    - Cost comparison:
      * AI mode: 50 pages = 500 credits
      * Markdown mode: 50 pages = 100 credits

depth (Optional[int]): Maximum depth of link traversal from the starting URL.
    - Default: unlimited (will follow links until max_pages or no more links)
    - Depth levels:
      * 0: Only the starting URL (no link following)
      * 1: Starting URL + pages directly linked from it
      * 2: Starting URL + direct links + links from those pages
      * 3+: Continues following links to specified depth
    - Examples:
      * 1: Crawl blog homepage + all blog posts
      * 2: Crawl docs homepage + category pages + individual doc pages
      * 3: Deep crawling for comprehensive site coverage
    - Considerations:
      * Higher depth can lead to exponential page growth
      * Use with max_pages to control scope and cost
      * Consider site structure when setting depth

max_pages (Optional[int]): Maximum number of pages to crawl in total.
    - Default: unlimited (will crawl until no more links or depth limit)
    - Recommended ranges:
      * 10-20: Testing and small sites
      * 50-100: Medium sites and focused crawling
      * 200-500: Large sites and comprehensive analysis
      * 1000+: Enterprise-level crawling (high cost)
    - Cost implications:
      * AI mode: max_pages × 10 credits
      * Markdown mode: max_pages × 2 credits
    - Examples:
      * 10: Quick site sampling (20-100 credits)
      * 50: Standard documentation crawl (100-500 credits)
      * 200: Comprehensive site analysis (400-2000 credits)
    - Note: Crawler stops when this limit is reached, regardless of remaining links

same_domain_only (Optional[bool]): Whether to crawl only within the same domain.
    - Default: true (recommended for most use cases)
    - Options:
      * true: Only crawl pages within the same domain as starting URL
        - Prevents following external links
        - Keeps crawling focused on the target site
        - Reduces risk of crawling unrelated content
        - Example: Starting at docs.example.com only crawls docs.example.com pages
      * false: Allow crawling external domains
        - Follows links to other domains
        - Can lead to very broad crawling scope
        - May crawl unrelated or unwanted content
        - Use with caution and appropriate max_pages limit
    - Recommendations:
      * Use true for focused site crawling
      * Use false only when you specifically need cross-domain data
      * Always set max_pages when using false to prevent runaway crawling

Returns: Dictionary containing: - request_id: Unique identifier for this crawl operation (use with smartcrawler_fetch_results) - status: Initial status of the crawl request ("initiated" or "processing") - estimated_cost: Estimated credit cost based on parameters (actual cost may vary) - crawl_parameters: Summary of the crawling configuration - estimated_time: Rough estimate of processing time - next_steps: Instructions for retrieving results

Raises: ValueError: If URL is malformed, prompt is missing for AI mode, or parameters are invalid HTTPError: If the starting URL cannot be accessed RateLimitError: If too many crawl requests are initiated too quickly

Note: - This operation is asynchronous and may take several minutes to complete - Use smartcrawler_fetch_results with the returned request_id to get results - Keep polling smartcrawler_fetch_results until status is "completed" - Actual pages crawled may be less than max_pages if fewer links are found - Processing time increases with max_pages, depth, and extraction_mode complexity

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
promptNo
extraction_modeNoai
depthNo
max_pagesNo
same_domain_onlyNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description adds significant behavioral context beyond annotations: it explains the asynchronous nature, credit costs per mode (10 vs 2 credits/page), non-idempotent behavior, estimated processing time, and need for polling with fetch_results. Annotations only indicate it's not read-only/idempotent/destructive, so this provides crucial operational details.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured with clear sections (overview, modes, args, returns, raises, notes) but is quite lengthy. While every sentence adds value, it could be more front-loaded; the core functionality is clear early, but parameter details are extensive. Still, no wasted text.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (6 parameters, async operation, cost implications) and 0% schema coverage, the description is exceptionally complete. It covers purpose, usage, all parameters, return values, errors, costs, and next steps. The output schema exists but the description still usefully summarizes returns.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 0% schema description coverage, the description fully compensates by providing detailed semantics for all 6 parameters: url requirements, prompt usage, extraction_mode options with costs, depth levels, max_pages ranges, and same_domain_only implications. Each parameter includes examples, defaults, and practical guidance.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool 'starts an intelligent crawler that discovers and processes multiple pages from a starting URL' and distinguishes it from siblings by specifying it's for 'asynchronous multi-page web crawling' with AI extraction or markdown conversion, unlike simpler scraping tools like 'scrape' or 'smartscraper'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit guidance on when to use this tool vs alternatives: it specifies this is for 'asynchronous multi-page web crawling' and directs users to 'use smartcrawler_fetch_results to retrieve results'. It also contrasts modes (AI vs markdown) with cost implications and best-use cases.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

smartscraperA
Read-onlyIdempotent
Inspect
Extract structured data from a webpage, HTML, or markdown using AI-powered extraction.

This tool uses advanced AI to understand your natural language prompt and extract specific
structured data from web content. Supports three input modes: URL scraping. Ideal for extracting product information, contact details,
article metadata, or any structured content. Costs 10 credits per page. Read-only operation.

Args:
    user_prompt (str): Natural language instructions describing what data to extract.
        - Be specific about the fields you want for better results
        - Use clear, descriptive language about the target data
        - Examples:
          * "Extract product name, price, description, and availability status"
          * "Find all contact methods: email addresses, phone numbers, and social media links"
          * "Get article title, author, publication date, and summary"
          * "Extract all job listings with title, company, location, and salary"
        - Tips for better results:
          * Specify exact field names you want
          * Mention data types (numbers, dates, URLs, etc.)
          * Include context about where data might be located

    website_url (Optional[str]): The complete URL of the webpage to scrape.
        - Mutually exclusive with website_html and website_markdown
        - Must include protocol (http:// or https://)
        - Supports dynamic and static content
        - Examples:
          * https://example.com/products/item
          * https://news.site.com/article/123
          * https://company.com/contact
        - Default: None (must provide one of the three input sources)

    website_html (Optional[str]): Raw HTML content to process locally.
        - Mutually exclusive with website_url and website_markdown
        - Maximum size: 2MB
        - Useful for processing pre-fetched or generated HTML
        - Use when you already have HTML content from another source
        - Example: "<html><body><h1>Title</h1><p>Content</p></body></html>"
        - Default: None

    website_markdown (Optional[str]): Markdown content to process locally.
        - Mutually exclusive with website_url and website_html
        - Maximum size: 2MB
        - Useful for extracting from markdown documents or converted content
        - Works well with documentation, README files, or converted web content
        - Example: "# Title

Section

Content here..." - Default: None

    output_schema (Optional[Union[str, Dict]]): JSON schema defining expected output structure.
        - Can be provided as a dictionary or JSON string
        - Helps ensure consistent, structured output format
        - Optional but recommended for complex extractions
        - IMPORTANT: Must include a "required" field (can be empty array [] if no fields are required)
        - Examples:
          * As dict: {'type': 'object', 'properties': {'title': {'type': 'string'}, 'price': {'type': 'number'}}, 'required': []}
          * As JSON string: '{"type": "object", "properties": {"name": {"type": "string"}}, "required": []}'
          * For arrays: {'type': 'array', 'items': {'type': 'object', 'properties': {...}, 'required': []}, 'required': []}
          * With required fields: {'type': 'object', 'properties': {'name': {'type': 'string'}, 'email': {'type': 'string'}}, 'required': ['name', 'email']}
        - Note: If "required" field is missing, it will be automatically added as an empty array []
        - Default: None (AI will infer structure from prompt)

    number_of_scrolls (Optional[int]): Number of infinite scrolls to perform before scraping.
        - Range: 0-50 scrolls
        - Default: 0 (no scrolling)
        - Useful for dynamically loaded content (lazy loading, infinite scroll)
        - Each scroll waits for content to load before continuing
        - Examples:
          * 0: Static content, no scrolling needed
          * 3: Social media feeds, product listings
          * 10: Long articles, extensive product catalogs
        - Note: Increases processing time proportionally

    total_pages (Optional[int]): Number of pages to process for pagination.
        - Range: 1-100 pages
        - Default: 1 (single page only)
        - Automatically follows pagination links when available
        - Useful for multi-page listings, search results, catalogs
        - Examples:
          * 1: Single page extraction
          * 5: First 5 pages of search results
          * 20: Comprehensive catalog scraping
        - Note: Each page counts toward credit usage (10 credits × pages)

    render_heavy_js (Optional[bool]): Enable heavy JavaScript rendering for dynamic sites.
        - Default: false
        - Set to true for Single Page Applications (SPAs), React apps, Vue.js sites
        - Increases processing time but captures client-side rendered content
        - Use when content is loaded dynamically via JavaScript
        - Examples of when to use:
          * React/Angular/Vue applications
          * Sites with dynamic content loading
          * AJAX-heavy interfaces
          * Content that appears after page load
        - Note: Significantly increases processing time (30-60 seconds vs 5-15 seconds)

    stealth (Optional[bool]): Enable stealth mode to avoid bot detection.
        - Default: false
        - Helps bypass basic anti-scraping measures
        - Uses techniques to appear more like a human browser
        - Useful for sites with bot detection systems
        - Examples of when to use:
          * Sites that block automated requests
          * E-commerce sites with protection
          * Sites that require "human-like" behavior
        - Note: May increase processing time and is not 100% guaranteed

Returns:
    Dictionary containing:
    - extracted_data: The structured data matching your prompt and optional schema
    - metadata: Information about the extraction process
    - credits_used: Number of credits consumed (10 per page processed)
    - processing_time: Time taken for the extraction
    - pages_processed: Number of pages that were analyzed
    - status: Success/error status of the operation

Raises:
    ValueError: If no input source provided or multiple sources provided
    HTTPError: If website_url cannot be accessed
    TimeoutError: If processing exceeds timeout limits
    ValidationError: If output_schema is malformed JSON
ParametersJSON Schema
NameRequiredDescriptionDefault
user_promptYes
website_urlNo
website_htmlNo
website_markdownNo
output_schemaNo
number_of_scrollsNo
total_pagesNo
render_heavy_jsNo
stealthNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.4/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description adds substantial behavioral context beyond what annotations provide. While annotations indicate read-only/idempotent/non-destructive operations, the description adds crucial details: cost (10 credits per page), processing time implications, rate limits (size limits of 2MB for HTML/markdown), authentication needs (none mentioned), and specific behavioral traits like mutual exclusivity of input sources, scroll behavior, pagination handling, and JavaScript rendering options.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely long (over 800 words) with extensive parameter documentation that might be better placed in a separate reference. While well-structured with clear sections, it's not front-loaded - the core purpose gets buried in verbose parameter details. Some sentences could be more concise while maintaining clarity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (9 parameters, AI-powered extraction, multiple input modes) and the presence of an output schema, the description is remarkably complete. It covers all parameters thoroughly, explains the return structure, documents errors/exceptions, provides cost information, and gives practical examples throughout. The output schema existence means the description doesn't need to detail return values, which it appropriately delegates.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 0% schema description coverage for 9 parameters, the description carries the full burden of explaining parameter semantics and does so comprehensively. Each parameter gets detailed explanations with examples, constraints, defaults, and practical usage guidance. The description transforms what would be opaque parameters into well-understood inputs.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose as 'Extract structured data from a webpage, HTML, or markdown using AI-powered extraction' with specific examples of use cases (product info, contact details, etc.). It distinguishes from siblings by mentioning 'AI-powered extraction' and 'structured data', but doesn't explicitly differentiate from all sibling tools like 'scrape' or 'searchscraper'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context about when to use this tool (for AI-powered structured extraction from web content) and includes some usage tips. However, it doesn't explicitly state when NOT to use it or name specific alternatives among the sibling tools, though it implies this is for structured extraction vs. other scraping approaches.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. 8 tool updatesv1.0.0
    • Addedagentic_scrapper
    • Changedmarkdownify6 fields changed
      • removedInput schema / properties / website_url / title
        Removed value: -"Website Url"
      • removedInput schema / title
        Removed value: -"markdownifyArguments"
      • addedOutput schema / additionalProperties
        Added value: +true
      • removedOutput schema / properties
        Removed value: -{
-  "result": {
-    "additionalProperties": true,
-    "title": "Result",
-    "type": "object"
-  }
-}
      • removedOutput schema / required
        Removed value: -[
-  "result"
-]
      • removedOutput schema / title
        Removed value: -"markdownifyOutput"
    • Addedscrape
    • Changedsearchscraper12 fields changed
      • addedInput schema / properties / num_results / anyOf
        Added value: +[
+  {
+    "type": "integer"
+  },
+  {
+    "type": "null"
+  }
+]
      • removedInput schema / properties / num_results / title
        Removed value: -"Num Results"
      • removedInput schema / properties / num_results / type
        Removed value: -"integer"
      • addedInput schema / properties / number_of_scrolls / anyOf
        Added value: +[
+  {
+    "type": "integer"
+  },
+  {
+    "type": "null"
+  }
+]
      • removedInput schema / properties / number_of_scrolls / title
        Removed value: -"Number Of Scrolls"
      • removedInput schema / properties / number_of_scrolls / type
        Removed value: -"integer"
      • removedInput schema / properties / user_prompt / title
        Removed value: -"User Prompt"
      • removedInput schema / title
        Removed value: -"searchscraperArguments"
      • addedOutput schema / additionalProperties
        Added value: +true
      • removedOutput schema / properties
        Removed value: -{
-  "result": {
-    "additionalProperties": true,
-    "title": "Result",
-    "type": "object"
-  }
-}
      • removedOutput schema / required
        Removed value: -[
-  "result"
-]
      • removedOutput schema / title
        Removed value: -"searchscraperOutput"
    • Addedsitemap
    • Changedsmartcrawler_fetch_results6 fields changed
      • removedInput schema / properties / request_id / title
        Removed value: -"Request Id"
      • removedInput schema / title
        Removed value: -"smartcrawler_fetch_resultsArguments"
      • addedOutput schema / additionalProperties
        Added value: +true
      • removedOutput schema / properties
        Removed value: -{
-  "result": {
-    "additionalProperties": true,
-    "title": "Result",
-    "type": "object"
-  }
-}
      • removedOutput schema / required
        Removed value: -[
-  "result"
-]
      • removedOutput schema / title
        Removed value: -"smartcrawler_fetch_resultsOutput"
    • Changedsmartcrawler_initiate19 fields changed
      • addedInput schema / properties / depth / anyOf
        Added value: +[
+  {
+    "type": "integer"
+  },
+  {
+    "type": "null"
+  }
+]
      • removedInput schema / properties / depth / title
        Removed value: -"Depth"
      • removedInput schema / properties / depth / type
        Removed value: -"integer"
      • removedInput schema / properties / extraction_mode / title
        Removed value: -"Extraction Mode"
      • addedInput schema / properties / max_pages / anyOf
        Added value: +[
+  {
+    "type": "integer"
+  },
+  {
+    "type": "null"
+  }
+]
      • removedInput schema / properties / max_pages / title
        Removed value: -"Max Pages"
      • removedInput schema / properties / max_pages / type
        Removed value: -"integer"
      • addedInput schema / properties / prompt / anyOf
        Added value: +[
+  {
+    "type": "string"
+  },
+  {
+    "type": "null"
+  }
+]
      • removedInput schema / properties / prompt / title
        Removed value: -"Prompt"
      • removedInput schema / properties / prompt / type
        Removed value: -"string"
      • addedInput schema / properties / same_domain_only / anyOf
        Added value: +[
+  {
+    "type": "boolean"
+  },
+  {
+    "type": "null"
+  }
+]
      • removedInput schema / properties / same_domain_only / title
        Removed value: -"Same Domain Only"
      • removedInput schema / properties / same_domain_only / type
        Removed value: -"boolean"
      • removedInput schema / properties / url / title
        Removed value: -"Url"
      • removedInput schema / title
        Removed value: -"smartcrawler_initiateArguments"
      • addedOutput schema / additionalProperties
        Added value: +true
      • removedOutput schema / properties
        Removed value: -{
-  "result": {
-    "additionalProperties": true,
-    "title": "Result",
-    "type": "object"
-  }
-}
      • removedOutput schema / required
        Removed value: -[
-  "result"
-]
      • removedOutput schema / title
        Removed value: -"smartcrawler_initiateOutput"
    • Changedsmartscraper21 fields changed
      • removedInput schema / properties / markdown_only
        Removed value: -{
-  "default": null,
-  "title": "Markdown Only",
-  "type": "boolean"
-}
      • addedInput schema / properties / number_of_scrolls / anyOf
        Added value: +[
+  {
+    "type": "integer"
+  },
+  {
+    "type": "null"
+  }
+]
      • removedInput schema / properties / number_of_scrolls / title
        Removed value: -"Number Of Scrolls"
      • removedInput schema / properties / number_of_scrolls / type
        Removed value: -"integer"
      • addedInput schema / properties / output_schema
        Added value: +{
+  "anyOf": [
+    {
+      "anyOf": [
+        {
+          "type": "string"
+        },
+        {
+          "additionalProperties": true,
+          "type": "object"
+        }
+      ],
+      "default": null,
+      "description": "JSON schema dict or JSON string defining the expected output structure",
+      "oneOf": [
+        {
+          "type": "string"
+        },
+        {
+          "type": "object"
+        }
+      ]
+    },
+    {
+      "type": "null"
+    }
+  ],
+  "default": null
+}
      • addedInput schema / properties / render_heavy_js
        Added value: +{
+  "anyOf": [
+    {
+      "type": "boolean"
+    },
+    {
+      "type": "null"
+    }
+  ],
+  "default": null
+}
      • addedInput schema / properties / stealth
        Added value: +{
+  "anyOf": [
+    {
+      "type": "boolean"
+    },
+    {
+      "type": "null"
+    }
+  ],
+  "default": null
+}
      • addedInput schema / properties / total_pages
        Added value: +{
+  "anyOf": [
+    {
+      "type": "integer"
+    },
+    {
+      "type": "null"
+    }
+  ],
+  "default": null
+}
      • removedInput schema / properties / user_prompt / title
        Removed value: -"User Prompt"
      • addedInput schema / properties / website_html
        Added value: +{
+  "anyOf": [
+    {
+      "type": "string"
+    },
+    {
+      "type": "null"
+    }
+  ],
+  "default": null
+}
      • addedInput schema / properties / website_markdown
        Added value: +{
+  "anyOf": [
+    {
+      "type": "string"
+    },
+    {
+      "type": "null"
+    }
+  ],
+  "default": null
+}
      • addedInput schema / properties / website_url / anyOf
        Added value: +[
+  {
+    "type": "string"
+  },
+  {
+    "type": "null"
+  }
+]
      • addedInput schema / properties / website_url / default
        Added value: +null
      • removedInput schema / properties / website_url / title
        Removed value: -"Website Url"
      • removedInput schema / properties / website_url / type
        Removed value: -"string"
      • changedInput schema / required
        Previous value: -[
-  "user_prompt",
-  "website_url"
-]New value: +[
+  "user_prompt"
+]
      • removedInput schema / title
        Removed value: -"smartscraperArguments"
      • addedOutput schema / additionalProperties
        Added value: +true
      • removedOutput schema / properties
        Removed value: -{
-  "result": {
-    "additionalProperties": true,
-    "title": "Result",
-    "type": "object"
-  }
-}
      • removedOutput schema / required
        Removed value: -[
-  "result"
-]
      • removedOutput schema / title
        Removed value: -"smartscraperOutput"
  2. 5 tool updates
    • First observedmarkdownify
    • First observedsearchscraper
    • First observedsmartcrawler_fetch_results
    • First observedsmartcrawler_initiate
    • First observedsmartscraper
Install Server
RayrunBuild, deploy, and operate MCP servers without becoming an infrastructure team.ray.runAd

TDQS

A4.5/5.0
Disambiguation4/5

Most tools have distinct purposes with clear boundaries: agentic_scrapper for multi-step automation, markdownify for content conversion, scrape for raw HTML, searchscraper for web searches, sitemap for site discovery, smartcrawler_initiate/fetch_results for multi-page crawling, and smartscraper for structured extraction. However, smartscraper and agentic_scrapper could be confused as both involve AI extraction, though their scopes differ (single-page vs. multi-step workflows).

Naming Consistency3/5

The naming is mixed with no consistent pattern. Some tools use snake_case (agentic_scrapper, markdownify, scrape, searchscraper, sitemap, smartscraper), while smartcrawler_initiate and smartcrawler_fetch_results use a verb_noun format but with camelCase-like compound words. There's also inconsistency in verb styles: 'scrape' vs. 'scrapper' vs. 'crawler', and 'markdownify' is a unique verb. The set is readable but lacks a unified convention.

Tool Count5/5

With 8 tools, the count is well-scoped for a web scraping server. Each tool serves a specific function in the scraping workflow, from basic fetching (scrape) to complex automation (agentic_scrapper) and multi-page operations (smartcrawler). No tool feels redundant or missing given the domain, making the set appropriately sized for comprehensive scraping tasks.

Completeness5/5

The tool set provides complete coverage for web scraping workflows. It includes basic fetching (scrape, markdownify), structured extraction (smartscraper), site discovery (sitemap), multi-page crawling (smartcrawler), web searches (searchscraper), and advanced automation (agentic_scrapper). There are no obvious gaps; agents can handle everything from simple page retrieval to complex, interactive scraping scenarios with full lifecycle support.

Maintenance

ActivitySlowing
ResponsivenessSyncing

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

  • MCP-Crawl4AI

    wyattowalsh
    A
    license
    Not graded
    quality
    C
    maintenance
    A Model Context Protocol server that enables AI systems to crawl and scrape the live web using Crawl4AI and headless Chromium. It provides tools for structured data extraction, deep site traversal, and session-aware workflows with LLM-optimized outputs like markdown.
    MIT

Appeared in Searches

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/ScrapeGraphAI/scrapegraph-mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server