Skip to main content
Glama
samirsaci

mcp-webscraper

by samirsaci

MCP Web Scraper for Claude Desktop

A Model Context Protocol (MCP) server that enables Claude Desktop to perform advanced web scraping and crawling operations. Extract structured data, analyze website architectures, and discover content relationships - all through natural conversation with Claude.

🎯 Features

  • Static & Dynamic Scraping: Handle both regular HTML and JavaScript-rendered pages

  • Website Crawling: Discover and map entire website structures

  • Data Extraction: Extract specific elements using CSS selectors

  • Batch Operations: Process multiple URLs efficiently

  • Link Analysis: Understand how pages connect and reference each other

Related MCP server: Katana MCP Server

πŸŽ₯ Watch the Tutorial

See the full demo and step-by-step setup guide on YouTube:

Tutorial + Code

πŸ“‹ Prerequisites

  • Python 3.10 or higher

  • WSL2 with Ubuntu (for Windows users)

  • Claude Desktop application

  • uv package manager

πŸš€ Installation

1. Clone the Repository

git clone https://github.com/samirsaci/mcp-webscraper.git
cd mcp-webscraper

2. Install uv Package Manager

If you don't have uv installed:

curl -LsSf https://astral.sh/uv/install.sh | sh

3. Initialize the project

# Initialize the virtual environment
uv init .

4. Install Dependencies

uv add "mcp[cli]"
source .venv/bin/activate
uv pip install -r requirements.txt

Do not forget to install playwright browser to scrape dynamic content

uv run playwright install chromium

5. Test the Installation

Run the test script to verify everything works using a website that loves to be scrapped https://books.toscrape.com/:

uv run python test_local.py

Expected Output:

Static Scraping Success: True
HTML length: 51294
---------
Dynamic Scraping Success: True
HTML length: 51004
---------
Testing Crawler...
Crawler Success: True
Pages crawled: 5
Pages discovered: 437
Failed URLs: 0

First 3 pages discovered:
  1. All products | Books to Scrape - Sandbox
     URL: https://books.toscrape.com/
     Links found: 73
     Depth: 0
  2. All products | Books to Scrape - Sandbox
     URL: https://books.toscrape.com/index.html
     Links found: 73
     Depth: 1
  3. Books |
     Books to Scrape - Sandbox
     URL: https://books.toscrape.com/catalogue/category/books_1/index.html
     Links found: 73
     Depth: 1

Statistics:
  Total unique links: 104
  Max depth reached: 1
  Avg load time: 0.21s

βš™οΈ Claude Desktop Configuration

For Windows Users with WSL

  1. Locate your Claude Desktop configuration file:

File -> Settings -> Edit Config
  1. Add the WebScrappingServer configuration

{
  "mcpServers": {
    "WebScrapingServer": {
      "command": "wsl",
      "args": [
        "-d",
        "Ubuntu",
        "bash",
        "-lc",
        "cd ~/path/to/mcp-webscraper && uv run --with mcp[cli] mcp run scrapping.py"
      ]
    }
  }
}

Important: Replace ~/path/to/mcp-webscraper with the actual path to your project folder in WSL. To find your WSL path:

pwd

3. Restart Claude Desktop

After updating the configuration:

  1. Completely quit Claude Desktop (not just close the window)

  2. Start Claude Desktop again

  3. Look for the πŸ”Œ icon in the text input area

  4. Click it to verify "WebScrapingServer" appears

πŸ”§ Usage Examples

Once configured, you can ask Claude to:

Basic Scraping

"Scrape the homepage of example.com and tell me what you find"

Advanced SEO analysis

Please help me to crawl my personal blog https://yourblog.com with a limit of 150 pages.
I would like to understand how articles are referring to each other.
Can you help me to perform this type of analysis?

πŸ“ Project Structure

mcp-webscraper/
β”œβ”€β”€ models/
β”‚   └── scraping_models.py      # Pydantic models for data validation
β”œβ”€β”€ utils/
β”‚   └── web_scraper.py          # Core WebScraper class
β”œβ”€β”€ scrapping.py                 # MCP server implementation
β”œβ”€β”€ test_local.py                # Local testing script
β”œβ”€β”€ requirements.txt             # Python dependencies
β”œβ”€β”€ README.md                    # This file
└── scraping_server.log          # Server logs (created at runtime)

πŸ› οΈ Available MCP Tools

The server exposes these tools to Claude:

  • scrape_url: Get raw HTML from any webpage

  • extract_data: Extract multiple elements using CSS selectors

  • extract_first: Get a single element from a page

  • batch_scrape: Process multiple URLs

  • crawl_website: Discover and map website structure

πŸ› Troubleshooting

Server not appearing in Claude

*If the server does not appear in Claude, try first to restart Claude Desktop by terminating its processus.`

If this does not work, try to

  1. Check the log file:

cat scraping_server.log
  1. Verify the path in config matches your WSL path:

pwd

The output should match what you have in your config file.

  1. Test the server directly:

uv run python scrapping.py

Playwright issues

If JavaScript scraping fails, try to reinstall the browser

uv run playwright install chromium

WSL-specific issues

Ensure WSL2 is properly installed:

Run this in Windows PowerShell opened as Administrator

wsl --status

πŸ“„ License

MIT License - feel free to use this in your own projects!

About me πŸ€“

Senior Supply Chain and Data Science consultant with international experience working on Logistics and Transportation operations. For consulting or advising on analytics and sustainable supply chain transformation, feel free to contact me via Logigreen Consulting or LinkedIn

Available Tools

5 tools
batch_scrapeB
Scrape multiple URLs efficiently.

Args:
    urls: List of URLs to scrape
    javascript: Set to True if the sites need JavaScript rendering

Returns:
    List of scraping results for each URL
ParametersJSON Schema
NameRequiredDescriptionDefault
urlsYes
javascriptNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

B3.2/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description must disclose behavioral traits. It mentions 'efficiently' but lacks details on concurrency, rate limits, error handling, or return format beyond a vague 'list of scraping results'.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Very concise with clear Args/Returns structure. Every sentence is meaningful without redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Adequate for a simple batch tool, but missing details on output format, error handling, and limits. With an output schema available, it could rely on that, but the description's return statement remains vague.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The description adds meaning to both parameters: urls is a list of URLs, javascript is a boolean defaulting to False with a clear usage note. This compensates for the 0% schema description coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool scrapes multiple URLs efficiently, distinguishing it from single-URL tools like scrape_url and from crawling or extraction tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool over siblings like crawl_website or extract_data. It does not suggest prerequisites or alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

crawl_websiteA
Crawl a website to discover its structure and pages.

Args:
    start_url: Starting URL
    max_pages: Maximum pages to crawl (default 50)
    max_depth: Maximum link depth (default 3)
    same_domain_only: Stay on same domain (default True)

Returns:
    Site map with discovered pages and statistics
ParametersJSON Schema
NameRequiredDescriptionDefault
start_urlYes
max_pagesNo
max_depthNo
same_domain_onlyNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.1/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries full weight. It lists parameters (max_pages, max_depth, same_domain_only) that hint at behavior but omits details on robots.txt compliance, rate limiting, or whether the crawl is polite.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is succinct and front-loaded with the main purpose, followed by a parameter list and return statement. Every sentence adds value with no extraneous content.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description adequately covers the basic crawling behavior and return type ('Site map with discovered pages and statistics'). However, it lacks specifics on output structure and error handling, which would be helpful given the tool's complexity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, but the description provides clear explanations for each parameter, including defaults. This adds significant meaning beyond the bare schema titles.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states 'Crawl a website to discover its structure and pages,' using specific verb and resource. It distinguishes from sibling tools like scrape_url and extract_data, which focus on content extraction from single pages.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for site mapping but lacks explicit when-to-use versus alternative tools. No 'do not use' or 'instead use' guidance is provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

extract_dataA
Scrape a webpage and extract specific data using CSS selectors.

Args:
    url: The webpage to scrape
    css_selectors: List of CSS selectors (e.g., ["h1", "a.link", "#content"])
    attributes: List of attributes to extract for each selector (e.g., ["text", "href", "text"])
               If not provided, defaults to "text" for all selectors
    javascript: Set to True for JavaScript-rendered sites

Returns:
    Dictionary with extracted data for each selector

Example:
    extract_data(
        url="https://example.com",
        css_selectors=["h1", "a"],
        attributes=["text", "href"]
    )
ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
css_selectorsYes
attributesNo
javascriptNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A3.8/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description is the sole source of behavioral info. It reveals that JavaScript rendering can be enabled via parameter, and the return type is a dictionary. However, it does not disclose error handling, rate limiting, origin restrictions, or what happens when selectors find no elements. The example provides some clarity but not comprehensive transparency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a structured docstring with Args, Returns, and Example. It is concise (8 lines) and front-loaded with the purpose. Every sentence adds value, though the example could be slightly shorter. No redundant information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given that the tool has 4 parameters and an output schema, the description explains each parameter and the return structure with an example. It lacks details on limitations (e.g., no mention of authentication, timeout, or error handling), but for a standard scraping tool with an output schema, it covers the main aspects well.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description must explain each parameter. It describes 'url' as the webpage, 'css_selectors' as list of selectors, 'attributes' as list of attributes to extract (defaults to 'text'), and 'javascript' as a toggle for JS-rendered sites. The example further clarifies usage. This fully compensates for the missing schema descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb 'scrape' and 'extract', the resource 'webpage', and the method 'using CSS selectors'. It distinguishes from siblings like batch_scrape (multiple pages) and crawl_website (following links) by focusing on a single-page, targeted extraction.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus siblings such as batch_scrape, crawl_website, extract_first, or scrape_url. There is no mention of trade-offs, prerequisites, or exclusions. A tool with clear sibling alternatives should explicitly differentiate usage scenarios.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

extract_firstA
Extract the first matching element from a webpage.
Useful for getting single values like page title, main heading, etc.

Args:
    url: The webpage to scrape
    css_selector: CSS selector for the element (e.g., "h1", "title", "meta[name='description']")
    attribute: What to extract - "text" for content, or attribute name like "href", "content", "src"
    javascript: Set to True for JavaScript-rendered sites

Returns:
    Dictionary with the extracted value

Example:
    extract_first(url="https://example.com", css_selector="title", attribute="text")
ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
css_selectorYes
attributeNotext
javascriptNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden. It mentions javascript handling for rendered sites and return format, but omits error handling, what happens on no match, or limits.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise yet complete, using bullet-point Args, Returns, and an Example. Every sentence serves a purpose, with no wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity, the description covers all necessary aspects: parameter semantics, return value, and usage context. The example solidifies understanding.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description fully compensates by explaining each parameter, including defaults, examples, and attribute choices. This adds substantial value beyond the bare schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states 'Extract the first matching element from a webpage' with a specific verb and resource. It provides examples and contrasts with batch operations, effectively distinguishing from sibling tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly states it is for single values like page title or heading, giving clear context for when to use. However, it does not explicitly mention when not to use or directly reference sibling alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

scrape_urlA
Scrape a webpage and return its HTML content.

Args:
    url: The webpage URL to scrape
    javascript: Set to True for JavaScript-rendered sites (slower but handles dynamic content)
    wait_seconds: How long to wait for JavaScript to load (only used when javascript=True)

Returns:
    Dictionary with html content, status code, and load time
ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
javascriptNo
wait_secondsNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Despite no annotations, the description discloses key behaviors: the effect of javascript=True (slower but handles dynamic content), the role of wait_seconds (only when javascript=True), and the return dictionary containing html, status code, and load time. It does not cover rate limits or auth, but given the tool's nature, this is sufficient.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise and well-structured with clearly labeled Args and Returns sections. Every sentence provides necessary information without redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity (3 parameters, output schema present) and no annotations, the description completely covers parameter roles, return value, and behavioral notes. No additional context is needed for effective use.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The description adds substantial meaning beyond the input schema: it explains the purpose of each parameter (url as webpage URL, javascript for JS rendering, wait_seconds for JS wait time) and their dependencies. This compensates for the schema's 0% description coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's action ('Scrape a webpage') and outcome ('return its HTML content'), using a specific verb and resource. It is distinct from siblings like batch_scrape (multiple pages) and extract_data (structured extraction).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description does not explicitly indicate when to use this tool versus alternatives like batch_scrape or crawl_website. Usage context is implied by the tool's name and the mention of 'simple' scraping, but no direct guidance is given.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. 5 tool updatesv0.1.0
    • First observedbatch_scrape
    • First observedcrawl_website
    • First observedextract_data
    • First observedextract_first
    • First observedscrape_url

TDQS

A4.1/5.0
Disambiguation5/5

Each tool serves a clearly distinct purpose: raw HTML fetch, structured extraction per URL, extraction of first matching element, batch scraping, and website crawling. No overlap or ambiguity.

Naming Consistency5/5

All tools follow a consistent verb_noun pattern in snake_case (batch_scrape, crawl_website, extract_data, extract_first, scrape_url). The naming is predictable and clear.

Tool Count5/5

With 5 tools, the coverage is focused and each tool adds unique value. The count is ideal for a web scraping server, avoiding bloat or gaps.

Completeness5/5

The tool set covers the core web scraping workflows: fetching raw HTML, extracting with CSS selectors, handling single/multiple URLs, and crawling site structure. No obvious missing operations for typical scraping tasks.

Maintenance

ActivityInactive
ResponsivenessNo issues

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/samirsaci/mcp-webscraper'

If you have feedback or need assistance with the MCP directory API, please join our Discord server