mcp-webscraper
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@mcp-webscraperScrape books.toscrape.com for all book titles and prices"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
MCP Web Scraper for Claude Desktop
A Model Context Protocol (MCP) server that enables Claude Desktop to perform advanced web scraping and crawling operations. Extract structured data, analyze website architectures, and discover content relationships - all through natural conversation with Claude.
π― Features
Static & Dynamic Scraping: Handle both regular HTML and JavaScript-rendered pages
Website Crawling: Discover and map entire website structures
Data Extraction: Extract specific elements using CSS selectors
Batch Operations: Process multiple URLs efficiently
Link Analysis: Understand how pages connect and reference each other
Related MCP server: Katana MCP Server
π₯ Watch the Tutorial
See the full demo and step-by-step setup guide on YouTube:

π Prerequisites
Python 3.10 or higher
WSL2 with Ubuntu (for Windows users)
Claude Desktop application
uvpackage manager
π Installation
1. Clone the Repository
git clone https://github.com/samirsaci/mcp-webscraper.git
cd mcp-webscraper2. Install uv Package Manager
If you don't have uv installed:
curl -LsSf https://astral.sh/uv/install.sh | sh3. Initialize the project
# Initialize the virtual environment
uv init .4. Install Dependencies
uv add "mcp[cli]"
source .venv/bin/activate
uv pip install -r requirements.txtDo not forget to install playwright browser to scrape dynamic content
uv run playwright install chromium5. Test the Installation
Run the test script to verify everything works using a website that loves to be scrapped https://books.toscrape.com/:
uv run python test_local.pyExpected Output:
Static Scraping Success: True
HTML length: 51294
---------
Dynamic Scraping Success: True
HTML length: 51004
---------
Testing Crawler...
Crawler Success: True
Pages crawled: 5
Pages discovered: 437
Failed URLs: 0
First 3 pages discovered:
1. All products | Books to Scrape - Sandbox
URL: https://books.toscrape.com/
Links found: 73
Depth: 0
2. All products | Books to Scrape - Sandbox
URL: https://books.toscrape.com/index.html
Links found: 73
Depth: 1
3. Books |
Books to Scrape - Sandbox
URL: https://books.toscrape.com/catalogue/category/books_1/index.html
Links found: 73
Depth: 1
Statistics:
Total unique links: 104
Max depth reached: 1
Avg load time: 0.21sβοΈ Claude Desktop Configuration
For Windows Users with WSL
Locate your Claude Desktop configuration file:
File -> Settings -> Edit ConfigAdd the WebScrappingServer configuration
{
"mcpServers": {
"WebScrapingServer": {
"command": "wsl",
"args": [
"-d",
"Ubuntu",
"bash",
"-lc",
"cd ~/path/to/mcp-webscraper && uv run --with mcp[cli] mcp run scrapping.py"
]
}
}
}Important: Replace ~/path/to/mcp-webscraper with the actual path to your project folder in WSL. To find your WSL path:
pwd3. Restart Claude Desktop
After updating the configuration:
Completely quit Claude Desktop (not just close the window)
Start Claude Desktop again
Look for the π icon in the text input area
Click it to verify "WebScrapingServer" appears
π§ Usage Examples
Once configured, you can ask Claude to:
Basic Scraping
"Scrape the homepage of example.com and tell me what you find"Advanced SEO analysis
Please help me to crawl my personal blog https://yourblog.com with a limit of 150 pages.
I would like to understand how articles are referring to each other.
Can you help me to perform this type of analysis?π Project Structure
mcp-webscraper/
βββ models/
β βββ scraping_models.py # Pydantic models for data validation
βββ utils/
β βββ web_scraper.py # Core WebScraper class
βββ scrapping.py # MCP server implementation
βββ test_local.py # Local testing script
βββ requirements.txt # Python dependencies
βββ README.md # This file
βββ scraping_server.log # Server logs (created at runtime)π οΈ Available MCP Tools
The server exposes these tools to Claude:
scrape_url: Get raw HTML from any webpageextract_data: Extract multiple elements using CSS selectorsextract_first: Get a single element from a pagebatch_scrape: Process multiple URLscrawl_website: Discover and map website structure
π Troubleshooting
Server not appearing in Claude
*If the server does not appear in Claude, try first to restart Claude Desktop by terminating its processus.`
If this does not work, try to
Check the log file:
cat scraping_server.logVerify the path in config matches your WSL path:
pwdThe output should match what you have in your config file.
Test the server directly:
uv run python scrapping.pyPlaywright issues
If JavaScript scraping fails, try to reinstall the browser
uv run playwright install chromiumWSL-specific issues
Ensure WSL2 is properly installed:
Run this in Windows PowerShell opened as Administrator
wsl --statusπ License
MIT License - feel free to use this in your own projects!
About me π€
Senior Supply Chain and Data Science consultant with international experience working on Logistics and Transportation operations. For consulting or advising on analytics and sustainable supply chain transformation, feel free to contact me via Logigreen Consulting or LinkedIn
Available Tools
5 toolsbatch_scrapeB
Scrape multiple URLs efficiently.
Args:
urls: List of URLs to scrape
javascript: Set to True if the sites need JavaScript rendering
Returns:
List of scraping results for each URL
| Name | Required | Description | Default |
|---|---|---|---|
| urls | Yes | ||
| javascript | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must disclose behavioral traits. It mentions 'efficiently' but lacks details on concurrency, rate limits, error handling, or return format beyond a vague 'list of scraping results'.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Very concise with clear Args/Returns structure. Every sentence is meaningful without redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Adequate for a simple batch tool, but missing details on output format, error handling, and limits. With an output schema available, it could rely on that, but the description's return statement remains vague.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The description adds meaning to both parameters: urls is a list of URLs, javascript is a boolean defaulting to False with a clear usage note. This compensates for the 0% schema description coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool scrapes multiple URLs efficiently, distinguishing it from single-URL tools like scrape_url and from crawling or extraction tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool over siblings like crawl_website or extract_data. It does not suggest prerequisites or alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crawl_websiteA
Crawl a website to discover its structure and pages.
Args:
start_url: Starting URL
max_pages: Maximum pages to crawl (default 50)
max_depth: Maximum link depth (default 3)
same_domain_only: Stay on same domain (default True)
Returns:
Site map with discovered pages and statistics
| Name | Required | Description | Default |
|---|---|---|---|
| start_url | Yes | ||
| max_pages | No | ||
| max_depth | No | ||
| same_domain_only | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full weight. It lists parameters (max_pages, max_depth, same_domain_only) that hint at behavior but omits details on robots.txt compliance, rate limiting, or whether the crawl is polite.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is succinct and front-loaded with the main purpose, followed by a parameter list and return statement. Every sentence adds value with no extraneous content.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description adequately covers the basic crawling behavior and return type ('Site map with discovered pages and statistics'). However, it lacks specifics on output structure and error handling, which would be helpful given the tool's complexity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, but the description provides clear explanations for each parameter, including defaults. This adds significant meaning beyond the bare schema titles.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Crawl a website to discover its structure and pages,' using specific verb and resource. It distinguishes from sibling tools like scrape_url and extract_data, which focus on content extraction from single pages.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for site mapping but lacks explicit when-to-use versus alternative tools. No 'do not use' or 'instead use' guidance is provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extract_dataA
Scrape a webpage and extract specific data using CSS selectors.
Args:
url: The webpage to scrape
css_selectors: List of CSS selectors (e.g., ["h1", "a.link", "#content"])
attributes: List of attributes to extract for each selector (e.g., ["text", "href", "text"])
If not provided, defaults to "text" for all selectors
javascript: Set to True for JavaScript-rendered sites
Returns:
Dictionary with extracted data for each selector
Example:
extract_data(
url="https://example.com",
css_selectors=["h1", "a"],
attributes=["text", "href"]
)
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| css_selectors | Yes | ||
| attributes | No | ||
| javascript | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description is the sole source of behavioral info. It reveals that JavaScript rendering can be enabled via parameter, and the return type is a dictionary. However, it does not disclose error handling, rate limiting, origin restrictions, or what happens when selectors find no elements. The example provides some clarity but not comprehensive transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a structured docstring with Args, Returns, and Example. It is concise (8 lines) and front-loaded with the purpose. Every sentence adds value, though the example could be slightly shorter. No redundant information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given that the tool has 4 parameters and an output schema, the description explains each parameter and the return structure with an example. It lacks details on limitations (e.g., no mention of authentication, timeout, or error handling), but for a standard scraping tool with an output schema, it covers the main aspects well.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must explain each parameter. It describes 'url' as the webpage, 'css_selectors' as list of selectors, 'attributes' as list of attributes to extract (defaults to 'text'), and 'javascript' as a toggle for JS-rendered sites. The example further clarifies usage. This fully compensates for the missing schema descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'scrape' and 'extract', the resource 'webpage', and the method 'using CSS selectors'. It distinguishes from siblings like batch_scrape (multiple pages) and crawl_website (following links) by focusing on a single-page, targeted extraction.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus siblings such as batch_scrape, crawl_website, extract_first, or scrape_url. There is no mention of trade-offs, prerequisites, or exclusions. A tool with clear sibling alternatives should explicitly differentiate usage scenarios.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extract_firstA
Extract the first matching element from a webpage.
Useful for getting single values like page title, main heading, etc.
Args:
url: The webpage to scrape
css_selector: CSS selector for the element (e.g., "h1", "title", "meta[name='description']")
attribute: What to extract - "text" for content, or attribute name like "href", "content", "src"
javascript: Set to True for JavaScript-rendered sites
Returns:
Dictionary with the extracted value
Example:
extract_first(url="https://example.com", css_selector="title", attribute="text")
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| css_selector | Yes | ||
| attribute | No | text | |
| javascript | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It mentions javascript handling for rendered sites and return format, but omits error handling, what happens on no match, or limits.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise yet complete, using bullet-point Args, Returns, and an Example. Every sentence serves a purpose, with no wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity, the description covers all necessary aspects: parameter semantics, return value, and usage context. The example solidifies understanding.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description fully compensates by explaining each parameter, including defaults, examples, and attribute choices. This adds substantial value beyond the bare schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Extract the first matching element from a webpage' with a specific verb and resource. It provides examples and contrasts with batch operations, effectively distinguishing from sibling tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states it is for single values like page title or heading, giving clear context for when to use. However, it does not explicitly mention when not to use or directly reference sibling alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scrape_urlA
Scrape a webpage and return its HTML content.
Args:
url: The webpage URL to scrape
javascript: Set to True for JavaScript-rendered sites (slower but handles dynamic content)
wait_seconds: How long to wait for JavaScript to load (only used when javascript=True)
Returns:
Dictionary with html content, status code, and load time
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| javascript | No | ||
| wait_seconds | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Despite no annotations, the description discloses key behaviors: the effect of javascript=True (slower but handles dynamic content), the role of wait_seconds (only when javascript=True), and the return dictionary containing html, status code, and load time. It does not cover rate limits or auth, but given the tool's nature, this is sufficient.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise and well-structured with clearly labeled Args and Returns sections. Every sentence provides necessary information without redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (3 parameters, output schema present) and no annotations, the description completely covers parameter roles, return value, and behavioral notes. No additional context is needed for effective use.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The description adds substantial meaning beyond the input schema: it explains the purpose of each parameter (url as webpage URL, javascript for JS rendering, wait_seconds for JS wait time) and their dependencies. This compensates for the schema's 0% description coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's action ('Scrape a webpage') and outcome ('return its HTML content'), using a specific verb and resource. It is distinct from siblings like batch_scrape (multiple pages) and extract_data (structured extraction).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description does not explicitly indicate when to use this tool versus alternatives like batch_scrape or crawl_website. Usage context is implied by the tool's name and the mention of 'simple' scraping, but no direct guidance is given.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
5 tool updates
v0.1.0- First observed
batch_scrape - First observed
crawl_website - First observed
extract_data - First observed
extract_first - First observed
scrape_url
TDQS
Each tool serves a clearly distinct purpose: raw HTML fetch, structured extraction per URL, extraction of first matching element, batch scraping, and website crawling. No overlap or ambiguity.
All tools follow a consistent verb_noun pattern in snake_case (batch_scrape, crawl_website, extract_data, extract_first, scrape_url). The naming is predictable and clear.
With 5 tools, the coverage is focused and each tool adds unique value. The count is ideal for a web scraping server, avoiding bloat or gaps.
The tool set covers the core web scraping workflows: fetching raw HTML, extracting with CSS selectors, handling single/multiple URLs, and crawling site structure. No obvious missing operations for typical scraping tasks.
Maintenance
Resources
Unclaimed servers have limited discoverability.
Looking for Admin?
If you are the server author, to access and configure the admin panel.
Related MCP Connectors
Enable language models to perform advanced AI-powered web scraping with enterprise-grade reliabiliβ¦
Live SEO workflow tools for Claude Code, Codex, and AI agents.
Live web access for agents: scrape, SERP search, crawl/map, 74 collectors, datasets, proxies.
- mcpOAuthcom.sequentum
Turn the web into structured, reliable, actionable enterprise data for AI Agents
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceEnables web scraping, crawling, structured data extraction, and browser automation through multiple AI agents including OpenAI's CUA, Anthropic's Claude Computer Use, and Browser Use.17MIT
- AlicenseNot gradedqualityDmaintenanceIntegrates ProjectDiscovery's Katana web crawler with Claude Desktop, enabling users to crawl websites, discover endpoints and hidden resources, extract JavaScript files, and perform reconnaissance with customizable depth, scope, and filtering options.MIT
- FlicenseNot gradedqualityDmaintenanceIntegrates WebcrawlerAPI's web scraping and crawling capabilities into MCP-compatible applications like Claude Code, allowing content extraction from any website.102-

spidra-mcp-serverofficial
AlicenseAqualityBmaintenanceEnables AI assistants to scrape pages, batch-process URLs, and crawl entire websites with AI-powered extraction.1237MIT
Appeared in Searches
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/samirsaci/mcp-webscraper'
If you have feedback or need assistance with the MCP directory API, please join our Discord server