Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@Website Content Crawler MCP Server Crawl https://example.com and give me the page text"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
MCP wrapper that runs the apify/website-content-crawler actor for real via apify-client
and returns its pages as JSON. Every call either returns crawled content or a structured
error — it never reports a fake success.
Runs bill your Apify account at the rate listed for
apify/website-content-crawler — this
server adds nothing on top. No token, no charge: calls return missing_token before any
run starts.
Live output demo (same crawl logic, rendered): https://website-content-crawler.vercel.app
Related MCP server: Crawl4AI MCP Server
npm install
export APIFY_TOKEN=apify_api_... # https://console.apify.com/settings/integrations
npm start # stdio MCP servercrawl_websiteInput:
field | type | default | notes |
| string | required | http/https URL to crawl |
| number | 10 | capped at 50 |
| string |
| or |
Output envelope (same shape every call):
{
"status": "ok",
"run": { "id": "<apify run id>", "status": "SUCCEEDED" },
"page_count": 3,
"total_in_dataset": 3,
"pages": [{ "url": "...", "title": "...", "text": "...(≤5000 chars)" }]
}Error codes: invalid_url, missing_token, apify_auth_failed, actor_run_failed,
run_not_succeeded, dataset_fetch_failed.
printf '%s\n' \
'{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"t","version":"0"}}}' \
'{"jsonrpc":"2.0","method":"notifications/initialized"}' \
'{"jsonrpc":"2.0","id":2,"method":"tools/list"}' \
'{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"crawl_website","arguments":{"url":"https://example.com"}}}' \
| node index.jsWith no APIFY_TOKEN set, request id 3 must return
{"status":"error","error":{"code":"missing_token",...}} — proof the handler reaches the
real Apify boundary instead of inventing a result.
Crawl a website and extract page content by running the apify/website-content-crawler actor. Returns a fixed-shape envelope: {status, run:{id,status}, page_count, pages:[{url,title,text}]}. Requires APIFY_TOKEN in the environment.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to crawl (http/https) | |
| maxPages | No | Maximum pages to crawl, capped at 50 (default 10) | |
| crawlerType | No | cheerio = fast/static and cheaper; playwright:chrome = JS-rendered sites (default cheerio) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the behavioral burden. It reveals that this is an external actor invocation, requires APIFY_TOKEN, and returns a fixed envelope, which gives the agent an honest model of what happens. It does not detail cost/time/side effects, but the core execution model is disclosed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three compact sentences deliver purpose, mechanism, return shape, and a prerequisite without wasted words. The key purpose is front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The hard-coded return envelope compensate for the absent output schema, and the auth requirement is noted. It doesn't cover failure modes or performance expectations, but the essentials for calling and understanding the result are present.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the schema already documents all three parameters. The description adds no parameter-level meaning, but the baseline 3 is appropriate because it does not need to compensate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a precise action ('Crawl a website and extract page content') plus the underlying mechanism (running the apify/website-content-crawler actor), so its purpose is clear even without siblings. It also states the return shape to help the tool be disambiguated.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description conveys when the tool is appropriate: when a website needs to be crawled and page text extracted. It names the required environment token and return envelope, but does not explicitly state when not to use or what alternatives might apply.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
v1.0.0crawl_websiteOnly one tool is exposed, so there is no chance of confusing it with another. Its purpose is clear and unambiguous.
The single tool name 'crawl_website' uses a consistent verb_noun structure. With no other tools, there are no conflicting naming conventions.
A one-tool server is borderline. The tool covers the core need, but the server feels thin and could reasonably expose additional operations like crawl status or history.
The server offers a direct crawl-and-extract action, but lacks supporting operations such as configurable crawling options, job inspection, or the ability to cancel or list previous runs. The surface is functional but minimal.
Unclaimed servers have limited discoverability.
If you are the server author, to access and configure the admin panel.
Turn any website into structured JSON data matching your custom schema.
Crawl, scrape, search the web, and automate browsers at scale with anti-bot bypass.
Scrape, crawl, map and extract the web. Pay per call in USDC, no account or API key.
Extract and parse web pages into clean HTML, links, or Markdown. Handle dynamic, complex, or block…
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/dev55acc-ai/website-content-crawler-mcp'
If you have feedback or need assistance with the MCP directory API, please join our Discord server
