common-crawl-mcp
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@common-crawl-mcpShow the crawl presence and dynamics for example.com"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
common-crawl-mcp
MCP-сервер и CLI для запросов к Common Crawl — открытому корпусу веба, на котором учат языковые модели.
Отвечает на вопрос «виден ли сайт для ИИ»: сколько его страниц попало в корпус, какие разделы, как это менялось, и что видит модель, читая конкретную страницу.
Работает через AWS Athena поверх колоночного индекса (cc-index), а не через публичный
API Common Crawl — тот регулярно отдаёт 504. Запрос по домену стоит около $0.00005.
Установка
uv tool install git+https://github.com/nikolaymokh-dev/common-crawl-mcpНужны AWS-креды с доступом к Athena, Glue и чтением бакета commoncrawl. Берутся
из ~/.aws обычным способом (AWS_PROFILE, AWS_REGION; по умолчанию us-east-1).
Related MCP server: mcp-server-wayback
CLI
ccq example.com # страниц в свежем кравле
ccq example.com --dynamics # динамика по 18 последним кравлам
ccq example.com --locales --sections # разбивка по языкам и разделам
ccq example.com --compare example.org,example.net
ccq example.com --errors # 404/301/5xx, которые получил краулер
ccq example.com --robots # robots.txt глазами краулера
ccq --discover de --min-pages 100 # домены зоны .de
ccq --zone-suffixes uk # co.uk / org.uk / gov.uk ...
ccq --list-crawlsЛюбую команду можно получить в JSON: --json.
MCP
claude mcp add common-crawl -- uvx --from git+https://github.com/nikolaymokh-dev/common-crawl-mcp common-crawl-mcpТулы
Тул | Зачем |
| Сколько страниц домена в кравле |
| Динамика по кравлам — основа для любых выводов |
| Несколько доменов одним сканом |
| Разбивка по языковым префиксам пути |
| Топ разделов: где охват проседает |
| Конкретные URL + координаты в архиве |
| Страницы-дубли (одинаковый |
| Не-200 ответы у домена |
|
|
| Домены по зоне и/или подстроке |
| Суффиксы внутри зоны |
| Текст страницы из архива — то, что читает модель |
| Список доступных кравлов |
Что стоит знать
Ноль страниц ≠ сайта нет. Охват Common Crawl заметно скачет между кравлами: сайт,
стабильно дающий сотни страниц, может провалиться до единиц. Любой вывод делается по
cc_dynamics, а не по одному снимку.
Поиск по зоне почти бесплатен. Индекс отсортирован по url_surtkey — имени хоста
задом наперёд, — поэтому домены одной зоны лежат в файлах подряд, и Athena пропускает
всё остальное. Замерено на CC-MAIN-2026-30: .uz — 0.28 MB скана на 7 954 домена,
.de — 23 MB на 2 075 692 домена.
Поиск по подстроке — нет. host_contains не может воспользоваться этой сортировкой,
поэтому без указания зоны он читает колонку хостов по всему кравлу. Тул это запрещает,
пока не передашь allow_full_scan=True. Заодно зона отсекает ложные совпадения: короткая
подстрока по всему вебу цепляет случайные домены из чужих зон, внутри одной зоны — нет.
Составные зоны. tld='uk' захватывает и example.uk, и example.co.uk. Нужен
конкретный сегмент — фильтруй registry_suffix='co.uk'. Посмотреть, что вообще есть
в зоне, — cc_zone_suffixes.
Стоимость возвращается вместе с данными. В каждом ответе есть scanned_mb и
cost_usd. Athena billing округляет запрос вверх до 10 MB.
Разработка
uv sync --extra dev
uv run pytest # юнит-тесты, сети не требуютСтруктура: athena.py — транспорт, crawls.py — какие кравлы есть, queries.py —
построители SQL (чистые функции, тестируются без AWS), api.py — операции,
warc.py — извлечение текста, cli.py и server.py — две обёртки над api.
Лицензия
MIT
Available Tools
13 toolscc_compareA
Сравнить несколько доменов между собой по кравлам (матрица домен × кравл).
Один скан на все домены — дешевле, чем вызывать cc_dynamics по очереди.
| Name | Required | Description | Default |
|---|---|---|---|
| last_n | No | ||
| domains | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the burden. It discloses the output as a domain×crawl matrix and mentions a single scan, which implies an efficient batch operation. However, it does not describe any side effects, permissions, or data freshness, leaving some behavioral aspects uncovered.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description consists of two concise sentences. The first states the purpose and output structure, the second explains the cost advantage. Every word contributes value and information is front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has an output schema, which covers return value structure, but lacks annotations and has minimal parameter explanation. The description is clear about purpose and efficiency but omits details like how last_n works, what constitutes a 'scan', and prerequisites. It is adequate for a simple comparison tool but not complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 0% description coverage, and the description does not explain the meaning of 'domains' or 'last_n'. It only alludes to 'several domains' but provides no semantic guidance for parameters, so it fails to compensate for the schema's lack of descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb 'Сравнить' (compare) with a clear resource ('несколько доменов... по кравлам') and explicitly describes the output as a 'матрица домен × кравл'. It distinguishes itself from siblings by focusing on multi-domain comparison and even references the alternative cc_dynamics.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It provides clear context: 'Один скан на все домены — дешевле, чем вызывать cc_dynamics по очереди' (one scan for all domains is cheaper than calling cc_dynamics one by one). This gives an explicit alternative and a reason to prefer this tool, though it doesn't explicitly state when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
cc_crawlsA
Список доступных кравлов Common Crawl (свежий последним).
Полезно, чтобы узнать актуальное имя кравла перед остальными запросами.
| Name | Required | Description | Default |
|---|---|---|---|
| refresh | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must carry the behavioral transparency burden. It does not disclose whether the request is read-only, whether results are cached, or what the 'refresh' parameter does. This is a significant gap for a tool with an optional refresh parameter.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two short sentences, front-loaded with the main purpose. No wasted words, and it efficiently conveys the key point about ordering and usage.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple list tool with an output schema, the description is adequate but has clear gaps: it omits the refresh parameter's behavior and any note about read-only semantics. It lacks details that would help an agent fully anticipate the tool's behavior, even though the output schema covers return values.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has one boolean parameter 'refresh' with 0% description coverage. The description does not mention this parameter at all, failing to explain its purpose or effect. The parameter name hints at refreshing but is not explicit, so the description adds no value beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool lists available Common Crawl crawls, with the newest last. This distinguishes it from sibling tools that focus on specific aspects like presence or dynamics.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly says this is useful to get the current crawl name before making other requests, providing clear context for when to use it. It doesn't mention alternatives or exclusions, but the primary use case is well stated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
cc_discoverB
Найти домены по зоне и/или подстроке в имени хоста.
tld — любая зона: 'uz', 'de', 'uk'. Стоит копейки: индекс отсортирован по хосту задом наперёд, поэтому зона читается почти без скана. registry_suffix — точнее для составных зон: 'co.uk', 'com.au'. host_contains — подстрока; требует зоны, иначе это скан всего кравла.
| Name | Required | Description | Default |
|---|---|---|---|
| tld | No | ||
| crawl | No | ||
| limit | No | ||
| min_pages | No | ||
| host_contains | No | ||
| allow_full_scan | No | ||
| registry_suffix | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so the description carries the burden. It discloses performance characteristics (index sorted reversed, cheap zone lookup) and warns that host_contains without a zone triggers a full crawl scan. However, it omits details on limit/min_pages/allow_full_scan behavior and lacks a clear statement about return format.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is brief, with a clear opening statement followed by concise parameter notes. Every sentence adds information about usage or performance; no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 7-parameter tool with no annotations, the description covers only the search criteria and one performance warning. It omits the meaning of crawl, limit, min_pages, and allow_full_scan, making it incomplete. An output schema exists but is not provided to the agent in this context, so those gaps remain.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate. It explains tld, registry_suffix, and host_contains with examples and constraints, covering 3 of 7 parameters. The remaining parameters (crawl, limit, min_pages, allow_full_scan) are left undocumented.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states 'Find domains by zone and/or substring in host name' — a clear verb+resource+criteria. It does not explicitly differentiate from sibling tools, but the focus on zone/substring discovery is distinct enough.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use cc_discover vs sibling tools. It provides parameter-level usage advice (compound zones for registry_suffix, host_contains requires a zone) but does not address tool selection or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
cc_duplicatesC
Группы страниц с идентичным содержимым (совпадает content_digest).
Дублирующийся контент под разными URL — типовая SEO-проблема.
| Name | Required | Description | Default |
|---|---|---|---|
| crawl | No | ||
| limit | No | ||
| domain | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden. It reveals that grouping is based on content_digest, but does not disclose output format, ordering, pagination, whether empty groups are included, or meaning of the 'crawl' parameter. For a read-only analysis tool, this lacks sufficient behavioral detail beyond the basic grouping criterion.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loaded with the core function ('groups pages with identical content') and then adds context about SEO relevance. Every sentence earns its place; there is no unnecessary fluff or redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite having an output schema, the description does not explain the meaning of the required 'domain' parameter, the optional 'crawl' parameter, or the 'limit' parameter. Without these, a tool user cannot confidently invoke it correctly. The contextual gap is significant for a tool with three parameters and zero parameter descriptions.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, and the description does not explain any of the three parameters (domain, crawl, limit). It fails to mention that domain is required or that limit defaults to 25, and the 'crawl' parameter is entirely unaddressed, leaving agents without enough information to set parameters correctly.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it groups pages with identical content (matching content_digest) and frames this as a duplicate-content SEO issue. This is a specific verb+resource description that distinguishes it from most sibling tools, though it does not explicitly name an alternative like 'use cc_compare for...'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for detecting duplicate content in SEO contexts ('типовая SEO-проблема'), but it does not explicitly state when to prefer this tool over siblings or provide exclusions. Usage guidance is present only by implication.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
cc_dynamicsA
Динамика индексации домена по последним N кравлам.
Главный тул для выводов: один кравл шумит, тренд — нет. last_n=0 — все доступные кравлы (дороже).
| Name | Required | Description | Default |
|---|---|---|---|
| domain | Yes | ||
| last_n | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the disclosure burden. It adds useful behavioral insight: one crawl is noisy, trend is not, and last_n=0 fetches all available crawls at higher cost. It does not describe output details, but an output schema exists, which reduces the need for that.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise: two lines plus one usage note. Every sentence adds value—purpose, trend guidance, and special parameter meaning—with no filler or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a two-parameter tool with an output schema, the description covers purpose, usage guidance, parameter semantics, and cost behavior. It is fully self-contained enough for an agent to decide when and how to invoke it.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It explains last_n as the number of recent crawls and explicitly documents last_n=0 behavior and cost. The domain parameter is self-explanatory from the name, so the partial coverage is adequate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly identifies the tool as showing domain indexing dynamics over the last N crawls, and adds that it is the main tool for drawing conclusions (trend vs. single-crawl noise). This differentiates it from siblings like cc_crawls, though it does not explicitly name an alternative, so it falls just short of a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It provides clear usage context: use this tool to observe trends because a single crawl is noisy. It implies a distinction from sibling tools and mentions the cost of last_n=0, but does not explicitly state when not to use it or name alternatives, so it does not meet the full 'when/when-not/alternatives' criteria.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
cc_errorsC
Не-200 ответы, которые краулер получил от домена: 404, редиректы, 5xx.
| Name | Required | Description | Default |
|---|---|---|---|
| crawl | No | ||
| domain | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It states the output type but does not mention any limitations, such as pagination, time range, or whether redirects are followed. The examples are useful, but significant behavioral details are missing.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single concise sentence that front-loads the core concept ('Non-200 responses') and adds concrete examples. It avoids unnecessary words, though it omits useful context; but for conciseness and structure, it is efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has two parameters (one optional), an output schema, and a set of sibling tools, the description is under-specified. It does cover the primary purpose, but fails to explain the optional 'crawl' parameter or provide any usage guidance, leaving the agent with gaps in how to correctly invoke the tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The description provides zero explanation of the parameters. The 'domain' parameter is inferable from the description, but the optional 'crawl' parameter is completely unexplained. With schema description coverage at 0%, the description fails to compensate for the lack of parameter semantics.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly identifies the resource (non-200 HTTP responses) and scope (from a domain), with specific examples (404, redirects, 5xx). It distinguishes from sibling tools like cc_urls by focusing on errors rather than all URLs, though it lacks an explicit verb like 'list' or 'get'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The intended use is implied: to retrieve error responses for a given domain. However, there is no explicit guidance on when to use this tool over alternatives, nor any exclusion criteria or context about prerequisites. This is barely above no guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
cc_fetch_textA
Прочитать текст страницы из WARC-архива по координатам из cc_urls или cc_robots.
Показывает страницу так, как её видит модель при обучении. Athena не задействована, запрос бесплатный.
| Name | Required | Description | Default |
|---|---|---|---|
| length | Yes | ||
| offset | Yes | ||
| max_chars | No | ||
| warc_filename | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
There are no annotations, so the description must carry the burden. It adds useful behavioral context: shows the page as the model sees it during training, and states that Athena is not used and the request is free. However, it does not disclose potential side effects, authentication requirements, or what happens with invalid coordinates. This is moderate transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loaded with the primary action, and contains no fluff. It efficiently conveys the core purpose and a few key traits.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is simple and has an output schema, so return values are covered. The description covers purpose, source of coordinates, and cost, but lacks per-parameter explanation and any mention of error cases or limitations beyond the default max_chars. It is adequate but not fully complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It only collectively refers to parameters as 'coordinates from cc_urls or cc_robots' without explaining warc_filename, offset, length, or max_chars individually. This provides minimal semantic meaning and does not adequately compensate for the lack of schema descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource: 'Прочитать текст страницы из WARC-архива' (read page text from WARC archive), and clearly distinguishes itself from siblings like cc_urls and cc_robots by indicating it consumes their output. It also adds a unique perspective: 'Shows the page as the model sees it during training.'
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear usage context: it is meant for fetching text using coordinates from cc_urls or cc_robots, and notes that it does not use Athena and is free. It does not explicitly state exclusions or name alternative tools, but the prerequisite and cost/backend context are helpful.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
cc_localesC
Разбивка страниц домена по языковому префиксу пути (/ru, /en, /uz ...).
| Name | Required | Description | Default |
|---|---|---|---|
| crawl | No | ||
| domain | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
There are no annotations, so the description carries the full behavioral transparency burden. It does not state whether the operation is read-only, whether the crawl parameter affects execution, or any limitations, rate limits, or side effects. Only the grouping behavior is disclosed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, front-loaded sentence with no filler. Every word contributes to the core meaning, and the concrete examples make it easy to parse.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Although an output schema exists, the input side is incomplete: the `crawl` parameter is undefined, there are no usage guidelines, and there is no behavioral/safety disclosure. The description clearly communicates the core purpose but is not sufficient for reliable tool invocation in all cases.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The phrase 'домена' gives some meaning to the `domain` parameter, but the optional `crawl` parameter is entirely unexplained. With 0% schema description coverage, the description fails to compensate for the missing parameter documentation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool produces a breakdown of a domain's pages by path language prefix (/ru, /en, /uz), which is specific and distinguishes it from sibling tools like cc_zone_suffixes or cc_sections. However, it uses the noun 'разбивка' rather than an explicit verb such as 'returns' or 'lists', so it stops just short of a full 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No when-to-use guidance is provided. The description does not compare cc_locales with sibling tools, mention prerequisites, or state when it should be preferred over alternatives like cc_presence or cc_urls.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
cc_presenceA
Сколько страниц домена попало в кравл — есть ли сайт в данных, на которых учат LLM.
crawl не указан — берётся свежий. Ноль страниц не означает отсутствие сайта: охват сильно скачет, проверяй через cc_dynamics.
| Name | Required | Description | Default |
|---|---|---|---|
| crawl | No | ||
| domain | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It discloses the default crawl selection behavior and, importantly, the interpretive caveat that zero pages does not imply site absence. However, it does not describe the output format or potential edge cases (e.g., invalid domains), but the presence of an output schema mitigates the need for return details. Overall, it provides meaningful behavioral context beyond the schema.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise: two sentences, each earning its place. The first states the core function and purpose, the second provides essential caveats and a pointer to an alternative tool. No redundant information, ideal structure for an AI agent to quickly grasp.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (2 parameters, one optional), the presence of an output schema, and the no-annotation context, the description covers all essential aspects: what it does, default parameter behavior, and a critical interpretive caveat. It is complete for an agent to invoke correctly without additional documentation. The pointer to cc_dynamics also covers follow-up scenarios.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has 0% description coverage, so the description must compensate. It explicitly explains the 'crawl' parameter's default behavior ('crawl не указан — берётся свежий'), adding value beyond the schema's bare type/null default. The 'domain' parameter is not elaborated, but its purpose is implicitly clear from the tool name and description. This partial compensation justifies a score above baseline.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's function: 'Сколько страниц домена попало в кравл' (how many pages of a domain are in the crawl), with a specific verb and resource. It also frames the purpose as checking site presence in LLM training data, which distinguishes it from siblings like cc_urls or cc_sections. The reference to cc_dynamics for further checks further differentiates its role.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit usage guidance: it explains the default behavior when the 'crawl' parameter is omitted ('crawl не указан — берётся свежий') and gives a clear alternative: if the result is zero, it advises checking via cc_dynamics because coverage fluctuates. This tells the user when to use this tool and when to switch to another, which is exactly what this dimension requires.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
cc_robotsB
robots.txt домена, каким его увидел краулер в момент обхода.
Объясняет, почему сайт мог выпасть из индекса.
| Name | Required | Description | Default |
|---|---|---|---|
| crawl | No | ||
| domain | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the transparency burden. It does disclose that the data is a snapshot from the crawler's visit ('каким его увидел краулер в момент обхода'), which is valuable temporal context. However, it does not mention whether the output is raw content or parsed directives, any rate limits, or authentication requirements.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise—two short sentences with no filler. The first sentence front-loads the core definition, and the second adds a practical use case. Every word earns its place, making it well-structured for quick consumption.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no annotations, 0% schema coverage, and an output schema present, the description covers the basic purpose and one use case but omits parameter semantics entirely and provides no comparison with siblings. The unexplained 'crawl' parameter is a significant gap for a 2-parameter tool, making the description incomplete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has 0% description coverage and the description does not explain parameters. The word 'домена' implicitly references the domain parameter, but the optional 'crawl' parameter (likely a crawl ID) is completely unexplained. This fails to compensate for the lack of schema-level detail.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states that the tool provides the domain's robots.txt as seen by the crawler at crawl time, which is specific and identifies the resource. It also adds a diagnostic use case ('Объясняет, почему сайт мог выпасть из индекса'). However, it does not explicitly differentiate from sibling tools that might also explain index drops, so it is clear but not distinguishing.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage when investigating why a site might have dropped from the index, giving a clear context. But it offers no explicit alternatives or when-not-to-use guidance relative to sibling tools like cc_errors or cc_dynamics, so an agent gets only an implied usage scenario.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
cc_sectionsA
Топ разделов сайта по числу проиндексированных страниц.
Показывает перекос охвата: какие разделы краулер видит, а какие почти нет.
| Name | Required | Description | Default |
|---|---|---|---|
| crawl | No | ||
| limit | No | ||
| domain | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, so the description carries the transparency burden. It indicates a read-only reporting behavior, but it does not disclose how the crawl parameter affects results, how sections are derived, or any edge-case behavior. The second sentence adds meaningful context about what the output represents.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two short sentences with no redundancy. The main purpose is front-loaded, and the second sentence adds useful interpretation without bloating the text.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description explains the tool's purpose and its output concept, and an output schema exists, so return-value details are not required. However, the lack of parameter semantics, especially for the optional crawl parameter, leaves operational invocation partially underspecified.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, and the description provides no explanation for any of the three parameters (domain, crawl, limit). It does not clarify the role of the optional crawl parameter or the semantics of limit, failing to compensate for the missing schema descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool shows top site sections by number of indexed pages ('Топ разделов сайта по числу проиндексированных страниц') and adds interpretive value with the coverage-skew observation. This specific verb+resource combination distinguishes it from siblings like cc_urls or cc_locales.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives clear context for when this tool is useful: to reveal coverage skew and which sections the crawler sees. However, it does not explicitly name alternatives or state exclusions, so it falls short of full usage guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
cc_urlsC
Конкретные URL домена в кравле вместе с WARC-координатами.
Координаты (warc_filename, offset, length) передаются в cc_fetch_text.
| Name | Required | Description | Default |
|---|---|---|---|
| crawl | No | ||
| limit | No | ||
| domain | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It says the output includes URLs and WARC coordinates, and that coordinates are for cc_fetch_text, but it does not disclose pagination behavior, limit semantics, crawl selection, or whether any side effects occur. This is minimal for a query tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two short sentences with no filler. It front-loads the core purpose and adds a practical pointer to cc_fetch_text, making it highly concise and easy to parse.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return-value structure is likely covered. The description explains the main output (URLs + WARC coordinates) and the downstream tool, but it lacks guidance on limit/crawl parameter behavior and does not clarify when to choose this tool over siblings. It is adequate but not complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It implicitly ties 'domain' to the domain parameter and 'crawl' to the crawl parameter, but it does not explain the 'limit' parameter or the optional/nullable nature of 'crawl'. The parameter meaning is only partially conveyed.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool returns specific URLs for a domain in a crawl along with WARC coordinates. It distinguishes itself from the sibling cc_fetch_text by explicitly saying the coordinates are meant to be passed to that tool, though it lacks an explicit verb like 'list' or 'retrieve'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description mentions that WARC coordinates are passed to cc_fetch_text, which provides a downstream usage hint, but it does not explain when to use cc_urls versus other sibling tools like cc_presence or cc_crawls. There is no explicit alternative guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
cc_zone_suffixesA
Какие суффиксы встречаются внутри зоны (co.uk, org.uk, gov.uk ...).
Нужен, чтобы правильно сузить cc_discover в зонах с составными суффиксами.
| Name | Required | Description | Default |
|---|---|---|---|
| tld | Yes | ||
| crawl | No | ||
| limit | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It only states the output (list of suffixes) but does not disclose side effects, whether it is a read-only operation, or how parameters like crawl and limit affect behavior. This is insufficient for a tool with no annotation support.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two brief sentences, front-loaded with the core purpose and examples. It includes a usage context without any redundancy or filler. Every sentence contributes meaning.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
While the tool is simple and an output schema exists, the description omits parameter explanations and any behavioral details. The language barrier (Russian) further reduces accessibility. The agent cannot confidently invoke this tool correctly without inferring too much.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has 0% description coverage, and the description does not explain the parameters. The examples imply tld is the zone, but crawl and limit are undefined. The description fails to compensate for the missing schema descriptions, leaving the agent guessing about two of the three parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states that the tool lists suffixes within a TLD, providing concrete examples (co.uk, org.uk, gov.uk). It also references cc_discover, distinguishing this tool from sibling tools by specifying its role in narrowing down compound suffixes.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says the tool is needed to correctly narrow cc_discover in zones with compound suffixes, giving a clear when-to-use context. It does not explicitly mention when not to use it, but the guidance is strong enough for an agent to decide when to call this tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
13 tool updates
v0.1.0- First observed
cc_compare - First observed
cc_crawls - First observed
cc_discover - First observed
cc_duplicates - First observed
cc_dynamics - First observed
cc_errors - First observed
cc_fetch_text - First observed
cc_locales - First observed
cc_presence - First observed
cc_robots - First observed
cc_sections - First observed
cc_urls - First observed
cc_zone_suffixes
TDQS
Each tool targets a distinct piece of Common Crawl data: crawls, presence, dynamics, comparisons, locales, sections, URLs, duplicates, errors, robots, discovery, suffix enumeration, and text fetching. There is no overlap; even related tools like cc_presence and cc_dynamics are clearly differentiated by scope (single crawl vs. trend).
All tool names share the cc_ prefix, providing a strong family resemblance. The second part mixes nouns (cc_crawls, cc_urls) and verbs (cc_compare, cc_fetch_text), but the pattern is predictable and readable. Minor inconsistency: some names are plural nouns, some are singular abstract concepts, but the prefix makes the set coherent.
With 13 tools, the set is well-scoped for a Common Crawl analysis server. Each tool addresses a specific analytical need without redundancy or bloat. The count fits comfortably in the ideal 3-15 range.
The toolset covers the full lifecycle of Common Crawl exploration: discovering crawls, checking domain presence, tracking dynamics, comparing domains, drilling into URLs/content, diagnosing errors/robots, and identifying duplicates. It also supports domain discovery and suffix analysis, leaving no obvious dead ends for typical use cases.
Maintenance
Resources
Unclaimed servers have limited discoverability.
Looking for Admin?
If you are the server author, to access and configure the admin panel.
Related MCP Connectors
SEO MCP server — backlinks, domain authority, tech stack, and 18+ tools via Common Crawl.
One MCP for the Web. Easily search, crawl, navigate, and extract websites without getting blocked.…
An MCP server that provides congressional transcripts
Related MCP Servers
- FlicenseNot gradedqualityDmaintenanceBridge the gap between your web crawl and AI language models. With mcp-server-webcrawl, your AI client filters and analyzes web content under your direction or autonomously, extracting insights from your web content. Supports WARC, wget, InterroBot, Katana, and SiteOne crawlers.45Python-
- AlicenseAqualityCmaintenanceMCP server for the Internet Archive's Wayback Machine. Search archived snapshots, extract page text from a specific date, track how a site has changed over time, check if broken links are recoverable, and perform research across Internet Archive collections.63MIT
- AlicenseAqualityBmaintenanceMCP server for the CrawlGraph backlink-intelligence API. Gives any MCP client - Claude Desktop, Claude Code, Cursor, Cline, Zed, Windsurf - backlink lookups and competitor gap analysis built on the public Common Crawl webgraph (4.4B edges, 120M domains).4275MIT
- FlicenseAqualityDmaintenanceMCP server for Common Crawl CDX that enables backlink discovery, expired domain finding, and competitor gap analysis without requiring API keys.4-
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/nikolaymokh-dev/common-crawl-mcp'
If you have feedback or need assistance with the MCP directory API, please join our Discord server