Skip to main content
Glama
nikolaymokh-dev

common-crawl-mcp

common-crawl-mcp

MCP-сервер и CLI для запросов к Common Crawl — открытому корпусу веба, на котором учат языковые модели.

Отвечает на вопрос «виден ли сайт для ИИ»: сколько его страниц попало в корпус, какие разделы, как это менялось, и что видит модель, читая конкретную страницу.

Работает через AWS Athena поверх колоночного индекса (cc-index), а не через публичный API Common Crawl — тот регулярно отдаёт 504. Запрос по домену стоит около $0.00005.

Установка

uv tool install git+https://github.com/nikolaymokh-dev/common-crawl-mcp

Нужны AWS-креды с доступом к Athena, Glue и чтением бакета commoncrawl. Берутся из ~/.aws обычным способом (AWS_PROFILE, AWS_REGION; по умолчанию us-east-1).

Related MCP server: mcp-server-wayback

CLI

ccq example.com                          # страниц в свежем кравле
ccq example.com --dynamics               # динамика по 18 последним кравлам
ccq example.com --locales --sections     # разбивка по языкам и разделам
ccq example.com --compare example.org,example.net
ccq example.com --errors                 # 404/301/5xx, которые получил краулер
ccq example.com --robots                 # robots.txt глазами краулера
ccq --discover de --min-pages 100        # домены зоны .de
ccq --zone-suffixes uk                   # co.uk / org.uk / gov.uk ...
ccq --list-crawls

Любую команду можно получить в JSON: --json.

MCP

claude mcp add common-crawl -- uvx --from git+https://github.com/nikolaymokh-dev/common-crawl-mcp common-crawl-mcp

Тулы

Тул

Зачем

cc_presence

Сколько страниц домена в кравле

cc_dynamics

Динамика по кравлам — основа для любых выводов

cc_compare

Несколько доменов одним сканом

cc_locales

Разбивка по языковым префиксам пути

cc_sections

Топ разделов: где охват проседает

cc_urls

Конкретные URL + координаты в архиве

cc_duplicates

Страницы-дубли (одинаковый content_digest)

cc_errors

Не-200 ответы у домена

cc_robots

robots.txt в момент обхода

cc_discover

Домены по зоне и/или подстроке

cc_zone_suffixes

Суффиксы внутри зоны

cc_fetch_text

Текст страницы из архива — то, что читает модель

cc_crawls

Список доступных кравлов

Что стоит знать

Ноль страниц ≠ сайта нет. Охват Common Crawl заметно скачет между кравлами: сайт, стабильно дающий сотни страниц, может провалиться до единиц. Любой вывод делается по cc_dynamics, а не по одному снимку.

Поиск по зоне почти бесплатен. Индекс отсортирован по url_surtkey — имени хоста задом наперёд, — поэтому домены одной зоны лежат в файлах подряд, и Athena пропускает всё остальное. Замерено на CC-MAIN-2026-30: .uz — 0.28 MB скана на 7 954 домена, .de — 23 MB на 2 075 692 домена.

Поиск по подстроке — нет. host_contains не может воспользоваться этой сортировкой, поэтому без указания зоны он читает колонку хостов по всему кравлу. Тул это запрещает, пока не передашь allow_full_scan=True. Заодно зона отсекает ложные совпадения: короткая подстрока по всему вебу цепляет случайные домены из чужих зон, внутри одной зоны — нет.

Составные зоны. tld='uk' захватывает и example.uk, и example.co.uk. Нужен конкретный сегмент — фильтруй registry_suffix='co.uk'. Посмотреть, что вообще есть в зоне, — cc_zone_suffixes.

Стоимость возвращается вместе с данными. В каждом ответе есть scanned_mb и cost_usd. Athena billing округляет запрос вверх до 10 MB.

Разработка

uv sync --extra dev
uv run pytest              # юнит-тесты, сети не требуют

Структура: athena.py — транспорт, crawls.py — какие кравлы есть, queries.py — построители SQL (чистые функции, тестируются без AWS), api.py — операции, warc.py — извлечение текста, cli.py и server.py — две обёртки над api.

Лицензия

MIT

Available Tools

13 tools
cc_compareA

Сравнить несколько доменов между собой по кравлам (матрица домен × кравл).

Один скан на все домены — дешевле, чем вызывать cc_dynamics по очереди.

ParametersJSON Schema
NameRequiredDescriptionDefault
last_nNo
domainsYes

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A3.6/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the burden. It discloses the output as a domain×crawl matrix and mentions a single scan, which implies an efficient batch operation. However, it does not describe any side effects, permissions, or data freshness, leaving some behavioral aspects uncovered.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description consists of two concise sentences. The first states the purpose and output structure, the second explains the cost advantage. Every word contributes value and information is front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has an output schema, which covers return value structure, but lacks annotations and has minimal parameter explanation. The description is clear about purpose and efficiency but omits details like how last_n works, what constitutes a 'scan', and prerequisites. It is adequate for a simple comparison tool but not complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has 0% description coverage, and the description does not explain the meaning of 'domains' or 'last_n'. It only alludes to 'several domains' but provides no semantic guidance for parameters, so it fails to compensate for the schema's lack of descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb 'Сравнить' (compare) with a clear resource ('несколько доменов... по кравлам') and explicitly describes the output as a 'матрица домен × кравл'. It distinguishes itself from siblings by focusing on multi-domain comparison and even references the alternative cc_dynamics.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It provides clear context: 'Один скан на все домены — дешевле, чем вызывать cc_dynamics по очереди' (one scan for all domains is cheaper than calling cc_dynamics one by one). This gives an explicit alternative and a reason to prefer this tool, though it doesn't explicitly state when not to use it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

cc_crawlsA

Список доступных кравлов Common Crawl (свежий последним).

Полезно, чтобы узнать актуальное имя кравла перед остальными запросами.

ParametersJSON Schema
NameRequiredDescriptionDefault
refreshNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A3.6/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must carry the behavioral transparency burden. It does not disclose whether the request is read-only, whether results are cached, or what the 'refresh' parameter does. This is a significant gap for a tool with an optional refresh parameter.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two short sentences, front-loaded with the main purpose. No wasted words, and it efficiently conveys the key point about ordering and usage.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple list tool with an output schema, the description is adequate but has clear gaps: it omits the refresh parameter's behavior and any note about read-only semantics. It lacks details that would help an agent fully anticipate the tool's behavior, even though the output schema covers return values.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has one boolean parameter 'refresh' with 0% description coverage. The description does not mention this parameter at all, failing to explain its purpose or effect. The parameter name hints at refreshing but is not explicit, so the description adds no value beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool lists available Common Crawl crawls, with the newest last. This distinguishes it from sibling tools that focus on specific aspects like presence or dynamics.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It explicitly says this is useful to get the current crawl name before making other requests, providing clear context for when to use it. It doesn't mention alternatives or exclusions, but the primary use case is well stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

cc_discoverB

Найти домены по зоне и/или подстроке в имени хоста.

tld — любая зона: 'uz', 'de', 'uk'. Стоит копейки: индекс отсортирован по хосту задом наперёд, поэтому зона читается почти без скана. registry_suffix — точнее для составных зон: 'co.uk', 'com.au'. host_contains — подстрока; требует зоны, иначе это скан всего кравла.

ParametersJSON Schema
NameRequiredDescriptionDefault
tldNo
crawlNo
limitNo
min_pagesNo
host_containsNo
allow_full_scanNo
registry_suffixNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

B3.3/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so the description carries the burden. It discloses performance characteristics (index sorted reversed, cheap zone lookup) and warns that host_contains without a zone triggers a full crawl scan. However, it omits details on limit/min_pages/allow_full_scan behavior and lacks a clear statement about return format.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is brief, with a clear opening statement followed by concise parameter notes. Every sentence adds information about usage or performance; no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 7-parameter tool with no annotations, the description covers only the search criteria and one performance warning. It omits the meaning of crawl, limit, min_pages, and allow_full_scan, making it incomplete. An output schema exists but is not provided to the agent in this context, so those gaps remain.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description must compensate. It explains tld, registry_suffix, and host_contains with examples and constraints, covering 3 of 7 parameters. The remaining parameters (crawl, limit, min_pages, allow_full_scan) are left undocumented.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states 'Find domains by zone and/or substring in host name' — a clear verb+resource+criteria. It does not explicitly differentiate from sibling tools, but the focus on zone/substring discovery is distinct enough.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use cc_discover vs sibling tools. It provides parameter-level usage advice (compound zones for registry_suffix, host_contains requires a zone) but does not address tool selection or exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

cc_duplicatesC

Группы страниц с идентичным содержимым (совпадает content_digest).

Дублирующийся контент под разными URL — типовая SEO-проблема.

ParametersJSON Schema
NameRequiredDescriptionDefault
crawlNo
limitNo
domainYes

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden. It reveals that grouping is based on content_digest, but does not disclose output format, ordering, pagination, whether empty groups are included, or meaning of the 'crawl' parameter. For a read-only analysis tool, this lacks sufficient behavioral detail beyond the basic grouping criterion.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences, front-loaded with the core function ('groups pages with identical content') and then adds context about SEO relevance. Every sentence earns its place; there is no unnecessary fluff or redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite having an output schema, the description does not explain the meaning of the required 'domain' parameter, the optional 'crawl' parameter, or the 'limit' parameter. Without these, a tool user cannot confidently invoke it correctly. The contextual gap is significant for a tool with three parameters and zero parameter descriptions.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, and the description does not explain any of the three parameters (domain, crawl, limit). It fails to mention that domain is required or that limit defaults to 25, and the 'crawl' parameter is entirely unaddressed, leaving agents without enough information to set parameters correctly.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states it groups pages with identical content (matching content_digest) and frames this as a duplicate-content SEO issue. This is a specific verb+resource description that distinguishes it from most sibling tools, though it does not explicitly name an alternative like 'use cc_compare for...'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for detecting duplicate content in SEO contexts ('типовая SEO-проблема'), but it does not explicitly state when to prefer this tool over siblings or provide exclusions. Usage guidance is present only by implication.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

cc_dynamicsA

Динамика индексации домена по последним N кравлам.

Главный тул для выводов: один кравл шумит, тренд — нет. last_n=0 — все доступные кравлы (дороже).

ParametersJSON Schema
NameRequiredDescriptionDefault
domainYes
last_nNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the disclosure burden. It adds useful behavioral insight: one crawl is noisy, trend is not, and last_n=0 fetches all available crawls at higher cost. It does not describe output details, but an output schema exists, which reduces the need for that.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise: two lines plus one usage note. Every sentence adds value—purpose, trend guidance, and special parameter meaning—with no filler or repetition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a two-parameter tool with an output schema, the description covers purpose, usage guidance, parameter semantics, and cost behavior. It is fully self-contained enough for an agent to decide when and how to invoke it.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It explains last_n as the number of recent crawls and explicitly documents last_n=0 behavior and cost. The domain parameter is self-explanatory from the name, so the partial coverage is adequate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly identifies the tool as showing domain indexing dynamics over the last N crawls, and adds that it is the main tool for drawing conclusions (trend vs. single-crawl noise). This differentiates it from siblings like cc_crawls, though it does not explicitly name an alternative, so it falls just short of a 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It provides clear usage context: use this tool to observe trends because a single crawl is noisy. It implies a distinction from sibling tools and mentions the cost of last_n=0, but does not explicitly state when not to use it or name alternatives, so it does not meet the full 'when/when-not/alternatives' criteria.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

cc_errorsC

Не-200 ответы, которые краулер получил от домена: 404, редиректы, 5xx.

ParametersJSON Schema
NameRequiredDescriptionDefault
crawlNo
domainYes

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

C2.8/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It states the output type but does not mention any limitations, such as pagination, time range, or whether redirects are followed. The examples are useful, but significant behavioral details are missing.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single concise sentence that front-loads the core concept ('Non-200 responses') and adds concrete examples. It avoids unnecessary words, though it omits useful context; but for conciseness and structure, it is efficient.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool has two parameters (one optional), an output schema, and a set of sibling tools, the description is under-specified. It does cover the primary purpose, but fails to explain the optional 'crawl' parameter or provide any usage guidance, leaving the agent with gaps in how to correctly invoke the tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The description provides zero explanation of the parameters. The 'domain' parameter is inferable from the description, but the optional 'crawl' parameter is completely unexplained. With schema description coverage at 0%, the description fails to compensate for the lack of parameter semantics.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly identifies the resource (non-200 HTTP responses) and scope (from a domain), with specific examples (404, redirects, 5xx). It distinguishes from sibling tools like cc_urls by focusing on errors rather than all URLs, though it lacks an explicit verb like 'list' or 'get'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The intended use is implied: to retrieve error responses for a given domain. However, there is no explicit guidance on when to use this tool over alternatives, nor any exclusion criteria or context about prerequisites. This is barely above no guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

cc_fetch_textA

Прочитать текст страницы из WARC-архива по координатам из cc_urls или cc_robots.

Показывает страницу так, как её видит модель при обучении. Athena не задействована, запрос бесплатный.

ParametersJSON Schema
NameRequiredDescriptionDefault
lengthYes
offsetYes
max_charsNo
warc_filenameYes

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A3.8/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

There are no annotations, so the description must carry the burden. It adds useful behavioral context: shows the page as the model sees it during training, and states that Athena is not used and the request is free. However, it does not disclose potential side effects, authentication requirements, or what happens with invalid coordinates. This is moderate transparency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences, front-loaded with the primary action, and contains no fluff. It efficiently conveys the core purpose and a few key traits.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool is simple and has an output schema, so return values are covered. The description covers purpose, source of coordinates, and cost, but lacks per-parameter explanation and any mention of error cases or limitations beyond the default max_chars. It is adequate but not fully complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It only collectively refers to parameters as 'coordinates from cc_urls or cc_robots' without explaining warc_filename, offset, length, or max_chars individually. This provides minimal semantic meaning and does not adequately compensate for the lack of schema descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource: 'Прочитать текст страницы из WARC-архива' (read page text from WARC archive), and clearly distinguishes itself from siblings like cc_urls and cc_robots by indicating it consumes their output. It also adds a unique perspective: 'Shows the page as the model sees it during training.'

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear usage context: it is meant for fetching text using coordinates from cc_urls or cc_robots, and notes that it does not use Athena and is free. It does not explicitly state exclusions or name alternative tools, but the prerequisite and cost/backend context are helpful.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

cc_localesC

Разбивка страниц домена по языковому префиксу пути (/ru, /en, /uz ...).

ParametersJSON Schema
NameRequiredDescriptionDefault
crawlNo
domainYes

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

C2.8/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

There are no annotations, so the description carries the full behavioral transparency burden. It does not state whether the operation is read-only, whether the crawl parameter affects execution, or any limitations, rate limits, or side effects. Only the grouping behavior is disclosed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, front-loaded sentence with no filler. Every word contributes to the core meaning, and the concrete examples make it easy to parse.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Although an output schema exists, the input side is incomplete: the `crawl` parameter is undefined, there are no usage guidelines, and there is no behavioral/safety disclosure. The description clearly communicates the core purpose but is not sufficient for reliable tool invocation in all cases.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The phrase 'домена' gives some meaning to the `domain` parameter, but the optional `crawl` parameter is entirely unexplained. With 0% schema description coverage, the description fails to compensate for the missing parameter documentation.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool produces a breakdown of a domain's pages by path language prefix (/ru, /en, /uz), which is specific and distinguishes it from sibling tools like cc_zone_suffixes or cc_sections. However, it uses the noun 'разбивка' rather than an explicit verb such as 'returns' or 'lists', so it stops just short of a full 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No when-to-use guidance is provided. The description does not compare cc_locales with sibling tools, mention prerequisites, or state when it should be preferred over alternatives like cc_presence or cc_urls.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

cc_presenceA

Сколько страниц домена попало в кравл — есть ли сайт в данных, на которых учат LLM.

crawl не указан — берётся свежий. Ноль страниц не означает отсутствие сайта: охват сильно скачет, проверяй через cc_dynamics.

ParametersJSON Schema
NameRequiredDescriptionDefault
crawlNo
domainYes

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It discloses the default crawl selection behavior and, importantly, the interpretive caveat that zero pages does not imply site absence. However, it does not describe the output format or potential edge cases (e.g., invalid domains), but the presence of an output schema mitigates the need for return details. Overall, it provides meaningful behavioral context beyond the schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise: two sentences, each earning its place. The first states the core function and purpose, the second provides essential caveats and a pointer to an alternative tool. No redundant information, ideal structure for an AI agent to quickly grasp.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity (2 parameters, one optional), the presence of an output schema, and the no-annotation context, the description covers all essential aspects: what it does, default parameter behavior, and a critical interpretive caveat. It is complete for an agent to invoke correctly without additional documentation. The pointer to cc_dynamics also covers follow-up scenarios.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has 0% description coverage, so the description must compensate. It explicitly explains the 'crawl' parameter's default behavior ('crawl не указан — берётся свежий'), adding value beyond the schema's bare type/null default. The 'domain' parameter is not elaborated, but its purpose is implicitly clear from the tool name and description. This partial compensation justifies a score above baseline.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's function: 'Сколько страниц домена попало в кравл' (how many pages of a domain are in the crawl), with a specific verb and resource. It also frames the purpose as checking site presence in LLM training data, which distinguishes it from siblings like cc_urls or cc_sections. The reference to cc_dynamics for further checks further differentiates its role.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit usage guidance: it explains the default behavior when the 'crawl' parameter is omitted ('crawl не указан — берётся свежий') and gives a clear alternative: if the result is zero, it advises checking via cc_dynamics because coverage fluctuates. This tells the user when to use this tool and when to switch to another, which is exactly what this dimension requires.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

cc_robotsB

robots.txt домена, каким его увидел краулер в момент обхода.

Объясняет, почему сайт мог выпасть из индекса.

ParametersJSON Schema
NameRequiredDescriptionDefault
crawlNo
domainYes

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

B3.2/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the transparency burden. It does disclose that the data is a snapshot from the crawler's visit ('каким его увидел краулер в момент обхода'), which is valuable temporal context. However, it does not mention whether the output is raw content or parsed directives, any rate limits, or authentication requirements.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise—two short sentences with no filler. The first sentence front-loads the core definition, and the second adds a practical use case. Every word earns its place, making it well-structured for quick consumption.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no annotations, 0% schema coverage, and an output schema present, the description covers the basic purpose and one use case but omits parameter semantics entirely and provides no comparison with siblings. The unexplained 'crawl' parameter is a significant gap for a 2-parameter tool, making the description incomplete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has 0% description coverage and the description does not explain parameters. The word 'домена' implicitly references the domain parameter, but the optional 'crawl' parameter (likely a crawl ID) is completely unexplained. This fails to compensate for the lack of schema-level detail.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states that the tool provides the domain's robots.txt as seen by the crawler at crawl time, which is specific and identifies the resource. It also adds a diagnostic use case ('Объясняет, почему сайт мог выпасть из индекса'). However, it does not explicitly differentiate from sibling tools that might also explain index drops, so it is clear but not distinguishing.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage when investigating why a site might have dropped from the index, giving a clear context. But it offers no explicit alternatives or when-not-to-use guidance relative to sibling tools like cc_errors or cc_dynamics, so an agent gets only an implied usage scenario.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

cc_sectionsA

Топ разделов сайта по числу проиндексированных страниц.

Показывает перекос охвата: какие разделы краулер видит, а какие почти нет.

ParametersJSON Schema
NameRequiredDescriptionDefault
crawlNo
limitNo
domainYes

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A3.6/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations exist, so the description carries the transparency burden. It indicates a read-only reporting behavior, but it does not disclose how the crawl parameter affects results, how sections are derived, or any edge-case behavior. The second sentence adds meaningful context about what the output represents.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two short sentences with no redundancy. The main purpose is front-loaded, and the second sentence adds useful interpretation without bloating the text.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description explains the tool's purpose and its output concept, and an output schema exists, so return-value details are not required. However, the lack of parameter semantics, especially for the optional crawl parameter, leaves operational invocation partially underspecified.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, and the description provides no explanation for any of the three parameters (domain, crawl, limit). It does not clarify the role of the optional crawl parameter or the semantics of limit, failing to compensate for the missing schema descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool shows top site sections by number of indexed pages ('Топ разделов сайта по числу проиндексированных страниц') and adds interpretive value with the coverage-skew observation. This specific verb+resource combination distinguishes it from siblings like cc_urls or cc_locales.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives clear context for when this tool is useful: to reveal coverage skew and which sections the crawler sees. However, it does not explicitly name alternatives or state exclusions, so it falls short of full usage guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

cc_urlsC

Конкретные URL домена в кравле вместе с WARC-координатами.

Координаты (warc_filename, offset, length) передаются в cc_fetch_text.

ParametersJSON Schema
NameRequiredDescriptionDefault
crawlNo
limitNo
domainYes

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It says the output includes URLs and WARC coordinates, and that coordinates are for cc_fetch_text, but it does not disclose pagination behavior, limit semantics, crawl selection, or whether any side effects occur. This is minimal for a query tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two short sentences with no filler. It front-loads the core purpose and adds a practical pointer to cc_fetch_text, making it highly concise and easy to parse.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return-value structure is likely covered. The description explains the main output (URLs + WARC coordinates) and the downstream tool, but it lacks guidance on limit/crawl parameter behavior and does not clarify when to choose this tool over siblings. It is adequate but not complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It implicitly ties 'domain' to the domain parameter and 'crawl' to the crawl parameter, but it does not explain the 'limit' parameter or the optional/nullable nature of 'crawl'. The parameter meaning is only partially conveyed.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool returns specific URLs for a domain in a crawl along with WARC coordinates. It distinguishes itself from the sibling cc_fetch_text by explicitly saying the coordinates are meant to be passed to that tool, though it lacks an explicit verb like 'list' or 'retrieve'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description mentions that WARC coordinates are passed to cc_fetch_text, which provides a downstream usage hint, but it does not explain when to use cc_urls versus other sibling tools like cc_presence or cc_crawls. There is no explicit alternative guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

cc_zone_suffixesA

Какие суффиксы встречаются внутри зоны (co.uk, org.uk, gov.uk ...).

Нужен, чтобы правильно сузить cc_discover в зонах с составными суффиксами.

ParametersJSON Schema
NameRequiredDescriptionDefault
tldYes
crawlNo
limitNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A3.5/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It only states the output (list of suffixes) but does not disclose side effects, whether it is a read-only operation, or how parameters like crawl and limit affect behavior. This is insufficient for a tool with no annotation support.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two brief sentences, front-loaded with the core purpose and examples. It includes a usage context without any redundancy or filler. Every sentence contributes meaning.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

While the tool is simple and an output schema exists, the description omits parameter explanations and any behavioral details. The language barrier (Russian) further reduces accessibility. The agent cannot confidently invoke this tool correctly without inferring too much.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has 0% description coverage, and the description does not explain the parameters. The examples imply tld is the zone, but crawl and limit are undefined. The description fails to compensate for the missing schema descriptions, leaving the agent guessing about two of the three parameters.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states that the tool lists suffixes within a TLD, providing concrete examples (co.uk, org.uk, gov.uk). It also references cc_discover, distinguishing this tool from sibling tools by specifying its role in narrowing down compound suffixes.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly says the tool is needed to correctly narrow cc_discover in zones with compound suffixes, giving a clear when-to-use context. It does not explicitly mention when not to use it, but the guidance is strong enough for an agent to decide when to call this tool.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. 13 tool updatesv0.1.0
    • First observedcc_compare
    • First observedcc_crawls
    • First observedcc_discover
    • First observedcc_duplicates
    • First observedcc_dynamics
    • First observedcc_errors
    • First observedcc_fetch_text
    • First observedcc_locales
    • First observedcc_presence
    • First observedcc_robots
    • First observedcc_sections
    • First observedcc_urls
    • First observedcc_zone_suffixes

TDQS

A3.7/5.0
Disambiguation5/5

Each tool targets a distinct piece of Common Crawl data: crawls, presence, dynamics, comparisons, locales, sections, URLs, duplicates, errors, robots, discovery, suffix enumeration, and text fetching. There is no overlap; even related tools like cc_presence and cc_dynamics are clearly differentiated by scope (single crawl vs. trend).

Naming Consistency4/5

All tool names share the cc_ prefix, providing a strong family resemblance. The second part mixes nouns (cc_crawls, cc_urls) and verbs (cc_compare, cc_fetch_text), but the pattern is predictable and readable. Minor inconsistency: some names are plural nouns, some are singular abstract concepts, but the prefix makes the set coherent.

Tool Count5/5

With 13 tools, the set is well-scoped for a Common Crawl analysis server. Each tool addresses a specific analytical need without redundancy or bloat. The count fits comfortably in the ideal 3-15 range.

Completeness5/5

The toolset covers the full lifecycle of Common Crawl exploration: discovering crawls, checking domain presence, tracking dynamics, comparing domains, drilling into URLs/content, diagnosing errors/robots, and identifying duplicates. It also supports domain discovery and suffix analysis, leaving no obvious dead ends for typical use cases.

Maintenance

ActivitySlowing
ResponsivenessNo issues

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

  • F
    license
    Not graded
    quality
    D
    maintenance
    Bridge the gap between your web crawl and AI language models. With mcp-server-webcrawl, your AI client filters and analyzes web content under your direction or autonomously, extracting insights from your web content. Supports WARC, wget, InterroBot, Katana, and SiteOne crawlers.
    45
    Python
    -
  • A
    license
    A
    quality
    C
    maintenance
    MCP server for the Internet Archive's Wayback Machine. Search archived snapshots, extract page text from a specific date, track how a site has changed over time, check if broken links are recoverable, and perform research across Internet Archive collections.
    6
    3
    MIT
  • A
    license
    A
    quality
    B
    maintenance
    MCP server for the CrawlGraph backlink-intelligence API. Gives any MCP client - Claude Desktop, Claude Code, Cursor, Cline, Zed, Windsurf - backlink lookups and competitor gap analysis built on the public Common Crawl webgraph (4.4B edges, 120M domains).
    4
    27
    5
    MIT
  • F
    license
    A
    quality
    D
    maintenance
    MCP server for Common Crawl CDX that enables backlink discovery, expired domain finding, and competitor gap analysis without requiring API keys.
    4
    -

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/nikolaymokh-dev/common-crawl-mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server