How do I use crawl-census?

1. Click on "Install Server". 2. Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state. 3. In the chat, type @ followed by the MCP server name and your instructions, e.g., "@crawl-census preflight these domains for gptbot" That's it! The server will respond to your query, and you can continue using it as needed. Here is a step-by-step guide with screenshots.

de en es ja ko ru zh

crawl-census

by taylorsmithgg

Overview Schema Related Servers Score Discussions

JavaScript

Remote

crawl-census-client

Ask before you fetch. A drop-in client that stops your crawler spending requests on doors that are shut, and stops it routing around content someone is trying to sell.

Reading robots.txt answers one question and hides two others. Measured across 23,482 domains by Crawl Census:

2,874 domains permit AI agents in robots.txt and then refuse those same agents at the network edge. A parser sees permission; the fetch returns 403. You pay for the round trip and get nothing.
208 domains answer an AI user agent with HTTP 402 Payment Required. That is a price, not a refusal. Treating it as a block walks away from content the operator wants to sell you. Retrying around it takes something they are charging for.

No dependencies. No key required.

MCP server

The same measurement is exposed as a remote MCP server, so an agent can ask before it fetches rather than after it fails. Listed in the official MCP registry as io.github.taylorsmithgg/crawl-census.

{ "mcpServers": { "crawl-census": { "url": "https://crawlcensus.com/mcp" } } }

Tool	Answers
`crawl_preflight`	will these domains serve my agent, refuse it, or charge it?
`agent_profile`	what does this census publish about my crawler, and how do I correct it?
`census_facts`	the headline findings as dated records with denominators and citation lines
`site_report`	the stored audit for one domain
`scan_site`	measure a domain now
`census_stats`	corpus-level totals

No authentication for read tools. Streamable HTTP.

Related MCP server: Maango-mcp

Install

npm i github:taylorsmithgg/crawl-census-client
pip install git+https://github.com/taylorsmithgg/crawl-census-client

Use

import { politeFetch } from "crawl-census-client";

const r = await politeFetch("https://example.com/", { agent: "gptbot" });
if (r.skipped) console.log(r.verdict, r.reason);   // disallow | refuse | pay
else           process(await r.response.text());

from crawl_census import polite_fetch

r = polite_fetch("https://example.com/", agent="gptbot")
if r.skipped:
    print(r.verdict, r.reason)
else:
    process(r.body)

Skipping is returned, not raised. It is the normal outcome for a large share of the web, and a crawl loop should be able to count skips without a try/except around every URL.

Split a queue before crawling it

One call per 1,000 domains instead of one per host:

const { crawl, skip, pay, unknown } = await partition(urls, { agent: "gptbot" });

p = partition(urls, agent="gptbot")
p.crawl, p.skip, p.pay, p.unknown

Or just take the file

For a fetcher that only needs a deny list in memory, skip the per-domain calls entirely:

curl https://crawlcensus.com/agents/gptbot/blocklist.txt   # one domain per line, commented header

const sync = await syncBlocklist("gptbot");   // full list once
if (sync.blocked.has(host)) skip();
setInterval(() => sync.refresh(), 3600_000);  // then deltas only, a few hundred bytes

sync = BlocklistSync("gptbot")
if host in sync: skip()
sync.refresh()          # {'added': 3, 'removed': 1, 'size': 3310, 'cursor': ...}

The delta feed is https://crawlcensus.com/agents/<agent>/changes.json?since=<unix> and each response carries next_since, so a long-running crawler stays current on a few hundred bytes an hour instead of re-downloading the list.

That file covers robots.txt only. Edge refusal and HTTP 402 are per-request behaviours and still need preflight or politeFetch.

What a crawl costs the census

Measured, not asserted. Twenty hosts fetched concurrently used to cost twenty preflight calls carrying one domain each; the same host requested three times at once cost three, because the cache only helps after the first lookup resolves. The anonymous allowance is 240 calls an hour, so a crawler hit its ceiling at 240 hosts when one call covers twenty-five.

politeFetch now shares work automatically: lookups issued in the same tick leave as one batched call, and concurrent lookups for the same host await a single request.

pattern	before	now
20 hosts, concurrent	20 calls	1 call of 20
1 host, 3 URLs, concurrent	3 calls	1 call
60 hosts, concurrent	60 calls	3 calls (25 / 25 / 10)
`partition` then fetch	2 calls	1 call

batchSize defaults to 25, the per-call cap without a key. Raise it with a Pro or Data key. batchWaitMs widens the coalescing window for concurrency that arrives in waves rather than all at once; the default of zero flushes on the next tick.

Paying, when an origin quotes a price

A pay verdict carries the amount when the origin named one:

const r = await politeFetch(url, { agent: "claudebot" });
if (r.verdict === "pay") console.log(r.price);   // "USD 0.5", or null if none was quoted

Two things worth knowing. Most origins answering HTTP 402 name no amount at all, so price is usually null and the arrangement has to be made out of band. And pricing is per crawler: across the measured corpus, 78 of 213 charging origins charge some agents and serve others free, so ask with your own token rather than assuming a domain on the list will charge you.

Two kinds of unknown

partition splits a work queue into crawl, pay, skip, unknown and undecidable.

The last two look alike and are not. unknown means the census has not measured that domain yet: submit it and the next pass gets a real verdict. undecidable means the site's robots.txt disallows CrawlCensusBot, so this census will never measure it — retrying is guaranteed waste, and a loop that resubmits its unknowns each pass would resubmit those forever. The server marks the difference with a measurable boolean; read that, never the reason prose.

const p = await partition(urls, { agent: "gptbot" });
await Promise.all(p.crawl.map(politeFetchOne));
if (p.unmeasured.length) await submitUnmeasured(p, { agent: "gptbot" });
// p.undecidable: read their robots.txt yourself. Asking us again cannot help.

Submission is a separate call on purpose. A library that quietly POSTs during what reads as a lookup is a bad citizen, and you should choose when your queue positions are spent.

Skipping everything that will not serve you

A deny list is the smaller half. Measured against the live census, a crawler that skips only robots disallows still spends around 2,900 requests a pass on domains that permit it in robots.txt and refuse it at the edge, or that answer HTTP 402 — for PerplexityBot that set is larger than its deny list. Those fetches return nothing and cost a round trip each.

const skip = await syncSkipList("gptbot");
if (skip.has(host)) continue;        // disallowed, refused at the edge, or priced
skip.why(host);                      // "disallow" | "pay" | "refuse" | null
setInterval(() => skip.refresh(), 3600_000);

agent	deny list	also skippable	total
GPTBot	3,542	2,944	6,486
ClaudeBot	3,169	3,194	6,363
PerplexityBot	1,128	3,658	4,786

The three sets are kept apart internally, so a change moves the one it belongs to. why() follows the same precedence as preflight: a disallow outranks a price, because a price is not permission.

Keeping a deny list current

syncBlocklist / BlocklistSync download the list once, then apply only what changed.

The list is served with the exact position in the change feed it was built at, in an x-cursor header and a # cursor: comment. The clients read it and resume from there, so there is no gap between the snapshot and the first poll, and no reliance on your clock being in step with the server's. Polling by second cannot express a position inside a second, and a crawl batch writes dozens of events into one, so a second-granularity resume can drop the remainder of it: measured live, resuming after the first of three same-second changes recovered both siblings by cursor and neither by second.

const sync = await syncBlocklist("gptbot");   // cursor comes from the list itself
if (sync.blocked.has(host)) skip();
setInterval(() => sync.refresh(), 3600_000);  // a few hundred bytes per poll

refresh() applies only robots transitions to the list, because that is what the list is made of. Edge refusals, new prices and llms.txt changes come back in other for you to act on separately — an earlier version deleted those domains from the deny list, so a crawler resumed fetching exactly what had just started refusing it.

Verdicts

The authoritative definition of each verdict — what it means, what it obliges a crawler to do, and whether asking again could change it — is published as data at /api/v1/verdicts. The list below is a summary; if the two ever disagree, the endpoint is right and this file is stale.

politeFetch skips disallow, refuse and pay by default, which is the endpoint's derived do_not_fetch set. The copy here is deliberate — a crawl loop should not need a network call to decide — and a test compares the two so it cannot drift unnoticed.

Verdict	Meaning	Default behaviour
`allow`	robots.txt permits this agent, and a live request carrying its user agent was served	fetch
`disallow`	robots.txt forbids this agent at the site root	skip
`refuse`	robots.txt permits it; the edge refused it anyway. The allowance is not real	skip
`pay`	the origin answered HTTP 402. It will serve this agent on commercial terms	skip
`unknown`	not measured recently enough to answer	fetch

onPay: "fetch" (on_pay="fetch") overrides the paywall default. It is an explicit opt-in and is recorded on the result as paidRouteOverridden so it shows up in your logs.

It degrades, it does not fail

If the census is unreachable every verdict becomes unknown and your crawl proceeds as it normally would. A third-party outage must never stop your pipeline. There is a live test for exactly this.

What we publish about your agent

const p = await agentProfile("claudebot");
// robots disallow rate, edge refusal rate, operator page, correction channel

If a figure is wrong, the correction channel is in that response and on your operator page. Registry facts are corrected without argument; disputed measurements are published alongside the dispute with the underlying scan records, rather than quietly amended.

Limits

25 domains per preflight call anonymously, 200 with a Pro key, 1,000 with a Data key. Pass apiKey. Details at https://crawlcensus.com/for-crawlers.

Tests

node test.mjs runs against the live census on purpose. The value of this client is whether its verdicts match reality, and a mocked test would assert only that the mock agrees with itself.

MIT. Data is CC BY 4.0, attribute as "Source: Crawl Census (crawlcensus.com)".

Available Tools

7 tools

agent_profileAInspect

What this census measures and publishes about one AI crawler: how often it is disallowed in robots.txt, how often live requests carrying its user agent are refused at the network edge whatever robots.txt says, whether its operator documents it as honouring robots.txt, and where to correct any of that. Intended for the operator of the agent as much as for anyone studying it, so it includes the correction channel and the public page a claim can be disputed against.

ParametersJSON Schema

Name	Required	Description	Default
`agent`	Yes	Crawler token, e.g. gptbot, claudebot, ccbot, google-extended.

TDQS

A4.1/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the behavioral burden and does so well: it discloses what is measured, that live requests are checked at the network edge, that operator documentation is assessed, and that correction/dispute channels are included. It does not explicitly state read-only status or error behavior, but the 'measures and publishes' framing strongly implies a read operation.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The purpose is front-loaded and the metric list is scannable. The second sentence slightly repeats the correction-channel idea already mentioned in the list, but it adds the audience and the dispute-page detail, so it earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a single-parameter tool with no output schema, the description adequately conveys what data the profile returns and that correction channels are included. It could be more explicit about output format or unknown-agent handling, but the essential guidance for selecting and invoking the tool is present.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema already documents `agent` with examples like gptbot and claudebot at 100% coverage. The description only adds that the tool is about 'one AI crawler,' contributing no new parameter syntax or constraints, so baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb+resource: 'What this census measures and publishes about one AI crawler.' It then enumerates the exact metrics reported, clearly distinguishing this per-agent profile tool from sibling tools that focus on sites, stats, or submissions.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The intended use is clear: retrieve published census data about one AI crawler, aimed at both the crawler operator and external researchers. It does not explicitly name alternative sibling tools or exclusion conditions, but the usage context is unambiguous.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

census_factsAInspect

Every headline finding from the census as discrete, dated records rather than prose. Each carries its value, unit, denominator, measurement date, the page it comes from and a ready-made citation line, plus the caveats that apply to all of them. Use this when answering a question about how open the web is to AI crawlers: lifting a percentage out of a rendered page loses the denominator and the date, which is what makes the number wrong when it is repeated.

ParametersJSON Schema

Name	Required	Description	Default
No parameters

TDQS

A4.4/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden, and it does so well. It discloses the structure of each returned record, the presence of caveats, and the important failure mode of losing denominators and dates when numbers are repeated. This gives an agent a clear picture of the tool's behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two dense, well-front-loaded sentences cover what the tool returns, how records are structured, and when to use it. Every clause earns its place, with no filler or repetition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a zero-parameter tool with no output schema, the description is remarkably complete: it explains the record fields, caveats, and the specific use case. An agent can understand what it will get back and why it is the right tool without further clarification.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has zero parameters, so this is the baseline-4 case. The description focuses on output record semantics rather than parameters, which is appropriate since there is nothing to document.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states that this tool returns census headline findings as discrete, dated records with value, unit, denominator, date, page, and citation. It distinguishes itself from prose-style reporting, but it does not explicitly differentiate itself from the sibling census_stats, so some overlap remains.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives an explicit 'Use this when...' trigger tied to questions about how open the web is to AI crawlers, and explains why the dated/denominator-preserving format matters. It does not mention alternatives or when not to use it, so routing guidance is strong but not complete.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

census_statsBInspect

Corpus-level statistics: how many measured domains block each AI crawler, mean access score, llms.txt adoption.

ParametersJSON Schema

Name	Required	Description	Default
No parameters

TDQS

B3.1/5.0

Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden of disclosing behavior. It states what data is included but does not reveal whether it is read-only, how recent the statistics are, whether any parameters or filters exist, or what the response structure looks like. The colon-led list of statistics is content, not behavioral transparency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single concise sentence that leads with the core concept ('corpus-level statistics') and then lists specific examples. There is no filler or redundant language, making it easy to parse.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a parameterless tool, the description gives enough to understand what kind of data comes back, but it omits an explicit return format and does not clarify the scope of 'measured domains' or any access prerequisites. The missing output schema means the description could reasonably be expected to say more about the result shape.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has zero parameters, so the schema fully covers any input concerns. The description adds value by explaining what the resulting statistics measure, which complements the empty schema without needing to compensate for undocumented parameters.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names the resource ('corpus-level statistics') and specifies three concrete metrics: per-crawler blocking counts, mean access score, and llms.txt adoption. This gives an agent a clear idea of what the tool computes, though it does not explicitly distinguish itself from the similarly named sibling census_facts.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The phrase 'corpus-level statistics' implies this is for aggregate data, but there is no explicit statement of when to use this tool versus alternatives like census_facts or scan_site. No when-not conditions or alternative names are provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

crawl_preflightAInspect

Decide whether a crawler may fetch a list of domains before spending requests on them. Works for any crawler token, not only the ones this census tracks: an unrecognised agent is resolved from each domain's stored robots.txt rather than refused. For each domain returns one of: allow (robots permits it and a live request carrying that agent's user agent was served), disallow (robots.txt forbids it), refuse (robots permits it but the edge refused the agent anyway, so the allowance is not real), pay (the origin answered HTTP 402 Payment Required, meaning it will serve this agent on commercial terms), or unknown. The full definition of each, including what it obliges a crawler to do, is published at https://crawlcensus.com/api/v1/verdicts. Built for crawler operators rather than site owners: it prevents wasted fetches against doors that are shut, and flags content an operator is trying to sell rather than withhold.

ParametersJSON Schema

Name	Required	Description	Default
`agent`	Yes	Crawler token, e.g. gptbot, claudebot, perplexitybot, oai-searchbot, ccbot.
`domains`	Yes	Domains to check. Up to 25 per call anonymously; send an Authorization: Bearer key for more. An over-large batch is refused outright rather than partly answered.

TDQS

A4.6/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and goes beyond the basics: it reveals that a live request is sent to verify serving, that unrecognised agents are resolved from robots.txt rather than refused, and that 'refuse' means an allowance is not real. It also links to full verdict definitions. Minor omissions like rate-limit behavior are already covered in the schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is dense but every sentence earns its place: purpose, verdict list, unknown-agent behavior, authoritative link, and audience. It is front-loaded with the core decision purpose. Slightly longer than necessary, but no filler or repetition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description is complete despite having no output schema: it enumerates all possible return verdicts with definitions, explains edge-case behavior for unknown agents, and provides a link for full semantics. Combined with a 100%-covered input schema, an agent has all needed information to invoke the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so baseline is 3. The description adds meaningful context for the 'agent' parameter by explaining it works for any crawler token and how unrecognised agents are handled, which goes beyond the schema's list of examples. It also clarifies the 'domains' parameter's per-domain output semantics.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description opens with a specific verb+resource ('Decide whether a crawler may fetch a list of domains'), then enumerates the five possible verdicts with precise meanings. It also distinguishes itself from sibling census/site tools by stating it is 'Built for crawler operators rather than site owners' and by handling 'any crawler token, not only the ones this census tracks.'

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly tells when to use the tool ('before spending requests on them') and for whom ('crawler operators rather than site owners'), which excludes site-owner workflows that would be served by other siblings. It also explains the unrecognised-agent behavior, aiding correct selection for non-census crawlers.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

scan_siteAInspect

Run a live AI-accessibility audit of a domain: robots.txt policy for every tracked AI crawler, live user-agent probes, JavaScript-free readability, structured data and llms.txt. Returns a score out of 100 with per-check detail.

ParametersJSON Schema

Name	Required	Description	Default
`domain`	Yes	Bare hostname, for example example.com

TDQS

A3.8/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of disclosure. It transparently explains that the tool performs live probes, checks robots.txt and llms.txt, and returns a score with per-check detail. It does not mention potential side effects or operational caveats, but the core behavior is well disclosed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single information-dense sentence that front-loads the purpose and uses a compact list to convey scope, then states the return format. No filler or redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with one parameter and no output schema, the description is largely complete: it explains what checks are performed and what the return value looks like (score out of 100 with per-check detail). Minor gaps are the lack of alternative routing and any timing/side-effect context, but nothing critical is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, and the single parameter 'domain' is already well documented with an example. The tool description adds no additional parameter semantics beyond what the schema provides, so the baseline of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states a specific verb ('Run') and resource ('a live AI-accessibility audit of a domain'), and enumerates concrete checks. It does not explicitly distinguish itself from sibling tools like site_report or crawl_preflight, but the scope is specific enough that an agent can infer its function.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies when to use it: when a live AI-accessibility audit of a domain is needed. However, it gives no explicit guidance on when to choose this tool over siblings such as site_report or crawl_preflight, and offers no exclusions or prerequisites.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

site_reportAInspect

Return the most recent stored audit for a domain without triggering a new scan. Faster and free of load on the target site.

ParametersJSON Schema

Name	Required	Description	Default
`domain`	Yes	Bare hostname

TDQS

A4.2/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the burden. It discloses the key behavioral traits: it is a read of stored data, it does not trigger a scan, and it imposes no load on the target. It stops short of describing behavior when no stored audit exists, but the main side-effect profile is clear.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences with no wasted words; the primary action and the no-scan guarantee are front-loaded, followed by a benefit. Every phrase earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple one-parameter read-only report tool with no output schema, the description covers the core contract: what is returned, the absence of a scan, and the speed/load benefit. It does not specify the return format or error behavior, but the tool's simplicity keeps these gaps minor.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema already describes domain as 'Bare hostname' with 100% coverage, so the description adds no additional parameter meaning. Baseline 3 applies because the schema handles parameter documentation.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb ('Return'), resource ('most recent stored audit for a domain'), and explicitly distinguishes itself from a scan by saying it does not trigger a new scan. This makes its purpose clear and differentiates it from the sibling scan_site.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The phrase 'without triggering a new scan' plus 'Faster and free of load' gives clear context for when to use this tool: when a stored report is acceptable and no active scan is desired. It does not explicitly name scan_site as the alternative, but the intended use case is evident.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

submit_domainsAInspect

Queue domains the census has not measured yet so a later crawl_preflight can answer them. This closes the loop crawl_preflight starts: anything it returns as unknown with measurable true is worth submitting, and the reply names any that were already fresh or that this census will never measure, so a caller looping over its own unknowns converges instead of resubmitting the same set. Queueing is a database write rather than a fetch, so the allowance is far higher than scan_site and submitted domains are measured ahead of the ranked backlog.

ParametersJSON Schema

Name	Required	Description	Default
`domains`	Yes	Hostnames to queue. Up to 50 per call anonymously; an over-large batch is refused outright rather than partly queued.

TDQS

A4.5/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description correctly discloses the core side effect: 'Queueing is a database write rather than a fetch.' It also reveals reply behavior, naming already-fresh or never-measured domains, though exact rate limits and idempotency semantics are not fully specified.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three sentences pack the purpose, workflow, and operational distinction without filler. The first sentence front-loads the core action, while the later sentences add necessary context about the crawl_preflight loop and the allowance difference from scan_site.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a one-parameter mutation tool with no annotations and no output schema, the description covers when to call, what to submit, the side effect, and the response's role in convergence. It leaves numeric limits and idempotency guarantees implicit, but enough information exists for correct invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with a clear field description, so the parameter itself is well documented. The tool description adds value by defining eligibility: only domains the census has not measured yet, specifically those flagged by crawl_preflight as unknown with measurable true.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and object: 'Queue domains the census has not measured yet.' It also explicitly ties the tool to crawl_preflight's 'unknown with measurable true' results, distinguishing it from retrieval-focused siblings like scan_site and site_report.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives an explicit selection rule: anything crawl_preflight returns as unknown with measurable true is worth submitting, and it warns that fresh or never-measured domains appear in the reply so a caller can avoid resubmission. It also contrasts the database-write allowance with scan_site, helping an agent choose between them.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

7 tool updatesv1.0.0Sep 4, 2026
- First observedagent_profile
- First observedcensus_facts
- First observedcensus_stats
- First observedcrawl_preflight
- First observedscan_site
- First observedsite_report
- First observedsubmit_domains

Install Server

RayrunBuild, deploy, and operate MCP servers without becoming an infrastructure team.ray.runAd

TDQS

A3.9/5.0

Disambiguation4/5

scan_site and site_report both return audits for a domain, but one is a live scan and the other is a stored result, so they are distinguishable. census_stats and census_facts also overlap somewhat as corpus-level outputs, but one gives direct statistics and the other gives citation-ready findings. The remaining tools each have clearly separate roles.

Naming Consistency4/5

All names are lowercase snake_case and mostly combine a domain noun with a result or action, such as site_report, census_stats, agent_profile, and crawl_preflight. scan_site and submit_domains are verb-first while the others are noun-first, so the pattern is not perfectly uniform, but the style is still readable and predictable.

Tool Count5/5

Seven tools is a well-scoped set for a crawl-census service: each tool covers a distinct consumer need, from live audits and stored reports to crawler preflight decisions, agent profiles, citation-ready facts, and domain submission. No tool is redundant, and the count feels appropriately sized for the domain.

Completeness4/5

The set covers the main workflows well: running and retrieving audits, querying corpus statistics and facts, profiling individual crawlers, deciding whether to fetch domains, and queueing unmeasured domains. Minor gaps include no obvious way to list all tracked agents or remove queued submissions, but these are not core to the documented purpose.

Maintenance

ActivityMaintained

ResponsivenessNo issues

Resources

Need Help?

Related Servers

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

agent-ready-kit A
com.mudko
Agent-readiness scanner (0-5 score), robots.txt + llms.txt generators, managed agent enablement.
Agent Orchestration Browser Automation Developer Tools
262
Desvela ARD Registry — free preflight(domain)A
dev.desvela
What a domain publishes for AI agents: ai-catalog.json, llms.txt, agents.md, robots.txt rules. Free.
Agent Orchestration Search
42
RoloCache A
io.github.rolocache
Independent, trusted, and verified advisories and routing for agentic workflows on the web.
Agent Orchestration Browser Automation
42
Domain & Company Intel MCP A
io.github.selflabbs
Domain intel for AI agents: RDAP registration, DNS, email deliverability, tech stack.
Networking & Infrastructure Email Search
2

Related MCP Servers

unphurl-mcp
Web Scraping Security Search
123Ergo
A
license
A
quality
D
maintenance
URL intelligence for AI agents. One URL in, structured security and data quality signals out across 7 dimensions. 13 tools, risk score 0-100 with 23 configurable weights.
Updated 4 months ago (2026-04-23 14:24 UTC)
16
110
1
MIT
Maango-mcpofficial
Web Scraping AI & Machine Learning Browser Automation
maango-io
A
license
A
quality
C
maintenance
Maango is the pre-flight check for AI agents on the web. Before an agent scrapes, summarises, trains on, or searches a site, it calls Maango and gets back whether the action is allowed for that domain, along with the reason and the policy signals that decided it.
Updated 4 months ago (2026-05-05 23:32 UTC)
7
1
MIT
maxaeo-ai-visibility-mcp
Web Scraping Search
maxaeo
A
license
A
quality
A
maintenance
Enables AI agents to check whether a public website is crawlable, understandable, and ready for AI search workflows through local-only audits of robots.txt, sitemaps, metadata, and llms.txt.
Updated 11 days ago (2026-08-26 07:56 UTC)
3
250
1
MIT
domain-intel-mcp
Search Research & Data Open Data
datakoot
A
license
Not graded
quality
B
maintenance
Domain and company intelligence for AI agents. Enables vetting companies, qualifying leads, and mapping targets from free public data without API keys.
Updated a month ago (2026-08-02 23:30 UTC)
MIT

Latest Blog Posts

Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)
By Om-Shree-0709 on July 25, 2026.
mcp
Agent Identity
OAuth 2.1
Your AI Chatbot Just Exposed Your CEO's Salary to an Intern
By Om-Shree-0709 on July 2, 2026.
Agent Identity
MCP Security
OAuth Delegation
Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)
By Om-Shree-0709 on June 30, 2026.
Agentic Ai
Prompt Injection
WebAssembly

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/taylorsmithgg/crawl-census-client'

If you have feedback or need assistance with the MCP directory API, please join our Discord server

crawl-census-client

MCP server

Install

Use

Split a queue before crawling it

Or just take the file

What a crawl costs the census

Paying, when an origin quotes a price

Two kinds of unknown

Skipping everything that will not serve you

Keeping a deny list current

Verdicts

It degrades, it does not fail

What we publish about your agent

Limits

Tests

Available Tools

Tool Schema Changelog

Maintenance

Resources

Looking for Admin?

Related MCP Connectors

Related MCP Servers

Maango-mcpofficial

Latest Blog Posts

MCP directory API