Ask before you fetch: will this domain serve your crawler, refuse it, or charge it?
What this census measures and publishes about one AI crawler: how often it is disallowed in robots.txt, how often live requests carrying its user agent are refused at the network edge whatever robots.txt says, whether its operator documents it as honouring robots.txt, and where to correct any of that. Intended for the operator of the agent as much as for anyone studying it, so it includes the correction channel and the public page a claim can be disputed against.
| Name | Required | Description | Default |
|---|---|---|---|
| agent | Yes | Crawler token, e.g. gptbot, claudebot, ccbot, google-extended. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full disclosure burden. It does this well by detailing what data is included, that the context is about network-edge behavior regardless of robots.txt, and that correction/dispute channels are part of the result. It still omits minor behavior details like whether data is cached, read-only guarantees, or response shape, but the core behavioral scope is transparent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single dense but understandable passage; it is front-loaded with the core purpose and adds contextual detail about audience and correction channels. It could be tightened into clearer structured list, but no sentence is truly wasted.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a single-parameter tool with no output schema and no annotations, the description does a solid job of painting the result space: coverage, intent, audience, and correction channel. It is not fully complete because it does not describe the output format, time range, or units, but given the simplicity of the tool, the description is adequate.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the agent parameter is already well documented with examples. The description adds context by framing the parameter as representing an AI crawler/user agent, but it does not provide new parameter-level guidance beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description identifies a specific purpose: returning census information about a single AI crawler, including disallow rates, network-edge refusals, documented honoring of robots.txt, and correction channels. It is clear about the resource and intent. However, it does not explicitly distinguish itself from sibling tools like census_facts or census_stats beyond the 'one AI crawler' framing, so it misses full sibling differentiation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives a clear intended audience—both the crawler operator and researchers—which implies when this tool is useful. It does not explicitly state when to prefer a sibling tool over this one, nor does it provide exclusion criteria or alternative routing.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Every headline finding from the census as discrete, dated records rather than prose. Each carries its value, unit, denominator, measurement date, the page it comes from and a ready-made citation line, plus the caveats that apply to all of them. Use this when answering a question about how open the web is to AI crawlers: lifting a percentage out of a rendered page loses the denominator and the date, which is what makes the number wrong when it is repeated.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden of showing behavior. It discloses that output is a set of records rather than prose, and that each record includes value, unit, denominator, measurement date, source page, citation line, and caveats. It does not explicitly state that the operation is read-only or side-effect-free, though that is strongly implied.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and front-loaded with the tool's core behavior, then enumerates the key output fields, and closes with the exact use case and a concrete reason why this tool matters. Every sentence adds meaningful guidance without padding.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema, so the description compensates by describing exactly what the records contain: value, unit, denominator, date, page, citation line, and caveats. With no parameters and a clearly explained use case, the description is complete enough for an agent to call the tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters and 100% schema description coverage, so there is no parameter detail missing. The baseline for a zero-parameter tool is 4, and the description appropriately focuses on what the output contains rather than input semantics.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly conveys that the tool returns every census headline finding as discrete, dated records with a specified field set, and it gives a concrete use case. It lacks an explicit transitive verb like 'retrieve' or 'list' and does not directly differentiate itself from census_stats, so it stops short of a perfect score.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives an explicit trigger condition: use this when answering a question about how open the web is to AI crawlers, and explains why simple percentage-lifting is wrong. It does not explicitly state when not to use it or name a preferred alternative, but the context is sufficiently clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Corpus-level statistics: how many measured domains block each AI crawler, mean access score, llms.txt adoption.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full behavioral burden. It clearly states the output stats, but does not mention read-only semantics, data source, freshness, or output format. This is acceptable for a benign stats endpoint but leaves operational behavior somewhat underexposed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is one short sentence of about 15 words, front-loaded with 'Corpus-level statistics' and immediately lists all meaningful outputs. No filler or repetition; every phrase adds unique value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter, no-output-schema tool, the description covers the primary outputs well, listing blocking counts, mean score, and llms.txt adoption. Minor gaps such as precise return structure, scale/units, and data source would be useful but are not critical given the simplicity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There are zero parameters and schema description coverage is 100%, so the schema fully handles parameter documentation. With no parameters, the baseline of 4 is appropriate and the description needs to add no additional parameter meaning.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states it provides corpus-level statistics and enumerates three specific metrics: blocking counts per AI crawler, mean access score, and llms.txt adoption. This is related to the resource-level context, and 'corpus-level' clearly contrasts with per-domain sibling tools like scan_site, but it does not name the alternatives.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage when aggregate/corpus-level statistics are needed, as opposed to site-specific functions. However, it never says when not to use it or explicitly points to sibling tools for per-domain data, leaving the routing somewhat implicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Decide whether a crawler may fetch a list of domains before spending requests on them. Works for any crawler token, not only the ones this census tracks: an unrecognised agent is resolved from each domain's stored robots.txt rather than refused. For each domain returns one of: allow (robots permits it and a live request carrying that agent's user agent was served), disallow (robots.txt forbids it), refuse (robots permits it but the edge refused the agent anyway, so the allowance is not real), pay (the origin answered HTTP 402 Payment Required, meaning it will serve this agent on commercial terms), or unknown. The full definition of each, including what it obliges a crawler to do, is published at https://crawlcensus.com/api/v1/verdicts. Built for crawler operators rather than site owners: it prevents wasted fetches against doors that are shut, and flags content an operator is trying to sell rather than withhold.
| Name | Required | Description | Default |
|---|---|---|---|
| agent | Yes | Crawler token, e.g. gptbot, claudebot, perplexitybot, oai-searchbot, ccbot. | |
| domains | Yes | Domains to check. Up to 25 per call anonymously; send an Authorization: Bearer key for more. An over-large batch is refused outright rather than partly answered. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden of behavioral disclosure, and it excels: it explains how unrecognized agents are handled, that verdicts are based on real edge behavior, the distinction between 'refuse' and 'disallow', the 402 pay case, and 'unknown'. This is far beyond a generic 'check crawling permission' statement.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is dense but every sentence earns its place: purpose, audience, token behavior, verdict vocabulary, and operational limits. The key purpose is front-loaded, and the external link offloads long-form details without leaving the description hollow.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite having no output schema and no annotations, the description is operationally complete. It specifies the five possible outcomes, what each implies, the core semantics of a 'real' allowance, and the limits of the domains parameter. An agent has enough to call this tool correctly in most realistic crawler-checking workflows.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is complete, and the description still adds real parameter-level value: examples of valid agent tokens, behavior for unrecognized tokens, per-call batch limits, the anonymous versus authorized key distinction, and rejection of over-large batches. This materially improves the agent's ability to call the tool correctly.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource ('Decide whether a crawler may fetch a list of domains before spending requests on them') and immediately distinguishes who the tool is for ('Built for crawler operators rather than site owners'). It is not a tautology, and an agent can tell it apart from adjacent tooling without opening the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives clear usage context: use it before spending requests, and it is aimed at crawler operators rather than site owners. It does not explicitly name sibling tools as better alternatives for other use cases, but the intended audience and trigger are clearly stated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run a live AI-accessibility audit of a domain: robots.txt policy for every tracked AI crawler, live user-agent probes, JavaScript-free readability, structured data and llms.txt. Returns a score out of 100 with per-check detail.
| Name | Required | Description | Default |
|---|---|---|---|
| domain | Yes | Bare hostname, for example example.com |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations supplied, the description carries the burden of disclosing behavior, and it does so well: it says the tool sends live user-agent probes, reads robots.txt policy for AI crawlers, and returns a 0-100 score with per-check detail. It doesn't spell out timing or failure behavior, but the non-destructive audit nature is reasonably clear.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is one tight, front-loaded sentence: purpose first, audit checks second, output shape last. Every clause contributes information and no schema detail is unnecessarily repeated.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a one-parameter tool without an output schema, the description adequately covers the input, the overall audit checks, and the result format. The only minor gaps are behavior for unreachable/no-llms.txt domains and potential timeouts, which do not hinder correct tool selection.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already fully documents the only required parameter, domain, including the example 'example.com' and the bare-hostname constraint. Since schema coverage is 100%, the description gains the baseline score of 3 for parameter semantics.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a concrete operation — 'run a live AI-accessibility audit of a domain' — and specifies the exact checks involved: robots.txt policy, live user-agent probes, JavaScript-free readability, structured data, and llms.txt. This is specific enough to distinguish scan_site from siblings like site_report, crawl_preflight, or census_stats.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The first clause clearly identifies the use case: choosing this tool when a live AI-accessibility audit of a domain is needed. However, it does not explicitly name alternatives or state when not to use it, so it earns a 4 rather than a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Return the most recent stored audit for a domain without triggering a new scan. Faster and free of load on the target site.
| Name | Required | Description | Default |
|---|---|---|---|
| domain | Yes | Bare hostname |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It transparently discloses the key behavioral trait: it does not cause a new scan and imposes no load on the target site. It could mention missing or stale audit behavior, but the main side-effect and safety risk (accidental scanning) is directly addressed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two short sentences with no wasted words. The core return action is front-loaded, and the key differentiator—no new scan, faster, no target load—is stated efficiently.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a one-parameter tool with no output schema and no annotations, the description covers the essential facts: what is returned, the domain scope, and the side-effect-free behavior. It could mention what happens if no stored audit exists, but the invocation-relevant context is largely complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The only parameter, domain, is already fully documented in the schema as 'Bare hostname' (100% schema description coverage). The description adds no further parameter-level semantics, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb and resource: 'Return the most recent stored audit for a domain.' It clearly distinguishes itself from scan_site by explicitly saying it does not trigger a new scan, so the purpose is unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
'Without triggering a new scan' gives clear context that this is the cached/report tool, with speed and reduced load as rationale. It does not explicitly name scan_site or say when not to use it, but the context strongly implies the correct choice versus a fresh scan.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Queue domains the census has not measured yet so a later crawl_preflight can answer them. This closes the loop crawl_preflight starts: anything it returns as unknown with measurable true is worth submitting, and the reply names any that were already fresh or that this census will never measure, so a caller looping over its own unknowns converges instead of resubmitting the same set. Queueing is a database write rather than a fetch, so the allowance is far higher than scan_site and submitted domains are measured ahead of the ranked backlog.
| Name | Required | Description | Default |
|---|---|---|---|
| domains | Yes | Hostnames to queue. Up to 50 per call anonymously; an over-large batch is refused outright rather than partly queued. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the behavioral burden. It explicitly discloses that queueing is a database write rather than a fetch, that submitted domains are measured ahead of the backlog, and that the reply identifies already-fresh or unmeasurable entries. It could additionally disclose idempotency/authentication requirements, but the mutating behavior is clear and not hidden.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
All three sentences carry information: operation, loop integration, and behavior/allowance. There is no fluff. However, the middle sentence is fairly dense and could be parsed more easily if split; this keeps it from a perfect score.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description is complete enough for a queueing tool that depends on crawl_preflight: it explains how to select domains, what happens to the queue, and what the reply contains. It stops short of fully decoding the reply structure, but given the context, it provides a coherent workflow.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3. The description adds semantic guidance beyond the schema: it clarifies that the domains should be returned by crawl_preflight, only unknown+measurable ones should be queued, and that resubmission of stale names should be avoided. This context helps select and filter correct values.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a concrete verb and resource: 'Queue domains the census has not measured yet'. It also states the high-level purpose ('so a later crawl_preflight can answer them') and positions the tool by explaining how it closes the loop that crawl_preflight starts. It clearly distinguishes this from siblings like scan_site or crawl_preflight.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives explicit when-to-use direction: domains that crawl_preflight returns as unknown with measurable true are the right candidates to submit. It also warns against resubmitting already-fresh or unmeasurable domains, and explains that the allowance is higher than scan_site, making it the right tool for bulk queueing. This is a strong, actionable usage guide.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
submit_domainsagent_profilecensus_factscensus_statscrawl_preflightscan_sitesite_reportClaiming proves that you control a remote MCP connector. It does not move, proxy, or interrupt the server.
Open the connector listing, choose Claim ownership, and sign in to Glama.
Complete one verification method:
GitHub identity — fastest for official registry listings. For a namespace such as io.github.alice/server, link the matching GitHub user, then choose Claim with GitHub. An organization namespace such as io.github.acme/server also needs that organization to have installed the Glama AI GitHub App and approved its permissions, because GitHub discloses organization membership only to apps it has installed. Use HTTP or DNS when it has not.
HTTP challenge — works when you can deploy a public file. Generate a token, publish the exact JSON Glama shows at /.well-known/glama.json on the same origin as the connector, then choose Check HTTP challenge.
DNS challenge — works when you control DNS but cannot change the server. Generate a token, create the exact TXT record Glama shows, wait for it to propagate, then choose Check DNS challenge.
After verification, Glama sends a confirmation email and gives you access to listing details, thumbnails, health checks, and analytics. Keep the HTTP file or DNS record in place: Glama periodically checks it and ownership remains verified while the token is discoverable.
The HTTP ownership file has this structure:
{
"$schema": "https://glama.ai/mcp/schemas/connector.json",
"claim": "glama_claim_..."
}Claim tokens are opaque, stable, and bound to the signed-in Glama account. They contain no email address or other personal information. If Glama can no longer discover a verified HTTP or DNS token, it starts a seven-day grace period before removing claim-based access. Restore the same token during that period to keep ownership verified. Never publish an email address, Glama session token, GitHub token, or connector credential as ownership proof.
If verification fails, confirm that you copied the current token exactly. The HTTP file must be public, return valid JSON with a successful HTTP response, and stay on the connector's origin. DNS changes may need more time to propagate. A claim cannot transfer to a different origin or hostname: if the connector target changes, Glama starts the grace period and the new target must be claimed separately after the previous claim is released.
For a connector linked to the official MCP Registry, registry updates continue to replace its name, description, and URL by default. After claiming, open Manage connector and enable Use Glama listing details as the source of truth if edits made on Glama should be preserved. Categories and thumbnails are always managed on Glama; registry linkage and technical connection settings continue to sync.
Control your server's listing on Glama, including description and metadata
Access analytics and receive server usage reports
Get monitoring and health status updates for your server
Feature your server to boost visibility and reach more users
To improve your MCP server's ranking:
Claim ownership of the server listing
Complete the server profile with an accurate description and thumbnail
Provide a test profile so Glama can connect to and evaluate the server
Keep tool definitions clear and complete to earn a high Tool Definition Quality Score (TDQS)
Route real usage through the Glama Gateway; more recorded successful server uses also improve the ranking
For users:
Full audit trail – every tool call is logged with inputs and outputs for compliance and debugging
Granular tool control – enable or disable individual tools per connector to limit what your AI agents can do
Centralized credential management – store and rotate API keys and OAuth tokens in one place
Change alerts – get notified when a connector changes its schema, adds or removes tools, or updates tool definitions, so nothing breaks silently
For server owners:
Proven adoption – public usage metrics on your listing show real-world traction and build trust with prospective users
Tool-level analytics – see which tools are being used most, helping you prioritize development and documentation
Direct user feedback – users can report issues and suggest improvements through the listing, giving you a channel you would not have otherwise
The connector status is unhealthy when Glama is unable to successfully connect to the server. This can happen for several reasons:
The server is experiencing an outage
The URL of the server is wrong
Credentials required to access the server are missing or invalid
If you are the owner of this MCP connector and would like to make modifications to the listing, including providing test credentials for accessing the server, please contact support@glama.ai.
No comments yet. Be the first to start the discussion!
Checks if a URL path is allowed by robots.txt. x402 payment required (testnet USDC).
Robots-aware website audits, pre-crawl quotes, and structured evidence for people and agents.
What a domain publishes for AI agents: ai-catalog.json, llms.txt, agents.md, robots.txt rules. Free.
Agent-readiness scanner (0-5 score), robots.txt + llms.txt generators, managed agent enablement.
Add one secure layer between your agents and this server.
census_facts and census_stats are easy to conflate, and crawl_preflight, scan_site, and site_report overlap in that they each inspect a domain's crawler accessibility. The descriptions eventually separate them by output intent — verdicts vs audit vs stored report — but an agent must read carefully to avoid a wrong call.
All names use snake_case and are readable, but there is no consistent verb-noun rule: scan_site and submit_domains start with verbs, while census_facts, census_stats, agent_profle, crawl_preflight, and site_report are noun-like phrases. This mixed convention is understandable, though it misses the clarity of a uniform pattern.
Seven tools are well-scoped for a census-focused server: they cover agent lookups, corpus statistics, domain verdicts, audits, stored reports, and domain submission. None feels like filler, and the count stays small enough for an agent to navigate.
The core read, audit, report, and submission loop is covered: preflight verdicts, full scanned audits, cached reports, and bulk domain queueing. The only minor gap is a lack of tools to update or delete domain submissions or cached reports, but this does not seriously affect the intended census and crawler workflows.
