Provides tools to analyze and manage robots.txt rules for Amazonbot, enabling control over how Amazon's AI crawlers access website content.
Provides tools to analyze and manage robots.txt rules for Baiduspider, enabling control over how Baidu's search crawlers access website content.
Provides tools to analyze and manage robots.txt rules for Googlebot and Google-Extended, enabling control over how Google's search and AI agents crawl website content.
Provides tools to analyze and manage robots.txt rules for Meta-ExternalAgent, enabling control over how Meta's AI agents crawl website content.
Provides tools to analyze and manage robots.txt rules for PerplexityBot, enabling control over how Perplexity's AI search agents crawl website content.
Manage which AI bots can crawl your website — visually.
Toggle GPTBot, ClaudeBot, PerplexityBot, and 20+ AI crawlers on/off with a simple UI. Analyze any site's robots.txt instantly.
Visual Toggle UI — Block or allow AI bots with simple on/off switches
20+ AI Bots Database — GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, cohere-ai, Amazonbot, Meta-ExternalAgent, and more
Analyze Existing robots.txt — Paste or fetch any robots.txt to see which AI bots are blocked
Generate robots.txt — Create a complete robots.txt with your chosen rules
MCP Server — Use with Claude Desktop, Cursor, or any MCP-compatible AI assistant
Check Bot Status — Verify if a specific bot is blocked on any website
Related MCP server: seo-audit-mcp
Tool | Description |
| Fetch and analyze a robots.txt from any URL |
| Analyze pasted robots.txt content for AI bot blocking status |
| Generate a robots.txt with specified blocked bots and custom rules |
| List all known AI bots with user-agents, companies, and descriptions |
| Check if a specific bot is blocked on a given website |
git clone https://github.com/sharozdawa/robotstxt-ai.git
cd robotstxt-ai
npm install
npm run devOpen http://localhost:3000 in your browser.
Add to your claude_desktop_config.json:
{
"mcpServers": {
"robotstxt-ai": {
"command": "npx",
"args": ["-y", "robotstxt-ai-mcp"]
}
}
}Add to .cursor/mcp.json:
{
"mcpServers": {
"robotstxt-ai": {
"command": "npx",
"args": ["-y", "robotstxt-ai-mcp"]
}
}
}The server knows about 25+ bots including:
AI Crawlers: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, cohere-ai, Amazonbot, Meta-ExternalAgent
AI Search: ChatGPT-User, OAI-SearchBot, PerplexityBot, YouBot
Search Engines: Googlebot, Bingbot, YandexBot, Baiduspider, DuckDuckBot
Feature | robotstxt.ai | Manual Editing |
Visual toggle UI | Yes | No |
20+ AI bots database | Yes | Research yourself |
Analyze existing robots.txt | Yes | No |
MCP Server | Yes | No |
Price | Free | Free but tedious |
Tool | Description |
Curated list of SEO MCP servers and agent skills | |
Instant URL indexing via IndexNow | |
Schema.org JSON-LD markup generator | |
AI brand visibility tracker |
MIT
Built by Sharoz Dawa — SEO Professional & Digital Marketing Expert
Analyze pasted robots.txt content. Returns which AI bots are blocked or allowed based on the rules.
| Name | Required | Description | Default |
|---|---|---|---|
| content | Yes | The robots.txt content to analyze |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It states the tool returns which AI bots are blocked/allowed, but does not disclose error handling, input format validation, or any side effects. For a read-only analysis tool, this is minimally adequate.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise with two sentences. It front-loads the action and resource, and every sentence provides essential information. There is no wasted text.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the simplicity of the tool (one parameter, no output schema), the description covers what it does and what it returns. It could mention the output format or limitations, but for a low-complexity tool it is largely complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and the parameter description ('The robots.txt content to analyze') is clear. The description adds 'pasted' which slightly clarifies expected input but does not provide significant additional meaning beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description uses a specific verb ('Analyze') and resource ('pasted robots.txt content'). It clearly distinguishes from sibling tools like 'fetch_robots' (retrieves content) and 'generate_robots' (creates content), making its unique function evident.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies using this tool when you have robots.txt content to analyze, but it does not explicitly state when to use it versus alternatives like checking a specific bot status or fetching content first. No exclusions or alternative tool names are mentioned.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Check if a specific bot is blocked or allowed on a given website by fetching and analyzing its robots.txt.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The website URL to check (e.g. https://example.com) | |
| bot_name | Yes | The bot user-agent string to check (e.g. 'GPTBot', 'ClaudeBot'). Use list_ai_bots to see available names. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations, so description fully relies on text. It discloses the method (robots.txt analysis) but doesn't state non-destructive nature, rate limits, or error handling. Adequate but not comprehensive.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Single, front-loaded sentence with no waste. Every word serves the purpose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema, so description should hint at return values. It only says 'check if blocked or allowed' without describing the result format. Sibling tools exist but no guidance on selection.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%. Description adds context (e.g., 'website URL', 'bot user-agent string') and suggests using list_ai_bots, adding value beyond schema field descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool checks bot status via robots.txt, with specific verb (check) and resource (bot on website). It distinguishes from siblings like list_ai_bots, which lists bot names.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Implied usage from required parameters, but no explicit when-to-use vs alternatives or exclusion criteria. It hints at using list_ai_bots for bot names, but no direct comparison with analyze_robots or fetch_robots.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Fetch and analyze a robots.txt file from a URL. Returns which AI bots are blocked or allowed.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The website URL to fetch robots.txt from (e.g. https://example.com) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must disclose behavioral traits. It only says 'Fetch and analyze' without detailing network behavior (e.g., redirects, timeouts, error handling) or the output format. This leaves significant uncertainty for an agent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, efficiently covering purpose and return. It is not verbose, but could be more structured (e.g., bullet points) for clarity. Still, it earns a high score for brevity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool is simple (1 param, no output schema), the description covers core purpose and return. However, it lacks details on error conditions (e.g., missing robots.txt) and does not explain how analysis works, which with siblings like 'analyze_robots' creates ambiguity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with one parameter 'url' described as 'The website URL to fetch robots.txt from'. The description repeats this purpose but adds no new semantics like expected format restrictions or handling of fragments. It meets the baseline but adds no extra value.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool fetches and analyzes a robots.txt file from a URL and returns which AI bots are blocked or allowed. It differentiates from siblings like 'analyze_robots' (which likely works on already fetched content) and 'generate_robots' (which creates files).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description does not explicitly state when to use this tool versus alternatives like 'analyze_robots' or 'check_bot_status'. It implies usage for fetching from a URL but provides no exclusions or context for choice.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Generate a robots.txt file with specified blocked bots, sitemap URLs, and custom rules.
| Name | Required | Description | Default |
|---|---|---|---|
| block_all_ai | No | If true, blocks all known AI crawlers (but not search engines like Googlebot) | |
| blocked_bots | No | Array of bot user-agent strings to block (e.g. ['GPTBot', 'ClaudeBot', 'CCBot']). Use list_ai_bots to see available user-agents. | |
| custom_rules | No | Additional custom robots.txt rules to append (raw robots.txt syntax) | |
| sitemap_urls | No | Array of sitemap URLs to include (e.g. ['https://example.com/sitemap.xml']) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It only says 'Generate a robots.txt file' without disclosing side effects, authorization needs, or output behavior (e.g., file creation vs. returning content).
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single concise sentence listing key elements. It is front-loaded and efficient, though it could benefit from structured formatting for clarity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a generation tool with 4 optional parameters and no output schema, the description lacks critical details: default behavior when no parameters provided, output format (file vs. string), and prerequisites (e.g., domain context).
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, but the description adds meaningful context: explains block_all_ai excludes Googlebot, references list_ai_bots for blocked_bots, and clarifies custom_rules uses raw syntax. This goes beyond names and schema descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool generates a robots.txt file with specified blocked bots, sitemaps, and custom rules. It distinguishes from siblings like analyze_robots and list_ai_bots which have different purposes.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus siblings. It only briefly mentions using list_ai_bots for available user-agents within a single parameter description, but lacks explicit when-to-use or when-not-to-use context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
List all known AI bots with their user-agents, companies, and descriptions. Useful for deciding which bots to block.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the burden. It indicates a read-only listing operation, but does not explicitly state safety, idempotency, or potential side effects. The name and context imply no destructive behavior, but more clarity could be provided.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description consists of two short sentences: the first clearly states the purpose and result, and the second gives a usage hint. No superfluous words, well-structured.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given zero parameters, no output schema, and a simple listing task, the description is fully sufficient. It explains what the tool returns (user-agents, companies, descriptions) and its usefulness. No gaps remain for an agent to invoke it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has no parameters and the schema description coverage is 100%. The description adds no parameter details because none are needed. The baseline for zero parameters is 4, and the description fulfills that by not requiring additional explanation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool lists all known AI bots with specific fields (user-agents, companies, descriptions), using a specific verb and resource. It distinguishes from sibling tools like analyze_robots or generate_robots, which have different purposes.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides context for when to use the tool ('Useful for deciding which bots to block'), implying a decision-making scenario. Though it does not explicitly mention alternatives or when-not-to-use, the intended usage is clear given sibling tool names.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
v1.0.0analyze_robotscheck_bot_statusfetch_robotsgenerate_robotslist_ai_botsEach tool targets a distinct operation: analyzing pasted content, checking a specific bot, fetching from a URL, generating, and listing bots. No overlap in functionality.
All tool names follow a consistent verb_noun pattern with snake_case (e.g., analyze_robots, check_bot_status). No mixing of conventions.
Five tools is a well-scoped set for this domain, covering the essential tasks without being too few or excessive.
The tools cover the full lifecycle: fetching, analyzing, checking specific bots, generating, and listing known AI bots. No obvious gaps for the intended purpose.
Unclaimed servers have limited discoverability.
If you are the server author, to access and configure the admin panel.
Checks llms.txt, AI crawler access in robots.txt, and sitemap - with a 0-100 AI readiness score.
Agent-readiness scanner (0-5 score), robots.txt + llms.txt generators, managed agent enablement.
Generate 18 AI readiness files (llms.txt, ai.txt, RAG indexes, schema) for any website.
Checks whether a website is readable and citable by AI systems (ChatGPT, Claude, Perplexity, etc.)
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/sharozdawa/robotstxt-ai'
If you have feedback or need assistance with the MCP directory API, please join our Discord server
