robots.txt / llms.txt
v1_robotsrobots.txt / llms.txt: Fetch and parse a site's robots.txt and llms.txt. Source: direct fetch. $0.005 per call · GET /v1/robots
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Any URL on the target site. Example: 'https://www.cloudflare.com'. |