Crawl Site
crawl_siteRecursively crawl a website from a starting URL to build a list of discovered pages with titles and HTTP status codes. Set crawl depth, page limits, concurrency, and URL filters to control scope and impact.
Instructions
Crawl a website recursively starting from a URL. Follows internal links up to a specified depth and max pages. Returns a list of discovered pages with their titles and status codes.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The starting URL to crawl | |
| delay | No | Base delay in ms between requests. A random jitter of ±30% is applied automatically (default: 300ms) | |
| maxDepth | No | Maximum crawl depth (0 = only the starting page) | |
| maxPages | No | Maximum number of pages to crawl | |
| concurrency | No | Number of pages to fetch in parallel (default: 3). Lower values are safer for small sites | |
| excludePattern | No | Regex pattern: skip URLs matching this pattern | |
| includePattern | No | Regex pattern: only crawl URLs matching this pattern | |
| respectRobotsTxt | No | Respect robots.txt rules and Crawl-delay (default: true) |