Detect Duplicate Content
detect_duplicate_contentAnalyze titles, meta descriptions, headings, and body text to identify duplicate and near-duplicate pages from sitemaps, crawls, or custom URL lists.
Instructions
Detect duplicate and near-duplicate content across a site's pages. Analyzes titles, meta descriptions, H1 headings, and text content. Groups exact duplicates and identifies near-duplicates based on word similarity. Supports sitemap, crawl, or custom URL list as page source.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The site URL to analyze | |
| urls | No | List of URLs to analyze (only used when source = 'urls') | |
| delay | No | Delay in ms between batches | |
| limit | No | Max pages to analyze | |
| source | No | Source of URLs: sitemap, crawl, or a provided list | crawl |
| concurrency | No | Pages fetched in parallel | |
| similarityThreshold | No | Similarity threshold (0-1) above which pages are considered near-duplicates |