Verify Citations
verify_citationsVerify citations in an output by fetching sources and using an LLM judge to confirm each source supports its claim, returning per-citation verdicts and an overall score.
Instructions
Extract the citations in an output, fetch the sources (opt-in, SSRF-guarded) and ask an LLM judge on your key whether each source supports its claim.
What it does. Three phases. Extraction, no network: [N] references, (Author, Year), bare URLs and DOIs. Fetch of URL and DOI citations only when allow_fetch is true or IRIS_CITATION_ALLOW_FETCH=1, through a scheme allowlist, private and cloud-metadata address blocking, an optional hostname allowlist (domain_allowlist, merged with IRIS_CITATION_DOMAINS), a per-source timeout and byte cap, and at most three re-checked redirects. Then one judge call per resolved citation on your own key, reading the first part of each source, capped in total by max_cost_usd_total. Up to max_citations are verified; extras are skipped, not errored. overall_score is supported / judged and null when nothing was judged. Per-citation failures (bad scheme, blocked address, timeout, too large, cost cap, fetch disabled) are reported on the citation, never scored as unsupported. One evaluation row is stored.
When not to use it. When the output has no citations: the score is null, and evaluate_output's hallucination signals are the cheap check. Without a key (IRIS_ANTHROPIC_API_KEY or IRIS_OPENAI_API_KEY): the call returns IRIS_JUDGE_NOT_ENABLED with the enable steps. With fetch enabled and an open allowlist on untrusted output: you are running a user-directed fetcher — set IRIS_CITATION_DOMAINS.
Returns. JSON with id (the evaluation id; read it back at iris://evaluations/{id}); trace_id (the linked trace, when one was named); overall_score (supported / judged; null when nothing was judged); passed (true when every judged citation was supported; false when any judged citation was not; NULL when nothing was judged — no verdict, because nothing was verified. Until 0.10.0 that last case returned true.); total_unsupported (judged citations the judge ruled unsupported — the number the verdict turns on); total_citations_found (citations extracted from the output); total_resolved (citations whose source was fetched); total_judged (citations the judge ruled on); total_supported (citations the judge found supported); total_cost_usd (the spend across every judge call); citations (per citation: the citation (raw, kind, identifier, offsets), resolve_status ok | skipped | error, resolve_error, source (url, status, content_type, bytes_fetched, truncated), judge (supported, confidence, rationale, cost_usd, latency_ms, tokens)).
Errors. IRIS_JUDGE_NOT_ENABLED, IRIS_JUDGE_UNKNOWN_MODEL and IRIS_UNKNOWN_TRACE before any fetch or spend. IRIS_JUDGE_FAILED when citations resolved but the judge failed on every one — an error, not a passing verdict; nothing is stored. Every failure returns {"error":{"code","message","recovery":[]}} with isError true; follow recovery before retrying.
Siblings. evaluate_with_llm_judge — general semantic scoring; evaluate_output — the free deterministic path, including the hallucination signals; log_trace — record the execution first.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| model | Yes | Judge model for per-citation verification. Supported: anthropic = claude-opus-4-7 | claude-sonnet-4-6 | claude-haiku-4-5 | claude-haiku-4-5-20251001; openai = gpt-4o | gpt-4o-mini | o1-mini. | |
| output | Yes | The agent output containing citations to verify | |
| provider | No | Auto-detected from model when omitted | |
| trace_id | No | Link verification result to a stored trace (id from log_trace / get_traces); an unknown id is rejected before any fetch or judge call | |
| allow_fetch | No | Permit outbound HTTP to resolve URLs/DOIs. Defaults to IRIS_CITATION_ALLOW_FETCH=1; false otherwise. SSRF-guarded regardless. | |
| max_citations | No | Max citations to verify (extras skipped, not errored); default 20, at most 50 | |
| domain_allowlist | No | Restrict fetches to hostnames in this list (suffix match allowed). Merged with IRIS_CITATION_DOMAINS env. | |
| max_cost_usd_total | No | Cap TOTAL judge cost across all citations in this call; default 1.00 USD — the pipeline stops when the next call would exceed it | |
| per_source_max_bytes | No | Per-URL body cap; default 5MB | |
| per_source_timeout_ms | No | Per-URL fetch timeout; default 10_000 |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| id | Yes | the evaluation id; read it back at iris://evaluations/{id} | |
| passed | Yes | true when every judged citation was supported; false when any judged citation was not; NULL when nothing was judged — no verdict, because nothing was verified. Until 0.10.0 that last case returned true. | |
| trace_id | No | the linked trace, when one was named | |
| citations | Yes | per citation: the citation (raw, kind, identifier, offsets), resolve_status ok | skipped | error, resolve_error, source (url, status, content_type, bytes_fetched, truncated), judge (supported, confidence, rationale, cost_usd, latency_ms, tokens) | |
| total_judged | Yes | citations the judge ruled on | |
| overall_score | Yes | supported / judged; null when nothing was judged | |
| total_cost_usd | Yes | the spend across every judge call | |
| total_resolved | Yes | citations whose source was fetched | |
| total_supported | Yes | citations the judge found supported | |
| total_unsupported | Yes | judged citations the judge ruled unsupported — the number the verdict turns on | |
| total_citations_found | Yes | citations extracted from the output |