Evaluate With LLM Judge
evaluate_with_llm_judgeScore an output with an LLM judge on your key: get a 0–1 score, rationale, sub-scores, and exact cost. Templates cover accuracy, helpfulness, safety, correctness, and faithfulness.
Instructions
Score an output with an LLM judge on your own provider key: a 0..1 score, a rationale, per-dimension sub-scores and the exact spend.
What it does. Calls Anthropic or OpenAI directly with the key in this process's environment (IRIS_ANTHROPIC_API_KEY or IRIS_OPENAI_API_KEY); Iris never proxies. template picks the question: accuracy, helpfulness, safety, correctness (needs expected) or faithfulness (needs source_material); input improves helpfulness and safety. model is required; provider is inferred from it. The worst-case spend — both attempts, full max_output_tokens — is computed BEFORE the call and refused if it exceeds max_cost_usd (default IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL or 0.25). temperature defaults to 0; a rate-limited call is retried once. One evaluation row is stored with the provider response id, tokens, cost and latency, linked to trace_id when given. The judge's own accuracy is measurable on a key you supply and is not yet published (see iris://proof).
When not to use it. For length, keyword, PII, injection or cost checks: evaluate_output is free and deterministic. Without a key: the call returns IRIS_JUDGE_NOT_ENABLED with the enable steps — do not search for them. On very large outputs without raising max_cost_usd: the pre-check refuses.
Returns. JSON with id (the evaluation id; read it back at iris://evaluations/{id}); trace_id (the linked trace, when one was named); score (0..1 from the judge); passed (the verdict: the score against the template's threshold, which is pass_threshold below. Not the model's own boolean — that is self_reported_pass); pass_threshold (the threshold the score was read against, so you can check the arithmetic); self_reported_pass (what the model said about passing, when it said anything. Recorded, never obeyed); disagreement (true when the model's own boolean disagrees with the threshold verdict — its rubric and its judgement have come apart on this output); rationale (the judge's reasoning, in its words); dimensions (per-dimension sub-scores for the template); model (the model that judged); provider (the provider called); template (the template used); input_tokens (tokens sent, across both attempts when a retry ran); output_tokens (tokens received, across both attempts when a retry ran); cost_usd (the exact spend from the pricing table); latency_ms (wall time of the provider call(s)); raw_response_id (the provider's response id, for your own audit).
Errors. IRIS_JUDGE_NOT_ENABLED (no key for the provider reached this process; recovery carries the steps). IRIS_JUDGE_UNKNOWN_MODEL (valid lists the models). IRIS_UNKNOWN_TRACE, checked before any spend. IRIS_BUDGET_EXCEEDED (nothing spent; the message carries both numbers). IRIS_PROVIDER_ERROR with kind auth, rate_limit, bad_request, server_error, timeout or malformed_response, and retryable set. Every failure returns {"error":{"code","message","recovery":[]}} with isError true; follow recovery before retrying.
Siblings. evaluate_output — the free deterministic path; verify_citations — citation grounding, the narrower judge; log_trace — record the execution first.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| input | No | User question / prompt that produced the output (improves accuracy for helpfulness/safety) | |
| model | Yes | Model ID. Supported: anthropic = claude-opus-4-7 | claude-sonnet-4-6 | claude-haiku-4-5 | claude-haiku-4-5-20251001; openai = gpt-4o | gpt-4o-mini | o1-mini. Required — cost varies a hundredfold across models | |
| output | Yes | The agent output text to evaluate | |
| expected | No | Reference answer (required for correctness template) | |
| provider | No | Auto-detected from model when omitted | |
| template | Yes | Judge dimension: accuracy (factual correctness), helpfulness (does it address the ask), safety (harm potential), correctness (vs reference answer — requires `expected`), faithfulness (RAG grounding — requires `source_material`). | |
| trace_id | No | Link this evaluation to a stored trace (id from log_trace / get_traces); an unknown id is rejected BEFORE the judge is called | |
| timeout_ms | No | Per-request timeout; default 60_000 | |
| temperature | No | Sampling temperature; default 0 (deterministic) | |
| max_cost_usd | No | Cost cap in USD for this call; defaults to IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL or 0.25. The worst case (two attempts, full max_output_tokens) is computed before the call and refused if it exceeds the cap | |
| source_material | No | Provided RAG sources (required for faithfulness template) | |
| max_output_tokens | No | Judge output token cap; default 512 |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| id | Yes | the evaluation id; read it back at iris://evaluations/{id} | |
| model | Yes | the model that judged | |
| score | Yes | 0..1 from the judge | |
| passed | Yes | the verdict: the score against the template's threshold, which is pass_threshold below. Not the model's own boolean — that is self_reported_pass | |
| cost_usd | Yes | the exact spend from the pricing table | |
| provider | Yes | the provider called | |
| template | Yes | the template used | |
| trace_id | No | the linked trace, when one was named | |
| rationale | Yes | the judge's reasoning, in its words | |
| dimensions | Yes | per-dimension sub-scores for the template | |
| latency_ms | Yes | wall time of the provider call(s) | |
| disagreement | No | true when the model's own boolean disagrees with the threshold verdict — its rubric and its judgement have come apart on this output | |
| input_tokens | Yes | tokens sent, across both attempts when a retry ran | |
| output_tokens | Yes | tokens received, across both attempts when a retry ran | |
| pass_threshold | Yes | the threshold the score was read against, so you can check the arithmetic | |
| raw_response_id | No | the provider's response id, for your own audit | |
| self_reported_pass | No | what the model said about passing, when it said anything. Recorded, never obeyed |