Evaluate Output
evaluate_outputScore agent outputs against deterministic rule bundles (completeness, relevance, safety, cost, custom) and get a verdict with rule-level evidence, uncertainty, and skipped rules.
Instructions
Score an agent output against the deterministic rule bundles: the ship verdict with its basis, every rule result with evidence and uncertainty, and what was not judged.
What it does. In-process, no network, no key. eval_type picks one bundle (completeness, relevance, safety, cost, custom) or all (the default): every bundle plus deployed and inline custom rules, with a per-bundle breakdown in categories. Inputs decide what can be judged: input is REQUIRED when eval_type="relevance" (keyword_overlap and topic_consistency compare the output against it and skip without it) and grounds the hallucination signals; tool_calls, or a trace_id whose stored tool_calls are reused, feed the trajectory rules; cost_usd and token_usage feed the cost rules; expected feeds only expected_coverage. A rule without its input SKIPS, is named, and never counts as a pass. custom_rules always fire. One row is stored, linked to trace_id when given.
When not to use it. To validate arbitrary JSON Schema (the json_schema custom type asserts an output's shape only). To screen inputs before they reach an agent: no_injection_patterns inspects the agent's OUTPUT text for injection-shaped content — attack phrasing and structural directives the output echoes or complies with — and never reads the input, so it is not an input firewall. For semantic judgment, evaluate_with_llm_judge and verify_citations need a key you supply.
Returns. JSON with id (the evaluation id, readable at iris://evaluations/{id}); trace_id (the linked trace, when named); verdict (state, passed, basis (which layer decided), by (the rules), risk); coverage (per question: judged, unjudged and why, or not_applicable; plus the inputs carried); provenance (Iris version, ruleset and config hashes, thresholds, corpus version, time); erased_at (set once the linked trace was deleted); eval_type (the bundle that ran); score (0..1 weighted quality over the rules that ran); passed (the ship verdict; false when nothing was judged); rule_results (per rule: verdict, message, kind, role, question, saw, evidence, uncertainty); suggestions (what to change); rules_evaluated (rules that judged); rules_skipped (rules that skipped); insufficient_data (true when no rule could judge); critical_failures (critical rules that failed and vetoed passed); critical_skipped (critical rules that could not judge; treat as unknown); categories (per-bundle verdicts for eval_type all); note (present when eval_type was omitted).
Errors. IRIS_UNKNOWN_TRACE when trace_id names no stored trace — checked first, nothing scored or written. IRIS_STORAGE_ERROR when the row cannot be written. Unknown arguments or keys are refused before the handler runs, naming the valid ones; a regex rule over its budget or with a broken config reports skipped, not an error. Every failure returns {"error":{"code","message","recovery":[]}} with isError true; follow recovery before retrying.
Siblings. log_trace — record the execution first; evaluate_with_llm_judge — semantic scoring on your key; verify_citations — citation grounding on your key; list_rules — the roster, needs and published accuracy.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| input | No | Original input for context (the ask + any source material the agent was given) — REQUIRED when eval_type="relevance" (keyword_overlap and topic_consistency compare the output against it and skip without it); also grounds the safety bundle's hallucination signals | |
| output | Yes | The output text to evaluate (the agent's response that gets scored against rules) | |
| cost_usd | No | Cost in USD — consulted by the cost bundle (eval_type="cost" or "all") AND by any cost_threshold custom rule regardless of eval_type; omit it and such a rule skips rather than passes (a critical one is listed in critical_skipped) | |
| expected | No | Expected output for comparison — consulted only by the completeness bundle's expected_coverage rule; NOT used by relevance (the relevance rules compare the output against `input`) | |
| trace_id | No | Link evaluation to a trace — surfaces this eval in the dashboard's trace drill-through and lets the tool reuse the trace's stored tool_calls. Must be the id of a stored trace (from log_trace / get_traces); an unknown id is rejected before anything is evaluated | |
| eval_type | No | Rule bundle to apply: completeness | relevance | safety | cost | custom | all — picks which built-in rules fire. "all" runs every bundle in one call and adds a per-category breakdown. Defaults to "all" when omitted — every bundle runs, safety included, and the response carries a note saying the default ran | |
| tool_calls | No | What the agent DID — the tool calls it made, in order, each { tool_name, input?, output?, latency_ms?, error? } exactly as log_trace records them. Read by the trajectory rules — the rules that judge what the agent DID rather than what it wrote. Omit it and those rules SKIP rather than pass — an evaluation with no trajectory data reports "not judged", never "clean". When trace_id names a stored trace and this argument is omitted, the tool_calls stored on that trace are loaded and used, so a caller who already logged them need not resend them | |
| token_usage | No | Token usage breakdown — only consulted by the cost bundle (eval_type="cost" or "all"; used for token-budget rules) | |
| custom_rules | No | Custom evaluation rules, max 10 per call (deploy persistent rule sets via deploy_rule instead) — fires REGARDLESS of eval_type; pass eval_type="custom" if you want ONLY these. Each entry accepts exactly name, type, config, weight — an unknown key (e.g. a misspelled weight) is rejected |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| id | Yes | the evaluation id, readable at iris://evaluations/{id} | |
| note | No | present when eval_type was omitted | |
| score | Yes | 0..1 weighted quality over the rules that ran | |
| passed | Yes | the ship verdict; false when nothing was judged | |
| verdict | No | state, passed, basis (which layer decided), by (the rules), risk | |
| coverage | No | per question: judged, unjudged and why, or not_applicable; plus the inputs carried | |
| trace_id | No | the linked trace, when named | |
| erased_at | No | set once the linked trace was deleted | |
| eval_type | Yes | the bundle that ran | |
| categories | No | per-bundle verdicts for eval_type all | |
| provenance | No | Iris version, ruleset and config hashes, thresholds, corpus version, time | |
| suggestions | Yes | what to change | |
| rule_results | Yes | per rule: verdict, message, kind, role, question, saw, evidence, uncertainty | |
| rules_skipped | Yes | rules that skipped | |
| rules_evaluated | Yes | rules that judged | |
| critical_skipped | No | critical rules that could not judge; treat as unknown | |
| critical_failures | No | critical rules that failed and vetoed passed | |
| insufficient_data | Yes | true when no rule could judge |