Store one agent execution — input, output, tool calls, spans, cost, latency, token usage — and get the trace_id every later call keys on.

What it does. Writes one trace row to local SQLite and mints a fresh trace_id; nothing is deduplicated, so resubmitting the same payload stores a second trace. Only agent_name is required. Store what you have: tool_calls so the trajectory rules can later judge what the agent did, cost_usd and token_usage so the cost rules can, input and output so everything else can. When IRIS_OTEL_ENDPOINT is set the trace is also exported to that collector, best-effort and asynchronous; the local write never waits on it. Traces are immutable: there is no update path. In stdio mode nothing authenticates the caller; over HTTP a Bearer token is required only when an API key is configured.

When not to use it. For a transient log line (use your logger). To score an output: log first, then call evaluate_output with the trace_id, which also lets it reuse the stored tool_calls. To change a stored trace: delete_trace and log again.

Returns. JSON with trace_id (the stored trace id, 32 hex — pass it to evaluate_output, get_traces or delete_trace); status (always "stored" on success).

Errors. IRIS_STORAGE_ERROR when the database cannot be written. An unknown argument or a malformed span or tool_calls entry is refused before the handler runs, naming the valid keys. Every failure returns {"error":{"code","message","recovery":[]}} with isError true; follow recovery before retrying.

Siblings. evaluate_output — score the stored output; get_traces — query what was logged; delete_trace — remove one trace.

Score an agent output against the deterministic rule bundles: the ship verdict with its basis, every rule result with evidence and uncertainty, and what was not judged.

What it does. In-process, no network, no key. eval_type picks one bundle (completeness, relevance, safety, cost, custom) or all (the default): every bundle plus deployed and inline custom rules, with a per-bundle breakdown in categories. Inputs decide what can be judged: input is REQUIRED when eval_type="relevance" (keyword_overlap and topic_consistency compare the output against it and skip without it) and grounds the hallucination signals; tool_calls, or a trace_id whose stored tool_calls are reused, feed the trajectory rules; cost_usd and token_usage feed the cost rules; expected feeds only expected_coverage. A rule without its input SKIPS, is named, and never counts as a pass. custom_rules always fire. One row is stored, linked to trace_id when given.

When not to use it. To validate arbitrary JSON Schema (the json_schema custom type asserts an output's shape only). To screen inputs before they reach an agent: no_injection_patterns inspects the agent's OUTPUT text for injection-shaped content — attack phrasing and structural directives the output echoes or complies with — and never reads the input, so it is not an input firewall. For semantic judgment, evaluate_with_llm_judge and verify_citations need a key you supply.

Returns. JSON with id (the evaluation id, readable at iris://evaluations/{id}); trace_id (the linked trace, when named); verdict (state, passed, basis (which layer decided), by (the rules), risk); coverage (per question: judged, unjudged and why, or not_applicable; plus the inputs carried); provenance (Iris version, ruleset and config hashes, thresholds, corpus version, time); erased_at (set once the linked trace was deleted); eval_type (the bundle that ran); score (0..1 weighted quality over the rules that ran); passed (the ship verdict; false when nothing was judged); rule_results (per rule: verdict, message, kind, role, question, saw, evidence, uncertainty); suggestions (what to change); rules_evaluated (rules that judged); rules_skipped (rules that skipped); insufficient_data (true when no rule could judge); critical_failures (critical rules that failed and vetoed passed); critical_skipped (critical rules that could not judge; treat as unknown); categories (per-bundle verdicts for eval_type all); note (present when eval_type was omitted).

Errors. IRIS_UNKNOWN_TRACE when trace_id names no stored trace — checked first, nothing scored or written. IRIS_STORAGE_ERROR when the row cannot be written. Unknown arguments or keys are refused before the handler runs, naming the valid ones; a regex rule over its budget or with a broken config reports skipped, not an error. Every failure returns {"error":{"code","message","recovery":[]}} with isError true; follow recovery before retrying.

Siblings. log_trace — record the execution first; evaluate_with_llm_judge — semantic scoring on your key; verify_citations — citation grounding on your key; list_rules — the roster, needs and published accuracy.

Query stored traces with filters, pagination and sorting; optionally include the dashboard summary in the same response.

What it does. Read-only, local storage only. Filters are exact-match (agent_name, framework), inclusive time bounds (since, until — an ISO 8601 timestamp or date) and a score range applied to the LATEST evaluation of each trace (min_score, max_score, 0..1). limit is 1..1000 (default 50), offset counts from 0, sort_by is timestamp, latency_ms or cost_usd, sort_order asc or desc (default: newest first). include_summary adds the one-hour dashboard aggregates. A crossed range (min above max, since after until) is refused naming both values rather than returning an empty page that reads as "no such traces".

When not to use it. To score a trace (evaluate_output). To create one (log_trace). As a live stream: this is a query, and Iris has no event stream — poll with backoff.

Returns. JSON with traces (the page of traces: trace_id, agent_name, framework, input, output, tool_calls, latency_ms, token_usage, cost_usd, metadata, timestamp); total (how many traces match the filters, across every page); limit (the page size applied); offset (the offset applied); summary (the dashboard aggregates for the last hour, when include_summary was true).

Errors. IRIS_STORAGE_ERROR when the database cannot be read. An out-of-range or crossed bound is refused before the handler runs, naming the values. An empty result is total 0, not an error. Every failure returns {"error":{"code","message","recovery":[]}} with isError true; follow recovery before retrying.

Siblings. log_trace — record an execution; evaluate_output — score one output; delete_trace — remove one trace.

The rule inventory: the built-in roster with what each rule needs, the criticality this server applies and its published accuracy, plus every deployed custom rule.

What it does. Read-only, no network. built_in is the shipped roster and is never narrowed by the filters. For each rule: kind (measurement, detection, inference, judgment, policy, verification), mechanism, needs (the inputs it reads — absent means the rule skips, never passes), question, classes, version, weight, the EFFECTIVE critical flag with criticalSource (default, or config when eval.criticalRules / eval.nonCriticalRules changed it on this server — read it before trusting a passed: true), and proof: precision and recall with 95% intervals and the positive predictive value at four prevalences, the numbers published at https://iris-eval.com/proof. rules is the custom-rule store, filterable by eval_type and enabled_only; total and enabled_count count custom rules. quarantined lists store entries this version could not validate; they do not fire.

When not to use it. To count traces (get_traces). To add, remove or pause a rule (deploy_rule, delete_rule). Built-in rules are not in the store and cannot be deployed, deleted or disabled.

Returns. JSON with rules (the deployed custom rules after the filters: id, name, description, evalType, severity, definition, enabled, createdAt, updatedAt, version, sourceMomentId); total (custom rules after the filters); enabled_count (of those, how many are enabled); built_in (the shipped roster, never filtered: name, category, description, weight, kind, mechanism, needs, question, classes, version, the EFFECTIVE critical flag with criticalSource, and proof (published precision, recall, intervals and ppvAt from https://iris-eval.com/proof; null where the proof is a conformance check)); quarantined (entries in the store this version could not validate; they do not fire and are never deleted by a deploy).

Errors. IRIS_INTERNAL_ERROR if the store file cannot be read. A missing store file is an empty list, not an error. Every failure returns {"error":{"code","message","recovery":[]}} with isError true; follow recovery before retrying.

Siblings. deploy_rule — add a custom rule; delete_rule — remove, disable or re-enable one; evaluate_output — run the rules.

Deploy a custom rule that fires on every future evaluate_output call of its bundle — persisted, active immediately, audited.

What it does. Writes the rule to ~/.iris/custom-rules.json, appends a rule.deploy audit entry and registers it with the running engine, so it fires on the very next call and survives restarts. eval_type says WHEN it fires (that bundle, and eval_type="all"); severity says what a failure DOES: low and medium only lower the weighted score, high and critical force passed to false and list the rule in critical_failures. definition.type picks the check (regex_match, regex_no_match, min_length, max_length, contains_keywords, excludes_keywords, json_schema, cost_threshold) and definition.config carries its keys (pattern; min_length; max_length; keywords; max_cost). Any bundle and type combine. Names are unique: a taken name is refused unless replace is true, which retires the earlier rule(s) first and reports them. Argument names are snake_case; the camelCase aliases evalType and sourceMomentId are accepted — pass one spelling of each.

When not to use it. To try a rule first: POST /api/v1/rules/custom/preview on the dashboard replays a definition against stored traces without deploying. For a one-off check on one call: the custom_rules argument of evaluate_output. To pause a rule: delete_rule with enabled: false.

Returns. JSON with rule (the rule as persisted: id (rule-, keep it for delete_rule), name, description, evalType, severity, definition, enabled, createdAt, updatedAt, version, sourceMomentId); replaced (with replace: true, the earlier rule(s) of the same name that were retired); warning (with replace: true, one sentence naming what was retired).

Errors. IRIS_DUPLICATE_RULE when the name is deployed and replace is false (the message names the existing id). IRIS_INVALID_RULE_CONFIG when the definition is rejected — a regex that fails the ReDoS check or exceeds 1000 characters, a missing config key — naming the field; nothing is deployed. IRIS_STORAGE_ERROR when the store cannot be written. An unknown key in definition, a name over 80 characters, a non-positive weight or both spellings of an alias are refused before the handler runs. Every failure returns {"error":{"code","message","recovery":[]}} with isError true; follow recovery before retrying.

Siblings. list_rules — see what is deployed and the built-in roster; delete_rule — remove, disable or re-enable; evaluate_output — where the rule fires.

Remove a deployed custom rule — or, with enabled, disable or re-enable it without removing it — effective on the next evaluate_output call.

What it does. Without enabled: deletes the rule from ~/.iris/custom-rules.json, appends a rule.delete audit entry and unregisters it from the running engine; deleted is false when no rule has that id (already gone, or not this tenant's), and no audit row is written twice. With enabled: the rule stays with its history and provenance; false stops it firing at once and keeps it off across restarts, true brings it back under the same id; a rule.toggle audit entry is written unless the flag was already in that state. Past evaluations that referenced the rule are untouched either way.

When not to use it. On built-in rules: they are not in the store and cannot be deleted or disabled. To delete a trace (delete_trace). To replace a rule: deploy_rule with the same name and replace: true.

Returns. JSON with deleted (true when a rule was removed; always false on a toggle); rule_id (the id that was asked for); toggled (toggle only: true when the rule exists (also when it was already in the requested state)); enabled (toggle only: the rule's state after the call); rule (toggle only: the rule as stored).

Errors. IRIS_STORAGE_ERROR when the store cannot be written. A malformed rule_id (not rule-) or an unknown argument is refused before the handler runs. Every failure returns {"error":{"code","message","recovery":[]}} with isError true; follow recovery before retrying.

Siblings. deploy_rule — add or replace a rule; list_rules — find the id; evaluate_output — where the rule fires.

Remove one stored trace by id; its spans go with it, and every evaluation linked to it keeps its verdict and loses its text.

What it does. Deletes the trace row for the caller's tenant. Spans cascade. Evaluations linked to it keep their verdict, scores, criticality and evidence offsets; their output text, expected text, suggestions and rule messages are erased in the same transaction and erased_at is stamped, so no text from the trace survives in any evaluation. deleted is false when no trace has that id — already removed, or not this tenant's — and that is not an error. No audit entry is written: traces are user data, not policy.

When not to use it. To expire old data in bulk (retention.days; the sweep runs at boot and every retention.sweepIntervalHours). To delete evaluations: they are not deleted per row; retention and --purge cover them. To pause anything: traces are immutable, there is nothing to pause.

Returns. JSON with deleted (true when a trace row was removed; false when no trace with that id existed for this tenant); trace_id (the id that was asked for).

Errors. IRIS_STORAGE_ERROR when the delete cannot run. A malformed trace_id (not 32 lowercase hex) is refused before the handler runs. Every failure returns {"error":{"code","message","recovery":[]}} with isError true; follow recovery before retrying.

Siblings. log_trace — store a trace; get_traces — find the trace to delete; delete_rule — the equivalent for custom rules.

Score an output with an LLM judge on your own provider key: a 0..1 score, a rationale, per-dimension sub-scores and the exact spend.

What it does. Calls Anthropic or OpenAI directly with the key in this process's environment (IRIS_ANTHROPIC_API_KEY or IRIS_OPENAI_API_KEY); Iris never proxies. template picks the question: accuracy, helpfulness, safety, correctness (needs expected) or faithfulness (needs source_material); input improves helpfulness and safety. model is required; provider is inferred from it. The worst-case spend — both attempts, full max_output_tokens — is computed BEFORE the call and refused if it exceeds max_cost_usd (default IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL or 0.25). temperature defaults to 0; a rate-limited call is retried once. One evaluation row is stored with the provider response id, tokens, cost and latency, linked to trace_id when given. The judge's own accuracy is measurable on a key you supply and is not yet published (see iris://proof).

When not to use it. For length, keyword, PII, injection or cost checks: evaluate_output is free and deterministic. Without a key: the call returns IRIS_JUDGE_NOT_ENABLED with the enable steps — do not search for them. On very large outputs without raising max_cost_usd: the pre-check refuses.

Returns. JSON with id (the evaluation id; read it back at iris://evaluations/{id}); trace_id (the linked trace, when one was named); score (0..1 from the judge); passed (the verdict: the score against the template's threshold, which is pass_threshold below. Not the model's own boolean — that is self_reported_pass); pass_threshold (the threshold the score was read against, so you can check the arithmetic); self_reported_pass (what the model said about passing, when it said anything. Recorded, never obeyed); disagreement (true when the model's own boolean disagrees with the threshold verdict — its rubric and its judgement have come apart on this output); rationale (the judge's reasoning, in its words); dimensions (per-dimension sub-scores for the template); model (the model that judged); provider (the provider called); template (the template used); input_tokens (tokens sent, across both attempts when a retry ran); output_tokens (tokens received, across both attempts when a retry ran); cost_usd (the exact spend from the pricing table); latency_ms (wall time of the provider call(s)); raw_response_id (the provider's response id, for your own audit).

Errors. IRIS_JUDGE_NOT_ENABLED (no key for the provider reached this process; recovery carries the steps). IRIS_JUDGE_UNKNOWN_MODEL (valid lists the models). IRIS_UNKNOWN_TRACE, checked before any spend. IRIS_BUDGET_EXCEEDED (nothing spent; the message carries both numbers). IRIS_PROVIDER_ERROR with kind auth, rate_limit, bad_request, server_error, timeout or malformed_response, and retryable set. Every failure returns {"error":{"code","message","recovery":[]}} with isError true; follow recovery before retrying.

Siblings. evaluate_output — the free deterministic path; verify_citations — citation grounding, the narrower judge; log_trace — record the execution first.

Extract the citations in an output, fetch the sources (opt-in, SSRF-guarded) and ask an LLM judge on your key whether each source supports its claim.

What it does. Three phases. Extraction, no network: [N] references, (Author, Year), bare URLs and DOIs. Fetch of URL and DOI citations only when allow_fetch is true or IRIS_CITATION_ALLOW_FETCH=1, through a scheme allowlist, private and cloud-metadata address blocking, an optional hostname allowlist (domain_allowlist, merged with IRIS_CITATION_DOMAINS), a per-source timeout and byte cap, and at most three re-checked redirects. Then one judge call per resolved citation on your own key, reading the first part of each source, capped in total by max_cost_usd_total. Up to max_citations are verified; extras are skipped, not errored. overall_score is supported / judged and null when nothing was judged. Per-citation failures (bad scheme, blocked address, timeout, too large, cost cap, fetch disabled) are reported on the citation, never scored as unsupported. One evaluation row is stored.

When not to use it. When the output has no citations: the score is null, and evaluate_output's hallucination signals are the cheap check. Without a key (IRIS_ANTHROPIC_API_KEY or IRIS_OPENAI_API_KEY): the call returns IRIS_JUDGE_NOT_ENABLED with the enable steps. With fetch enabled and an open allowlist on untrusted output: you are running a user-directed fetcher — set IRIS_CITATION_DOMAINS.

Returns. JSON with id (the evaluation id; read it back at iris://evaluations/{id}); trace_id (the linked trace, when one was named); overall_score (supported / judged; null when nothing was judged); passed (true when every judged citation was supported; false when any judged citation was not; NULL when nothing was judged — no verdict, because nothing was verified. Until 0.10.0 that last case returned true.); total_unsupported (judged citations the judge ruled unsupported — the number the verdict turns on); total_citations_found (citations extracted from the output); total_resolved (citations whose source was fetched); total_judged (citations the judge ruled on); total_supported (citations the judge found supported); total_cost_usd (the spend across every judge call); citations (per citation: the citation (raw, kind, identifier, offsets), resolve_status ok | skipped | error, resolve_error, source (url, status, content_type, bytes_fetched, truncated), judge (supported, confidence, rationale, cost_usd, latency_ms, tokens)).

Errors. IRIS_JUDGE_NOT_ENABLED, IRIS_JUDGE_UNKNOWN_MODEL and IRIS_UNKNOWN_TRACE before any fetch or spend. IRIS_JUDGE_FAILED when citations resolved but the judge failed on every one — an error, not a passing verdict; nothing is stored. Every failure returns {"error":{"code","message","recovery":[]}} with isError true; follow recovery before retrying.

Siblings. evaluate_with_llm_judge — general semantic scoring; evaluate_output — the free deterministic path, including the hallucination signals; log_trace — record the execution first.