Know whether your AI agents are actually good enough to ship. Iris is an open-source MCP server that scores output quality, catches safety failures, and enforces cost budgets across all your agents. Any MCP-compatible agent discovers and uses it automatically — no SDK, no code changes.
Your agents are running in production. Infrastructure monitoring sees 200 OK and moves on. It has no idea the agent just:
Leaked a social security number in its response
Hallucinated an answer with zero factual grounding
Burned $0.47 on a single query — 4.7x your budget threshold
Made 6 tool calls when 2 would have sufficed
Iris evaluates all of it.
Related MCP server: runmeter
Trace Logging | Hierarchical span trees with per-tool-call latency, token usage, and cost in USD. Stored in SQLite, queryable instantly. |
Output Evaluation | 13 built-in rules across 4 categories: completeness, relevance, safety, cost. PII detection (10 patterns: SSN, credit card, phone, email, IBAN, DOB, MRN, IP, API key, passport), prompt injection (13 patterns), stub-output detection, hallucination markers (17 hedging phrases + fabricated-citation heuristic). Add custom rules with Zod schemas. |
Cost Visibility | Aggregate cost across all agents over any time window. Set budget thresholds. Get flagged when agents overspend. |
Web Dashboard | Real-time dark-mode UI with trace visualization, eval results, and cost breakdowns. |
Requires Node.js 20 or later. Check with node --version.
Add Iris to your MCP config. Works with Claude Desktop, Cursor, Windsurf, and any MCP-compatible agent.
{
"mcpServers": {
"iris-eval": {
"command": "npx",
"args": ["@iris-eval/mcp-server"]
}
}
}That's it. Your agent discovers Iris and starts logging traces automatically.
Iris ships with a real-time web dashboard showing traces, eval results, cost breakdowns, and rule pass-rates. It's off by default so the MCP server stays lightweight — flip it on with a flag.
{
"mcpServers": {
"iris-eval": {
"command": "npx",
"args": ["@iris-eval/mcp-server", "--dashboard"]
}
}
}Then open http://localhost:6920 after your agent runs a trace. The same dashboard is available via CLI:
npx @iris-eval/mcp-server --dashboardEdit your MCP config file:
macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json
Add the JSON config above, then restart Claude Desktop.
claude mcp add --transport stdio iris-eval -- npx @iris-eval/mcp-serverThen restart the session (/clear or relaunch) for tools to load.
Windows note: Do not use
cmd /cwrapper — it causes path parsing issues. Thenpxcommand works directly.
Add to your workspace .cursor/mcp.json or global MCP settings using the JSON config above.
# Global install (recommended for persistent data and faster startup)
npm install -g @iris-eval/mcp-server
iris-mcp --dashboard
# Docker
docker run -p 3000:3000 -v iris-data:/data ghcr.io/iris-eval/mcp-serverTip: Global install (
npm install -g) stores traces persistently at~/.iris/iris.db. Withnpx, traces persist in the same location, but startup is slower due to package resolution.
Iris registers nine tools that any MCP-compatible agent can invoke — full rule + trace lifecycle + LLM-as-judge + semantic citation verification:
log_trace — Log an agent execution with spans, tool calls, token usage, and cost
evaluate_output — Score output quality against completeness, relevance, safety, and cost rules (heuristic, deterministic, free)
get_traces — Query stored traces with filtering, pagination, and time-range support
list_rules — Enumerate deployed custom eval rules (read-only)
deploy_rule — Register a new custom eval rule so it fires on every evaluate_output of that category
delete_rule — Remove a deployed custom rule (destructive, idempotent)
delete_trace — Remove a single stored trace by ID (destructive, tenant-scoped)
evaluate_with_llm_judge — Semantic eval via LLM (Anthropic or OpenAI). Five templates: accuracy, helpfulness, safety, correctness, faithfulness. Cost-capped, per-eval pricing disclosed. Bring your own API key (IRIS_ANTHROPIC_API_KEY or IRIS_OPENAI_API_KEY) — Iris doesn't proxy or relay LLM calls.
verify_citations — Extract citations from output (numbered, author-year, URLs, DOIs), fetch sources behind an SSRF-guarded + domain-allowlisted resolver, and use an LLM judge to check whether each source actually supports the cited claim. Opt-in outbound HTTP. Same BYOK requirement as evaluate_with_llm_judge.
When IRIS_OTEL_ENDPOINT is configured, log_trace calls also emit a best-effort OTLP/HTTP JSON export to any OpenTelemetry collector (Jaeger, Grafana Tempo, Datadog OTLP, Honeycomb, etc). See docs/otel-integration.md.
Full tool schemas and configuration: iris-eval.com
Self-hosted Iris runs on your machine with SQLite. As your team's eval needs grow, the cloud tier adds PostgreSQL, team dashboards, alerting on quality regressions, and managed infrastructure.
Join the waitlist to get early access.
Claude Desktop setup — MCP config for stdio and HTTP modes
TypeScript — MCP SDK client — connect and invoke tools
HTTP transport (TS + Python) — full client code for REST-style integration
LangChain instrumentation (Python, conceptual) — scaffold showing the shape; needs your agent code to be runnable
CrewAI instrumentation (Python, conceptual) — scaffold; same caveat
GitHub Issues — Bug reports and feature requests
GitHub Discussions — Questions and ideas
Contributing Guide — How to contribute
Roadmap — What's coming next
Flag | Default | Description |
|
| Transport type: |
|
| HTTP transport port |
|
| SQLite database path |
|
| Config file path |
| — | API key for HTTP authentication |
|
| Enable web dashboard |
|
| Dashboard port |
Variable | Description |
| Transport type ( |
| HTTP transport port |
| HTTP transport host (default |
| SQLite database path |
| Log level: |
| Enable web dashboard ( |
| Dashboard port (default |
| API key for HTTP authentication |
| Comma-separated allowed CORS origins |
CLI flags take precedence over environment variables when both are set.
When using HTTP transport, Iris includes:
API key authentication with timing-safe comparison
CORS restricted to localhost by default
Rate limiting (100 req/min API, 20 req/min MCP)
Helmet security headers
Zod input validation on all routes
ReDoS-safe regex for custom eval rules
1MB request body limits
# Production deployment
iris-mcp --transport http --port 3000 --api-key "$(openssl rand -hex 32)" --dashboardERR_MODULE_NOT_FOUNDYou may have a cached older version. Clear the npx cache and retry:
npx --yes @iris-eval/mcp-server@latestOr install globally to avoid cache issues entirely:
npm install -g @iris-eval/mcp-server@latestMCP tools only load at session start. After adding iris-eval, restart the session with /clear or relaunch the terminal.
Verify which version is running:
npx @iris-eval/mcp-server --help
# Shows "Iris — MCP-Native Agent Eval Server vX.Y.Z"# If using npx (clears cache and fetches latest)
npx --yes @iris-eval/mcp-server@latest
# If installed globally
npm update -g @iris-eval/mcp-serverIris requires Node.js 20 or later. Node 18 reached EOL in April 2025 and is not supported.
node --version # Must be v20.x or v22.x+cmd /c not neededClaude Code's /doctor may suggest wrapping npx with cmd /c. This is not needed and causes path parsing issues. Use npx directly:
# Correct
claude mcp add --transport stdio iris-eval -- npx @iris-eval/mcp-server
# Wrong (causes /c to be parsed as a path)
claude mcp add --transport stdio iris-eval -- cmd /c "npx @iris-eval/mcp-server"If Iris is useful to you, consider starring the repo — it helps others find it.
MIT Licensed.
Remove a deployed custom rule — or, with enabled, disable or re-enable it without removing it — effective on the next evaluate_output call.
What it does. Without enabled: deletes the rule from ~/.iris/custom-rules.json, appends a rule.delete audit entry and unregisters it from the running engine; deleted is false when no rule has that id (already gone, or not this tenant's), and no audit row is written twice. With enabled: the rule stays with its history and provenance; false stops it firing at once and keeps it off across restarts, true brings it back under the same id; a rule.toggle audit entry is written unless the flag was already in that state. Past evaluations that referenced the rule are untouched either way.
When not to use it. On built-in rules: they are not in the store and cannot be deleted or disabled. To delete a trace (delete_trace). To replace a rule: deploy_rule with the same name and replace: true.
Returns. JSON with deleted (true when a rule was removed; always false on a toggle); rule_id (the id that was asked for); toggled (toggle only: true when the rule exists (also when it was already in the requested state)); enabled (toggle only: the rule's state after the call); rule (toggle only: the rule as stored).
Errors. IRIS_STORAGE_ERROR when the store cannot be written. A malformed rule_id (not rule-) or an unknown argument is refused before the handler runs. Every failure returns {"error":{"code","message","recovery":[]}} with isError true; follow recovery before retrying.
Siblings. deploy_rule — add or replace a rule; list_rules — find the id; evaluate_output — where the rule fires.
| Name | Required | Description | Default |
|---|---|---|---|
| enabled | No | When present the rule is NOT deleted: false DISABLES it (kept in the store, stops firing immediately, history and provenance preserved); true RE-ENABLES a disabled rule. Omit to delete | |
| rule_id | Yes | Rule id to delete or toggle (format: rule-<hex>); obtained from list_rules or deploy_rule response |
| Name | Required | Description |
|---|---|---|
| rule | No | toggle only: the rule as stored |
| deleted | Yes | true when a rule was removed; always false on a toggle |
| enabled | No | toggle only: the rule's state after the call |
| rule_id | Yes | the id that was asked for |
| toggled | No | toggle only: true when the rule exists (also when it was already in the requested state) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Goes far beyond the destructiveHint annotation by disclosing the exact file modified, audit entries written, engine unregistration, persistence across restarts, no impact on past evaluations, error shape, and malformed-rule_id rejection. This gives the agent a rich behavioral model.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but tightly organized under clear labeled sections ('What it does', 'When not to use it', 'Returns', 'Errors', 'Siblings'), and the first sentence front-loads the core behavior. Every sentence contributes necessary information for a complex destructive/toggle tool.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Covers side effects, timing, return values, error recovery, exclusions, and sibling routing. Even with an output schema noted, the operational nuances described here are essential and nothing critical is missing for an agent to call this tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3, but the description adds critical operational meaning: omitting enabled means delete, enabled: false stops firing immediately while preserving history, enabled: true restores the same id, and return fields differ between delete and toggle modes. This materially improves correct invocation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: 'Remove a deployed custom rule — or, with enabled, disable or re-enable it'. It clearly distinguishes this tool from siblings by naming delete_trace, deploy_rule, and the built-in rule exclusion, so an agent can select it correctly without ambiguity.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Contains an explicit 'When not to use it' section naming alternatives and conditions: built-in rules cannot be modified, trace deletion belongs to delete_trace, and rule replacement belongs to deploy_rule with replace: true. This is ideal routing guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Remove one stored trace by id; its spans go with it, and every evaluation linked to it keeps its verdict and loses its text.
What it does. Deletes the trace row for the caller's tenant. Spans cascade. Evaluations linked to it keep their verdict, scores, criticality and evidence offsets; their output text, expected text, suggestions and rule messages are erased in the same transaction and erased_at is stamped, so no text from the trace survives in any evaluation. deleted is false when no trace has that id — already removed, or not this tenant's — and that is not an error. No audit entry is written: traces are user data, not policy.
When not to use it. To expire old data in bulk (retention.days; the sweep runs at boot and every retention.sweepIntervalHours). To delete evaluations: they are not deleted per row; retention and --purge cover them. To pause anything: traces are immutable, there is nothing to pause.
Returns. JSON with deleted (true when a trace row was removed; false when no trace with that id existed for this tenant); trace_id (the id that was asked for).
Errors. IRIS_STORAGE_ERROR when the delete cannot run. A malformed trace_id (not 32 lowercase hex) is refused before the handler runs. Every failure returns {"error":{"code","message","recovery":[]}} with isError true; follow recovery before retrying.
Siblings. log_trace — store a trace; get_traces — find the trace to delete; delete_rule — the equivalent for custom rules.
| Name | Required | Description | Default |
|---|---|---|---|
| trace_id | Yes | Trace id to delete (32-hex lowercase; obtained from log_trace response or get_traces) |
| Name | Required | Description |
|---|---|---|
| deleted | Yes | true when a trace row was removed; false when no trace with that id existed for this tenant |
| trace_id | Yes | the id that was asked for |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already signal destructiveness, but the description goes far beyond that: spans cascade, evaluations keep verdicts but lose text, erased_at is stamped, deleted=false for absent/foreign traces is not an error, and no audit entry is written. It also discloses error behavior and malformed-id refusal, which is valuable execution context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but well-structured with clear sections, and the core action is front-loaded. There is some redundancy between the opening sentence and the 'What it does' section, but the richness of behavioral detail justifies most of the length.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a destructive delete tool with an output schema, the description covers inputs, side effects, return semantics, error format, and exclusions. Nothing an agent needs to decide whether to call it and interpret the result is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and the single parameter trace_id is already fully documented with format and provenance. The description adds no new semantic details about the parameter itself, so baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a precise verb-resource statement: 'Remove one stored trace by id', and immediately distinguishes the tool from siblings like delete_rule. It names log_trace and get_traces as the sourcing/finding counterparts, so an agent can orient without opening schemas.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly covers when to use the tool and, more importantly, when not to use it: bulk retention expiry, deleting evaluations, or pausing traces. It also names the sibling delete_rule as the equivalent for custom rules, giving the agent a clear decision boundary.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Deploy a custom rule that fires on every future evaluate_output call of its bundle — persisted, active immediately, audited.
What it does. Writes the rule to ~/.iris/custom-rules.json, appends a rule.deploy audit entry and registers it with the running engine, so it fires on the very next call and survives restarts. eval_type says WHEN it fires (that bundle, and eval_type="all"); severity says what a failure DOES: low and medium only lower the weighted score, high and critical force passed to false and list the rule in critical_failures. definition.type picks the check (regex_match, regex_no_match, min_length, max_length, contains_keywords, excludes_keywords, json_schema, cost_threshold) and definition.config carries its keys (pattern; min_length; max_length; keywords; max_cost). Any bundle and type combine. Names are unique: a taken name is refused unless replace is true, which retires the earlier rule(s) first and reports them. Argument names are snake_case; the camelCase aliases evalType and sourceMomentId are accepted — pass one spelling of each.
When not to use it. To try a rule first: POST /api/v1/rules/custom/preview on the dashboard replays a definition against stored traces without deploying. For a one-off check on one call: the custom_rules argument of evaluate_output. To pause a rule: delete_rule with enabled: false.
Returns. JSON with rule (the rule as persisted: id (rule-, keep it for delete_rule), name, description, evalType, severity, definition, enabled, createdAt, updatedAt, version, sourceMomentId); replaced (with replace: true, the earlier rule(s) of the same name that were retired); warning (with replace: true, one sentence naming what was retired).
Errors. IRIS_DUPLICATE_RULE when the name is deployed and replace is false (the message names the existing id). IRIS_INVALID_RULE_CONFIG when the definition is rejected — a regex that fails the ReDoS check or exceeds 1000 characters, a missing config key — naming the field; nothing is deployed. IRIS_STORAGE_ERROR when the store cannot be written. An unknown key in definition, a name over 80 characters, a non-positive weight or both spellings of an alias are refused before the handler runs. Every failure returns {"error":{"code","message","recovery":[]}} with isError true; follow recovery before retrying.
Siblings. list_rules — see what is deployed and the built-in roster; delete_rule — remove, disable or re-enable; evaluate_output — where the rule fires.
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes | Human-readable rule name (1-80 chars; used in eval results). Must be unique among deployed rules unless replace=true | |
| replace | No | When a rule with this name is already deployed: false (default) rejects the call; true deletes the existing same-named rule(s) and deploys this one in their place (fresh id; audit rows preserved) | |
| evalType | No | camelCase alias of eval_type, accepted for compatibility — prefer eval_type (snake_case is canonical across the tools) | |
| severity | No | What a FAILURE of this rule means. low/medium: informational — contributes to the weighted score only (plus dashboard sort + audit alerts). high/critical: hard-fail — a failing evaluation of this rule forces the overall passed=false regardless of the weighted score | medium |
| eval_type | No | Eval category this rule belongs to; the rule fires on evaluate_output calls whose eval_type equals it (and on eval_type="all"). Canonical snake_case spelling — pass exactly one of eval_type / evalType | |
| definition | Yes | Check definition (regex, length, keyword, cost, or schema). Accepts exactly type, config, weight and an optional name — an unknown key is rejected | |
| description | No | What this rule checks for and why it matters | |
| sourceMomentId | No | camelCase alias of source_moment_id, accepted for compatibility — prefer source_moment_id | |
| source_moment_id | No | Optional Decision Moment ID the rule was derived from (preserves workflow-inversion provenance). Canonical snake_case — pass exactly one of source_moment_id / sourceMomentId |
| Name | Required | Description |
|---|---|---|
| rule | Yes | the rule as persisted: id (rule-<hex>, keep it for delete_rule), name, description, evalType, severity, definition, enabled, createdAt, updatedAt, version, sourceMomentId |
| warning | No | with replace: true, one sentence naming what was retired |
| replaced | No | with replace: true, the earlier rule(s) of the same name that were retired |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only indicate it is not read-only, not idempotent, and not destructive. The description substantially exceeds them by disclosing file writes, audit entries, engine registration, restart persistence, unique-name refusal, replacement semantics, and detailed error behavior. No contradictions with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but deliberately structured with clear sections: what it does, when not to use it, returns, errors, and siblings. It front-loads the core behavior in the first sentence and every paragraph covers a distinct needed concern.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the complex nested schema and 9 parameters, the description covers side effects, return shape, error codes, recovery behavior, and sibling relationships. An agent has enough information to decide when to call it, construct a valid definition, and interpret outcomes.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, but the description adds meaning beyond the schema: alias handling, the 'all' eval_type behavior, severity's hard-fail impact, definition.config key mapping, and rejection of unknown keys. It compensates richly for any ambiguity left by the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The opening sentence states a specific action and resource: deploying a custom rule that fires on future evaluate_output calls, with persistence, immediate activation, and auditing. It clearly distinguishes this from siblings like list_rules and delete_rule.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Includes an explicit 'When not to use it' section naming preview, evaluate_output's custom_rules argument, and delete_rule for pausing. This gives an agent concrete routing guidance for alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Score an agent output against the deterministic rule bundles: the ship verdict with its basis, every rule result with evidence and uncertainty, and what was not judged.
What it does. In-process, no network, no key. eval_type picks one bundle (completeness, relevance, safety, cost, custom) or all (the default): every bundle plus deployed and inline custom rules, with a per-bundle breakdown in categories. Inputs decide what can be judged: input is REQUIRED when eval_type="relevance" (keyword_overlap and topic_consistency compare the output against it and skip without it) and grounds the hallucination signals; tool_calls, or a trace_id whose stored tool_calls are reused, feed the trajectory rules; cost_usd and token_usage feed the cost rules; expected feeds only expected_coverage. A rule without its input SKIPS, is named, and never counts as a pass. custom_rules always fire. One row is stored, linked to trace_id when given.
When not to use it. To validate arbitrary JSON Schema (the json_schema custom type asserts an output's shape only). To screen inputs before they reach an agent: no_injection_patterns inspects the agent's OUTPUT text for injection-shaped content — attack phrasing and structural directives the output echoes or complies with — and never reads the input, so it is not an input firewall. For semantic judgment, evaluate_with_llm_judge and verify_citations need a key you supply.
Returns. JSON with id (the evaluation id, readable at iris://evaluations/{id}); trace_id (the linked trace, when named); verdict (state, passed, basis (which layer decided), by (the rules), risk); coverage (per question: judged, unjudged and why, or not_applicable; plus the inputs carried); provenance (Iris version, ruleset and config hashes, thresholds, corpus version, time); erased_at (set once the linked trace was deleted); eval_type (the bundle that ran); score (0..1 weighted quality over the rules that ran); passed (the ship verdict; false when nothing was judged); rule_results (per rule: verdict, message, kind, role, question, saw, evidence, uncertainty); suggestions (what to change); rules_evaluated (rules that judged); rules_skipped (rules that skipped); insufficient_data (true when no rule could judge); critical_failures (critical rules that failed and vetoed passed); critical_skipped (critical rules that could not judge; treat as unknown); categories (per-bundle verdicts for eval_type all); note (present when eval_type was omitted).
Errors. IRIS_UNKNOWN_TRACE when trace_id names no stored trace — checked first, nothing scored or written. IRIS_STORAGE_ERROR when the row cannot be written. Unknown arguments or keys are refused before the handler runs, naming the valid ones; a regex rule over its budget or with a broken config reports skipped, not an error. Every failure returns {"error":{"code","message","recovery":[]}} with isError true; follow recovery before retrying.
Siblings. log_trace — record the execution first; evaluate_with_llm_judge — semantic scoring on your key; verify_citations — citation grounding on your key; list_rules — the roster, needs and published accuracy.
| Name | Required | Description | Default |
|---|---|---|---|
| input | No | Original input for context (the ask + any source material the agent was given) — REQUIRED when eval_type="relevance" (keyword_overlap and topic_consistency compare the output against it and skip without it); also grounds the safety bundle's hallucination signals | |
| output | Yes | The output text to evaluate (the agent's response that gets scored against rules) | |
| cost_usd | No | Cost in USD — consulted by the cost bundle (eval_type="cost" or "all") AND by any cost_threshold custom rule regardless of eval_type; omit it and such a rule skips rather than passes (a critical one is listed in critical_skipped) | |
| expected | No | Expected output for comparison — consulted only by the completeness bundle's expected_coverage rule; NOT used by relevance (the relevance rules compare the output against `input`) | |
| trace_id | No | Link evaluation to a trace — surfaces this eval in the dashboard's trace drill-through and lets the tool reuse the trace's stored tool_calls. Must be the id of a stored trace (from log_trace / get_traces); an unknown id is rejected before anything is evaluated | |
| eval_type | No | Rule bundle to apply: completeness | relevance | safety | cost | custom | all — picks which built-in rules fire. "all" runs every bundle in one call and adds a per-category breakdown. Defaults to "all" when omitted — every bundle runs, safety included, and the response carries a note saying the default ran | |
| tool_calls | No | What the agent DID — the tool calls it made, in order, each { tool_name, input?, output?, latency_ms?, error? } exactly as log_trace records them. Read by the trajectory rules — the rules that judge what the agent DID rather than what it wrote. Omit it and those rules SKIP rather than pass — an evaluation with no trajectory data reports "not judged", never "clean". When trace_id names a stored trace and this argument is omitted, the tool_calls stored on that trace are loaded and used, so a caller who already logged them need not resend them | |
| token_usage | No | Token usage breakdown — only consulted by the cost bundle (eval_type="cost" or "all"; used for token-budget rules) | |
| custom_rules | No | Custom evaluation rules, max 10 per call (deploy persistent rule sets via deploy_rule instead) — fires REGARDLESS of eval_type; pass eval_type="custom" if you want ONLY these. Each entry accepts exactly name, type, config, weight — an unknown key (e.g. a misspelled weight) is rejected |
| Name | Required | Description |
|---|---|---|
| id | Yes | the evaluation id, readable at iris://evaluations/{id} |
| note | No | present when eval_type was omitted |
| score | Yes | 0..1 weighted quality over the rules that ran |
| passed | Yes | the ship verdict; false when nothing was judged |
| verdict | No | state, passed, basis (which layer decided), by (the rules), risk |
| coverage | No | per question: judged, unjudged and why, or not_applicable; plus the inputs carried |
| trace_id | No | the linked trace, when named |
| erased_at | No | set once the linked trace was deleted |
| eval_type | Yes | the bundle that ran |
| categories | No | per-bundle verdicts for eval_type all |
| provenance | No | Iris version, ruleset and config hashes, thresholds, corpus version, time |
| suggestions | Yes | what to change |
| rule_results | Yes | per rule: verdict, message, kind, role, question, saw, evidence, uncertainty |
| rules_skipped | Yes | rules that skipped |
| rules_evaluated | Yes | rules that judged |
| critical_skipped | No | critical rules that could not judge; treat as unknown |
| critical_failures | No | critical rules that failed and vetoed passed |
| insufficient_data | Yes | true when no rule could judge |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description heavily exceeds the annotations' coverage: it states 'In-process, no network, no key', says 'One row is stored, linked to trace_id when given', describes skip semantics for missing inputs, and enumerates error codes (IRIS_UNKNOWN_TRACE, IRIS_STORAGE_ERROR) with recovery guidance. It does not contradict annotations such as idempotentHint or destructiveHint.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but tightly sectioned with clear headings: What it does / When not to use it / Returns / Errors / Siblings. The core purpose is front-loaded, and each section carries non-redundant information needed for a 9-parameter tool with a rich output schema. No filler or tautology is present.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite already having an output schema, the description still summarizes all key return fields, error conditions, default behavior, and skip conditions, making the tool's full contract explicit. It also names siblings and distinguishes their scope, so nothing essential for correct invocation is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is already 100%, but the description adds critical behavior beyond the schema: input is 'REQUIRED when eval_type="relevance"', cost_usd is read by the cost bundle and any cost_threshold custom rule, expected 'feeds only expected_coverage', and tool_calls 'SKIP rather than pass' if omitted. custom_rules 'always fire' regardless of eval_type. This goes far beyond the standalone parameter descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with 'Score an agent output against the deterministic rule bundles' — a specific verb, object, and judgment mechanism. It further distinguishes itself from siblings by naming them in the Siblings section ('evaluate_with_llm_judge — semantic scoring on your key; verify_citations — citation grounding'). An agent can quickly tell this is the deterministic rule-based evaluator.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is an explicit 'When not to use it' section that names three alternative cases — JSON Schema validation, input screening, and semantic judgment — and points to the correct sibling for each. It also explains eval_type bundle selection, the default ('Defaults to "all"'), and when to use custom_rules alone with eval_type='custom'.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Score an output with an LLM judge on your own provider key: a 0..1 score, a rationale, per-dimension sub-scores and the exact spend.
What it does. Calls Anthropic or OpenAI directly with the key in this process's environment (IRIS_ANTHROPIC_API_KEY or IRIS_OPENAI_API_KEY); Iris never proxies. template picks the question: accuracy, helpfulness, safety, correctness (needs expected) or faithfulness (needs source_material); input improves helpfulness and safety. model is required; provider is inferred from it. The worst-case spend — both attempts, full max_output_tokens — is computed BEFORE the call and refused if it exceeds max_cost_usd (default IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL or 0.25). temperature defaults to 0; a rate-limited call is retried once. One evaluation row is stored with the provider response id, tokens, cost and latency, linked to trace_id when given. The judge's own accuracy is measurable on a key you supply and is not yet published (see iris://proof).
When not to use it. For length, keyword, PII, injection or cost checks: evaluate_output is free and deterministic. Without a key: the call returns IRIS_JUDGE_NOT_ENABLED with the enable steps — do not search for them. On very large outputs without raising max_cost_usd: the pre-check refuses.
Returns. JSON with id (the evaluation id; read it back at iris://evaluations/{id}); trace_id (the linked trace, when one was named); score (0..1 from the judge); passed (the verdict: the score against the template's threshold, which is pass_threshold below. Not the model's own boolean — that is self_reported_pass); pass_threshold (the threshold the score was read against, so you can check the arithmetic); self_reported_pass (what the model said about passing, when it said anything. Recorded, never obeyed); disagreement (true when the model's own boolean disagrees with the threshold verdict — its rubric and its judgement have come apart on this output); rationale (the judge's reasoning, in its words); dimensions (per-dimension sub-scores for the template); model (the model that judged); provider (the provider called); template (the template used); input_tokens (tokens sent, across both attempts when a retry ran); output_tokens (tokens received, across both attempts when a retry ran); cost_usd (the exact spend from the pricing table); latency_ms (wall time of the provider call(s)); raw_response_id (the provider's response id, for your own audit).
Errors. IRIS_JUDGE_NOT_ENABLED (no key for the provider reached this process; recovery carries the steps). IRIS_JUDGE_UNKNOWN_MODEL (valid lists the models). IRIS_UNKNOWN_TRACE, checked before any spend. IRIS_BUDGET_EXCEEDED (nothing spent; the message carries both numbers). IRIS_PROVIDER_ERROR with kind auth, rate_limit, bad_request, server_error, timeout or malformed_response, and retryable set. Every failure returns {"error":{"code","message","recovery":[]}} with isError true; follow recovery before retrying.
Siblings. evaluate_output — the free deterministic path; verify_citations — citation grounding, the narrower judge; log_trace — record the execution first.
| Name | Required | Description | Default |
|---|---|---|---|
| input | No | User question / prompt that produced the output (improves accuracy for helpfulness/safety) | |
| model | Yes | Model ID. Supported: anthropic = claude-opus-4-7 | claude-sonnet-4-6 | claude-haiku-4-5 | claude-haiku-4-5-20251001; openai = gpt-4o | gpt-4o-mini | o1-mini. Required — cost varies a hundredfold across models | |
| output | Yes | The agent output text to evaluate | |
| expected | No | Reference answer (required for correctness template) | |
| provider | No | Auto-detected from model when omitted | |
| template | Yes | Judge dimension: accuracy (factual correctness), helpfulness (does it address the ask), safety (harm potential), correctness (vs reference answer — requires `expected`), faithfulness (RAG grounding — requires `source_material`). | |
| trace_id | No | Link this evaluation to a stored trace (id from log_trace / get_traces); an unknown id is rejected BEFORE the judge is called | |
| timeout_ms | No | Per-request timeout; default 60_000 | |
| temperature | No | Sampling temperature; default 0 (deterministic) | |
| max_cost_usd | No | Cost cap in USD for this call; defaults to IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL or 0.25. The worst case (two attempts, full max_output_tokens) is computed before the call and refused if it exceeds the cap | |
| source_material | No | Provided RAG sources (required for faithfulness template) | |
| max_output_tokens | No | Judge output token cap; default 512 |
| Name | Required | Description |
|---|---|---|
| id | Yes | the evaluation id; read it back at iris://evaluations/{id} |
| model | Yes | the model that judged |
| score | Yes | 0..1 from the judge |
| passed | Yes | the verdict: the score against the template's threshold, which is pass_threshold below. Not the model's own boolean — that is self_reported_pass |
| cost_usd | Yes | the exact spend from the pricing table |
| provider | Yes | the provider called |
| template | Yes | the template used |
| trace_id | No | the linked trace, when one was named |
| rationale | Yes | the judge's reasoning, in its words |
| dimensions | Yes | per-dimension sub-scores for the template |
| latency_ms | Yes | wall time of the provider call(s) |
| disagreement | No | true when the model's own boolean disagrees with the threshold verdict — its rubric and its judgement have come apart on this output |
| input_tokens | Yes | tokens sent, across both attempts when a retry ran |
| output_tokens | Yes | tokens received, across both attempts when a retry ran |
| pass_threshold | Yes | the threshold the score was read against, so you can check the arithmetic |
| raw_response_id | No | the provider's response id, for your own audit |
| self_reported_pass | No | what the model said about passing, when it said anything. Recorded, never obeyed |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description goes far beyond annotations, disclosing that Iris never proxies, that spend is pre-computed and capped before the call, that rate-limited calls retry once, that temperature defaults to 0, and that a row is stored with provider response id/tokens/cost/latency. It even explains that self_reported_pass is recorded but never obeyed, and that the pre-check refuses before spending. No contradiction with annotations (readOnlyHint=false, destructiveHint=false) — the write/store behavior is consistent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long, but every section earns its place: What it does, When not to use it, Returns, Errors, Siblings. It is structured with headers and front-loaded with the essential purpose and cost/behavior caveats. Some redundancy exists (the template requirements are stated both in prose and in schema descriptions, and the return field list is exhaustive), but the structure makes it navigable and scannable. Not a 5 because it could be tightened by trimming the full return-field enumeration that the output schema already documents.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 12-parameter, externally-calling, cost-bearing tool with an output schema, the description covers everything an agent needs: what happens before the call (budget pre-check, trace validation), what happens during (retry, direct provider call), what is returned (mapped to the output schema), what errors look like (codes + recovery), and explicit when-not-to-use conditions. The error-code enumeration is especially valuable because the output schema exists but the error shape is partially described. This is as complete as a description of this complexity practically needs to be.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3, but the description adds substantive meaning beyond the schema: it explains template-purpose mapping in prose, clarifies model cost variance ('hundredfold'), states how max_cost_usd is applied ('worst case computed before the call'), and defines temperature default determinism. It doesn't fully re-explain every parameter, but it enriches the key decision parameters enough to push above baseline.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb ('Score an output'), a precise resource ('with an LLM judge on your own provider key'), and enumerates concrete outputs (0..1 score, rationale, sub-scores, spend). It clearly distinguishes itself from siblings by naming exactly what it is not (evaluate_output, verify_citations) and describing its provider-direct behavior. An agent can identify what this tool does without guessing.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description contains an explicit 'When not to use it' section that names alternatives (evaluate_output for deterministic checks) and conditions (no key -> not enabled; large outputs -> budget refusal). It also explains template selection requirements (correctness needs expected, faithfulness needs source_material) and how provider/model relate. This is comprehensive routing guidance beyond what the schema provides.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Query stored traces with filters, pagination and sorting; optionally include the dashboard summary in the same response.
What it does. Read-only, local storage only. Filters are exact-match (agent_name, framework), inclusive time bounds (since, until — an ISO 8601 timestamp or date) and a score range applied to the LATEST evaluation of each trace (min_score, max_score, 0..1). limit is 1..1000 (default 50), offset counts from 0, sort_by is timestamp, latency_ms or cost_usd, sort_order asc or desc (default: newest first). include_summary adds the one-hour dashboard aggregates. A crossed range (min above max, since after until) is refused naming both values rather than returning an empty page that reads as "no such traces".
When not to use it. To score a trace (evaluate_output). To create one (log_trace). As a live stream: this is a query, and Iris has no event stream — poll with backoff.
Returns. JSON with traces (the page of traces: trace_id, agent_name, framework, input, output, tool_calls, latency_ms, token_usage, cost_usd, metadata, timestamp); total (how many traces match the filters, across every page); limit (the page size applied); offset (the offset applied); summary (the dashboard aggregates for the last hour, when include_summary was true).
Errors. IRIS_STORAGE_ERROR when the database cannot be read. An out-of-range or crossed bound is refused before the handler runs, naming the values. An empty result is total 0, not an error. Every failure returns {"error":{"code","message","recovery":[]}} with isError true; follow recovery before retrying.
Siblings. log_trace — record an execution; evaluate_output — score one output; delete_trace — remove one trace.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Results per page (default 50, max 1000 — values above are rejected) | |
| since | No | ISO 8601 timestamp (or date) lower bound — return traces with timestamp >= this; anything that is not an ISO timestamp is rejected, never treated as "no bound" | |
| until | No | ISO 8601 timestamp (or date) upper bound — return traces with timestamp <= this; must not be earlier than `since` | |
| offset | No | Zero-based pagination offset — skip first N results (non-negative integer) | |
| sort_by | No | Sort by timestamp | latency_ms | cost_usd (default timestamp) | timestamp |
| framework | No | Filter by agent framework — exact match (e.g., langchain, autogen) | |
| max_score | No | Maximum eval score filter (0..1; values outside are rejected) — applied to LATEST eval per trace | |
| min_score | No | Minimum eval score filter (0..1; values outside are rejected) — applied to LATEST eval per trace, not all evals; must be <= max_score when both are set | |
| agent_name | No | Filter by agent name — exact match (no wildcards) | |
| sort_order | No | Sort order: asc | desc (default desc — most recent / highest first) | desc |
| include_summary | No | Include dashboard summary stats in same response — saves a round-trip when ingesting for dashboards |
| Name | Required | Description |
|---|---|---|
| limit | Yes | the page size applied |
| total | Yes | how many traces match the filters, across every page |
| offset | Yes | the offset applied |
| traces | Yes | the page of traces: trace_id, agent_name, framework, input, output, tool_calls, latency_ms, token_usage, cost_usd, metadata, timestamp |
| summary | No | the dashboard aggregates for the last hour, when include_summary was true |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint and idempotentHint, and the description reinforces and extends this by stating read-only, local-storage-only behavior. It also discloses exact-match semantics, inclusive time bounds, LATEST-eval score filtering, crossed-range refusal, error shape, empty-result behavior, and the absence of an event stream — far beyond what annotations alone convey.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but well structured with clear headings (What it does, When not to use it, Returns, Errors, Siblings) and a front-loaded summary sentence. Each section earns its place by covering important behavior, errors, and routing for an 11-parameter tool, so the length is appropriate to the complexity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity, 11 parameters, and rich annotations, the description is complete: it covers filters, pagination, sorting, optional summary, return shape, error codes, empty results, and sibling differentiation. An agent has everything needed to invoke it correctly and handle failures.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, but the description adds material meaning beyond the schema: exact-match vs wildcard behavior, inclusive bounds, score applied to the LATEST evaluation, default sort order, and the crossed-range validation rule that names both offending values. This is genuinely helpful parameter semantics rather than schema repetition.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The opening line states a specific verb and resource — 'Query stored traces with filters, pagination and sorting' — and the tool is immediately distinguished from siblings in the Siblings section. The description also names what it is not for (scoring, creating, live streaming), so an agent can tell it apart without inspecting other tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is an explicit 'When not to use it' section that names alternatives (evaluate_output, log_trace) and explains that this is a query, not a live stream, with polling guidance. This gives clear routing rules both for when to use the tool and when to choose a sibling.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
The rule inventory: the built-in roster with what each rule needs, the criticality this server applies and its published accuracy, plus every deployed custom rule.
What it does. Read-only, no network. built_in is the shipped roster and is never narrowed by the filters. For each rule: kind (measurement, detection, inference, judgment, policy, verification), mechanism, needs (the inputs it reads — absent means the rule skips, never passes), question, classes, version, weight, the EFFECTIVE critical flag with criticalSource (default, or config when eval.criticalRules / eval.nonCriticalRules changed it on this server — read it before trusting a passed: true), and proof: precision and recall with 95% intervals and the positive predictive value at four prevalences, the numbers published at https://iris-eval.com/proof. rules is the custom-rule store, filterable by eval_type and enabled_only; total and enabled_count count custom rules. quarantined lists store entries this version could not validate; they do not fire.
When not to use it. To count traces (get_traces). To add, remove or pause a rule (deploy_rule, delete_rule). Built-in rules are not in the store and cannot be deployed, deleted or disabled.
Returns. JSON with rules (the deployed custom rules after the filters: id, name, description, evalType, severity, definition, enabled, createdAt, updatedAt, version, sourceMomentId); total (custom rules after the filters); enabled_count (of those, how many are enabled); built_in (the shipped roster, never filtered: name, category, description, weight, kind, mechanism, needs, question, classes, version, the EFFECTIVE critical flag with criticalSource, and proof (published precision, recall, intervals and ppvAt from https://iris-eval.com/proof; null where the proof is a conformance check)); quarantined (entries in the store this version could not validate; they do not fire and are never deleted by a deploy).
Errors. IRIS_INTERNAL_ERROR if the store file cannot be read. A missing store file is an empty list, not an error. Every failure returns {"error":{"code","message","recovery":[]}} with isError true; follow recovery before retrying.
Siblings. deploy_rule — add a custom rule; delete_rule — remove, disable or re-enable one; evaluate_output — run the rules.
| Name | Required | Description | Default |
|---|---|---|---|
| eval_type | No | Filter the custom rules to one eval category (exact match); built_in is never filtered | |
| enabled_only | No | Return only enabled custom rules (a rule disabled with delete_rule stays in the store and does not fire) |
| Name | Required | Description |
|---|---|---|
| rules | Yes | the deployed custom rules after the filters: id, name, description, evalType, severity, definition, enabled, createdAt, updatedAt, version, sourceMomentId |
| total | Yes | custom rules after the filters |
| built_in | Yes | the shipped roster, never filtered: name, category, description, weight, kind, mechanism, needs, question, classes, version, the EFFECTIVE critical flag with criticalSource, and proof (published precision, recall, intervals and ppvAt from https://iris-eval.com/proof; null where the proof is a conformance check) |
| quarantined | Yes | entries in the store this version could not validate; they do not fire and are never deleted by a deploy |
| enabled_count | Yes | of those, how many are enabled |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare the tool read-only and idempotent, and the description adds substantial behavioral context: no network access, built_in is never filtered, filters apply only to custom rules, quarantined rules do not fire, and a missing store file is treated as an empty list rather than an error. The error response shape is also disclosed, which goes well beyond the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but exceptionally well structured with clear sections: What it does, When not to use it, Returns, Errors, and Siblings. Every section answers a likely agent question, and the most important scoping information is front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers purpose, parameter behavior, return structure, error handling, non-usage cases, and sibling relationships. Given the tool's complexity, this is complete enough for an agent to invoke it correctly and interpret its response without additional probing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Although schema coverage is 100%, the description adds important semantic detail: eval_type only filters custom rules and never built_in, and enabled_only interacts with delete_rule semantics (disabled rules stay in the store but do not fire). This goes beyond the schema's short descriptions and clarifies edge behavior.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly defines the tool as a rule inventory covering both built-in and custom rules, listing exact fields returned and the distinction between the two stores. It also explicitly names sibling tools to contrast with, so an agent can select this tool without ambiguity.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The 'When not to use it' section names the exact alternatives (get_traces, deploy_rule, delete_rule) and explains that built-in rules cannot be modified. This gives an agent concrete routing guidance and exclusion criteria, which is more than most tool descriptions provide.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Store one agent execution — input, output, tool calls, spans, cost, latency, token usage — and get the trace_id every later call keys on.
What it does. Writes one trace row to local SQLite and mints a fresh trace_id; nothing is deduplicated, so resubmitting the same payload stores a second trace. Only agent_name is required. Store what you have: tool_calls so the trajectory rules can later judge what the agent did, cost_usd and token_usage so the cost rules can, input and output so everything else can. When IRIS_OTEL_ENDPOINT is set the trace is also exported to that collector, best-effort and asynchronous; the local write never waits on it. Traces are immutable: there is no update path. In stdio mode nothing authenticates the caller; over HTTP a Bearer token is required only when an API key is configured.
When not to use it. For a transient log line (use your logger). To score an output: log first, then call evaluate_output with the trace_id, which also lets it reuse the stored tool_calls. To change a stored trace: delete_trace and log again.
Returns. JSON with trace_id (the stored trace id, 32 hex — pass it to evaluate_output, get_traces or delete_trace); status (always "stored" on success).
Errors. IRIS_STORAGE_ERROR when the database cannot be written. An unknown argument or a malformed span or tool_calls entry is refused before the handler runs, naming the valid keys. Every failure returns {"error":{"code","message","recovery":[]}} with isError true; follow recovery before retrying.
Siblings. evaluate_output — score the stored output; get_traces — query what was logged; delete_trace — remove one trace.
| Name | Required | Description | Default |
|---|---|---|---|
| input | No | Agent input text — the user prompt or upstream input that produced this output | |
| spans | No | Detailed execution spans (hierarchical span tree with timings, attributes, events); a span without start_time takes the trace timestamp | |
| output | No | Agent output text — what the agent produced (pass to evaluate_output for scoring) | |
| cost_usd | No | Total cost in USD — overrides per-span aggregation when provided (treated as authoritative) | |
| metadata | No | Opaque key-value tags (e.g. {requestId, userId, env}) — queryable in dashboard, not via get_traces filters | |
| framework | No | Agent framework identifier (e.g., langchain, autogen, custom) | |
| timestamp | No | Trace timestamp (ISO 8601); defaults to now() when omitted | |
| agent_name | Yes | Agent name — used for filtering in get_traces (e.g., "customer-support-bot") | |
| latency_ms | No | Total execution time in milliseconds (end-to-end agent latency) | |
| tool_calls | No | Tool calls made during execution, in order, each { tool_name, input?, output?, latency_ms?, error? } — what the trajectory rules judge; evaluate_output reuses them when given this trace_id | |
| token_usage | No | Token usage breakdown (prompt/completion/total — used for cost analysis) |
| Name | Required | Description |
|---|---|---|
| status | Yes | always "stored" on success |
| trace_id | Yes | the stored trace id, 32 hex — pass it to evaluate_output, get_traces or delete_trace |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only indicate readOnlyHint=false, openWorldHint=false, idempotentHint=false, destructiveHint=false. The description goes far beyond those hints: it discloses that traces are immutable with no update path, that resubmission stores a second trace (no deduplication), that the local write never waits on the best-effort async OTel export, and that authentication is optional in stdio mode but requires a Bearer token over HTTP when an API key is configured. It also explains error behavior and return shape. No contradiction with annotations; the description carries the full burden and succeeds.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but every section earns its place and is heavily front-loaded: the first sentence states the core action and the returned key. The 'What it does', 'When not to use it', 'Returns', 'Errors', and 'Siblings' labels give it scannable structure. It is verbose, but the density of decision-relevant information justifies the length. Loses one point for minor redundancy (e.g., restating evaluate_output reuse in both the What and param description).
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given an output schema exists (describing the return JSON), 11 parameters with 100% schema coverage, and a rich nested schema, the description is complete. It covers the operation's side effects (immutable write, second trace on resubmit), error contract, authentication nuance, and sibling routing. An agent has everything needed to decide when to call log_trace and what to expect.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents all 11 parameters in detail (e.g., agent_name is used for filtering in get_traces, cost_usd overrides per-span aggregation, metadata is queryable in dashboard but not via get_traces filters). The description adds cross-parameter context (tool_calls for trajectory rules, cost_usd/token_usage for cost rules, input/output for everything else), but much of that is already in the field descriptions. Baseline 3 is appropriate: the description complements rather than compensates.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a precise verb-object pair, 'Store one agent execution', and enumerates exactly what a trace captures (input, output, tool calls, spans, cost, latency, token usage) and what it returns (trace_id). It clearly distinguishes log_trace from siblings like evaluate_output and get_traces by naming them and their purposes. This is a specific, actionable statement of what the tool does.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The 'When not to use it' section explicitly routes the agent: use a logger for transient log lines, call evaluate_output after logging to score, and use delete_trace then log again to modify a stored trace. It also gives a conditional alternative (IRIS_OTEL_ENDPOINT export behavior). This is exemplary guidance for selecting log_trace versus alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Extract the citations in an output, fetch the sources (opt-in, SSRF-guarded) and ask an LLM judge on your key whether each source supports its claim.
What it does. Three phases. Extraction, no network: [N] references, (Author, Year), bare URLs and DOIs. Fetch of URL and DOI citations only when allow_fetch is true or IRIS_CITATION_ALLOW_FETCH=1, through a scheme allowlist, private and cloud-metadata address blocking, an optional hostname allowlist (domain_allowlist, merged with IRIS_CITATION_DOMAINS), a per-source timeout and byte cap, and at most three re-checked redirects. Then one judge call per resolved citation on your own key, reading the first part of each source, capped in total by max_cost_usd_total. Up to max_citations are verified; extras are skipped, not errored. overall_score is supported / judged and null when nothing was judged. Per-citation failures (bad scheme, blocked address, timeout, too large, cost cap, fetch disabled) are reported on the citation, never scored as unsupported. One evaluation row is stored.
When not to use it. When the output has no citations: the score is null, and evaluate_output's hallucination signals are the cheap check. Without a key (IRIS_ANTHROPIC_API_KEY or IRIS_OPENAI_API_KEY): the call returns IRIS_JUDGE_NOT_ENABLED with the enable steps. With fetch enabled and an open allowlist on untrusted output: you are running a user-directed fetcher — set IRIS_CITATION_DOMAINS.
Returns. JSON with id (the evaluation id; read it back at iris://evaluations/{id}); trace_id (the linked trace, when one was named); overall_score (supported / judged; null when nothing was judged); passed (true when every judged citation was supported; false when any judged citation was not; NULL when nothing was judged — no verdict, because nothing was verified. Until 0.10.0 that last case returned true.); total_unsupported (judged citations the judge ruled unsupported — the number the verdict turns on); total_citations_found (citations extracted from the output); total_resolved (citations whose source was fetched); total_judged (citations the judge ruled on); total_supported (citations the judge found supported); total_cost_usd (the spend across every judge call); citations (per citation: the citation (raw, kind, identifier, offsets), resolve_status ok | skipped | error, resolve_error, source (url, status, content_type, bytes_fetched, truncated), judge (supported, confidence, rationale, cost_usd, latency_ms, tokens)).
Errors. IRIS_JUDGE_NOT_ENABLED, IRIS_JUDGE_UNKNOWN_MODEL and IRIS_UNKNOWN_TRACE before any fetch or spend. IRIS_JUDGE_FAILED when citations resolved but the judge failed on every one — an error, not a passing verdict; nothing is stored. Every failure returns {"error":{"code","message","recovery":[]}} with isError true; follow recovery before retrying.
Siblings. evaluate_with_llm_judge — general semantic scoring; evaluate_output — the free deterministic path, including the hallucination signals; log_trace — record the execution first.
| Name | Required | Description | Default |
|---|---|---|---|
| model | Yes | Judge model for per-citation verification. Supported: anthropic = claude-opus-4-7 | claude-sonnet-4-6 | claude-haiku-4-5 | claude-haiku-4-5-20251001; openai = gpt-4o | gpt-4o-mini | o1-mini. | |
| output | Yes | The agent output containing citations to verify | |
| provider | No | Auto-detected from model when omitted | |
| trace_id | No | Link verification result to a stored trace (id from log_trace / get_traces); an unknown id is rejected before any fetch or judge call | |
| allow_fetch | No | Permit outbound HTTP to resolve URLs/DOIs. Defaults to IRIS_CITATION_ALLOW_FETCH=1; false otherwise. SSRF-guarded regardless. | |
| max_citations | No | Max citations to verify (extras skipped, not errored); default 20, at most 50 | |
| domain_allowlist | No | Restrict fetches to hostnames in this list (suffix match allowed). Merged with IRIS_CITATION_DOMAINS env. | |
| max_cost_usd_total | No | Cap TOTAL judge cost across all citations in this call; default 1.00 USD — the pipeline stops when the next call would exceed it | |
| per_source_max_bytes | No | Per-URL body cap; default 5MB | |
| per_source_timeout_ms | No | Per-URL fetch timeout; default 10_000 |
| Name | Required | Description |
|---|---|---|
| id | Yes | the evaluation id; read it back at iris://evaluations/{id} |
| passed | Yes | true when every judged citation was supported; false when any judged citation was not; NULL when nothing was judged — no verdict, because nothing was verified. Until 0.10.0 that last case returned true. |
| trace_id | No | the linked trace, when one was named |
| citations | Yes | per citation: the citation (raw, kind, identifier, offsets), resolve_status ok | skipped | error, resolve_error, source (url, status, content_type, bytes_fetched, truncated), judge (supported, confidence, rationale, cost_usd, latency_ms, tokens) |
| total_judged | Yes | citations the judge ruled on |
| overall_score | Yes | supported / judged; null when nothing was judged |
| total_cost_usd | Yes | the spend across every judge call |
| total_resolved | Yes | citations whose source was fetched |
| total_supported | Yes | citations the judge found supported |
| total_unsupported | Yes | judged citations the judge ruled unsupported — the number the verdict turns on |
| total_citations_found | Yes | citations extracted from the output |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond annotations, the description discloses: fetch is opt-in via allow_fetch or env var, SSRF protections (scheme allowlist, blocked private/cloud-metadata addresses, hostname allowlist, timeouts, byte caps, redirect limits), cost caps, per-citation failures never scored as unsupported, one evaluation row stored, and the null-verdict behavior including a 0.10.0 change note. It also explains errors like IRIS_JUDGE_FAILED. Annotations are minimal (readOnly=false, openWorld=true, idempotent=false, destructive=false), so the description carries the burden and does so thoroughly.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but earns its length: it uses labeled sections (What it does, When not to use it, Returns, Errors, Siblings) and front-loads the headline behavior. Some redundancy exists (e.g., 'When not to use it' repeats the key requirement mentioned in the opener), and the Returns section is dense, but the structure makes it navigable.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 10-parameter, complex tool with an output schema, the description covers the pipeline phases, error codes, env var interactions, security posture, edge cases, and return fields. The output schema exists, so the description need not spell out return shapes, but it still summarizes them and adds the critical behavioral notes about passed/NULL and stored evaluations. Nothing essential is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema documents every parameter meaning. The description adds operational context that interacts with parameters — e.g., allow_fetch defaulting from env, domain_allowlist merging with env, max_citations extras being skipped, max_cost_usd_total behavior — but the core semantic meaning per parameter is already in the schema. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb phrase — 'Extract the citations in an output, fetch the sources (opt-in, SSRF-guarded) and ask an LLM judge on your key whether each source supports its claim.' This precisely states the tool's operation and resource. It differentiates from siblings by name ('verify_citations') and behavior, and the Siblings section explicitly distinguishes it from evaluate_with_llm_judge and evaluate_output.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description includes an explicit 'When not to use it' section naming concrete conditions: no citations, no key, and the risk of an open allowlist on untrusted output. It even points to evaluate_output's hallucination signals as the cheap alternative. This is exemplary guidance for when to use the tool vs alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
v0.10.0evaluate_with_llm_judge5 fields changedOutput schema / properties / disagreementAdded value: +{
+ "description": "true when the model's own boolean disagrees with the threshold verdict — its rubric and its judgement have come apart on this output",
+ "type": "boolean"
+}Output schema / properties / pass_thresholdAdded value: +{
+ "description": "the threshold the score was read against, so you can check the arithmetic",
+ "type": "number"
+}Output schema / properties / passed / descriptionPrevious value: -"the judge's own pass verdict for the template"New value: +"the verdict: the score against the template's threshold, which is pass_threshold below. Not the model's own boolean — that is self_reported_pass"Output schema / properties / self_reported_passAdded value: +{
+ "description": "what the model said about passing, when it said anything. Recorded, never obeyed",
+ "type": "boolean"
+}Output schema / requiredPrevious value: -[
- "id",
- "score",
- "passed",
- "rationale",
- "dimensions",
- "model",
- "provider",
- "template",
- "input_tokens",
- "output_tokens",
- "cost_usd",
- "latency_ms"
-]New value: +[
+ "id",
+ "score",
+ "passed",
+ "pass_threshold",
+ "rationale",
+ "dimensions",
+ "model",
+ "provider",
+ "template",
+ "input_tokens",
+ "output_tokens",
+ "cost_usd",
+ "latency_ms"
+]verify_citations4 fields changedOutput schema / properties / passed / descriptionPrevious value: -"true when every judged citation was supported, or nothing was judged and nothing failed"New value: +"true when every judged citation was supported; false when any judged citation was not; NULL when nothing was judged — no verdict, because nothing was verified. Until 0.10.0 that last case returned true."Output schema / properties / passed / typePrevious value: -"boolean"New value: +[
+ "boolean",
+ "null"
+]Output schema / properties / total_unsupportedAdded value: +{
+ "description": "judged citations the judge ruled unsupported — the number the verdict turns on",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+}Output schema / requiredPrevious value: -[
- "id",
- "overall_score",
- "passed",
- "total_citations_found",
- "total_resolved",
- "total_judged",
- "total_supported",
- "total_cost_usd",
- "citations"
-]New value: +[
+ "id",
+ "overall_score",
+ "passed",
+ "total_unsupported",
+ "total_citations_found",
+ "total_resolved",
+ "total_judged",
+ "total_supported",
+ "total_cost_usd",
+ "citations"
+]v0.9.0delete_rule1 field changedOutput schema / (root)Previous value: -nullNew value: +{
+ "$schema": "http://json-schema.org/draft-07/schema#",
+ "additionalProperties": {},
+ "properties": {
+ "deleted": {
+ "description": "true when a rule was removed; always false on a toggle",
+ "type": "boolean"
+ },
+ "enabled": {
+ "description": "toggle only: the rule's state after the call",
+ "type": "boolean"
+ },
+ "rule": {
+ "additionalProperties": {},
+ "description": "toggle only: the rule as stored",
+ "properties": {},
+ "type": "object"
+ },
+ "rule_id": {
+ "description": "the id that was asked for",
+ "type": "string"
+ },
+ "toggled": {
+ "description": "toggle only: true when the rule exists (also when it was already in the requested state)",
+ "type": "boolean"
+ }
+ },
+ "required": [
+ "deleted",
+ "rule_id"
+ ],
+ "type": "object"
+}delete_trace1 field changedOutput schema / (root)Previous value: -nullNew value: +{
+ "$schema": "http://json-schema.org/draft-07/schema#",
+ "additionalProperties": {},
+ "properties": {
+ "deleted": {
+ "description": "true when a trace row was removed; false when no trace with that id existed for this tenant",
+ "type": "boolean"
+ },
+ "trace_id": {
+ "description": "the id that was asked for",
+ "type": "string"
+ }
+ },
+ "required": [
+ "deleted",
+ "trace_id"
+ ],
+ "type": "object"
+}deploy_rule1 field changedOutput schema / (root)Previous value: -nullNew value: +{
+ "$schema": "http://json-schema.org/draft-07/schema#",
+ "additionalProperties": {},
+ "properties": {
+ "replaced": {
+ "description": "with replace: true, the earlier rule(s) of the same name that were retired",
+ "items": {
+ "additionalProperties": {},
+ "properties": {
+ "id": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "id"
+ ],
+ "type": "object"
+ },
+ "type": "array"
+ },
+ "rule": {
+ "additionalProperties": {},
+ "description": "the rule as persisted: id (rule-<hex>, keep it for delete_rule), name, description, evalType, severity, definition, enabled, createdAt, updatedAt, version, sourceMomentId",
+ "properties": {
+ "id": {
+ "type": "string"
+ },
+ "name": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "id",
+ "name"
+ ],
+ "type": "object"
+ },
+ "warning": {
+ "description": "with replace: true, one sentence naming what was retired",
+ "type": "string"
+ }
+ },
+ "required": [
+ "rule"
+ ],
+ "type": "object"
+}evaluate_output2 fields changedInput schema / properties / trace_id / descriptionPrevious value: -"Link evaluation to a trace — surfaces this eval in the dashboard's trace drill-through. Must be the id of a stored trace (from log_trace / get_traces); an unknown id is rejected before anything is evaluated"New value: +"Link evaluation to a trace — surfaces this eval in the dashboard's trace drill-through and lets the tool reuse the trace's stored tool_calls. Must be the id of a stored trace (from log_trace / get_traces); an unknown id is rejected before anything is evaluated"Output schema / (root)Previous value: -nullNew value: +{
+ "$schema": "http://json-schema.org/draft-07/schema#",
+ "additionalProperties": {},
+ "properties": {
+ "categories": {
+ "additionalProperties": {
+ "additionalProperties": {},
+ "properties": {
+ "critical_failures": {
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "critical_skipped": {
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "insufficient_data": {
+ "type": "boolean"
+ },
+ "passed": {
+ "type": [
+ "boolean",
+ "null"
+ ]
+ },
+ "rules_evaluated": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "rules_skipped": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "score": {
+ "type": [
+ "number",
+ "null"
+ ]
+ }
+ },
+ "required": [
+ "score",
+ "passed",
+ "rules_evaluated",
+ "rules_skipped",
+ "insufficient_data"
+ ],
+ "type": "object"
+ },
+ "description": "per-bundle verdicts for eval_type all",
+ "propertyNames": {
+ "type": "string"
+ },
+ "type": "object"
+ },
+ "coverage": {
+ "additionalProperties": {},
+ "description": "per question: judged, unjudged and why, or not_applicable; plus the inputs carried",
+ "properties": {
+ "dormant": {
+ "items": {
+ "additionalProperties": {},
+ "properties": {
+ "name": {
+ "type": "string"
+ },
+ "reason": {
+ "type": "string"
+ },
+ "ruleId": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "ruleId",
+ "name",
+ "reason"
+ ],
+ "type": "object"
+ },
+ "type": "array"
+ },
+ "inputs": {
+ "additionalProperties": {
+ "type": "boolean"
+ },
+ "propertyNames": {
+ "type": "string"
+ },
+ "type": "object"
+ },
+ "questions": {
+ "items": {
+ "additionalProperties": {},
+ "properties": {
+ "id": {
+ "enum": [
+ "safe_output",
+ "grounded",
+ "complete",
+ "relevant",
+ "task_completed",
+ "tool_use_correct",
+ "within_budget"
+ ],
+ "type": "string"
+ },
+ "status": {
+ "enum": [
+ "judged",
+ "unjudged",
+ "not_applicable"
+ ],
+ "type": "string"
+ },
+ "why": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "id",
+ "status"
+ ],
+ "type": "object"
+ },
+ "type": "array"
+ }
+ },
+ "required": [
+ "inputs",
+ "questions"
+ ],
+ "type": "object"
+ },
+ "critical_failures": {
+ "description": "critical rules that failed and vetoed passed",
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "critical_skipped": {
+ "description": "critical rules that could not judge; treat as unknown",
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "erased_at": {
+ "description": "set once the linked trace was deleted",
+ "type": "string"
+ },
+ "eval_type": {
+ "description": "the bundle that ran",
+ "enum": [
+ "completeness",
+ "relevance",
+ "safety",
+ "cost",
+ "custom",
+ "all"
+ ],
+ "type": "string"
+ },
+ "id": {
+ "description": "the evaluation id, readable at iris://evaluations/{id}",
+ "type": "string"
+ },
+ "insufficient_data": {
+ "description": "true when no rule could judge",
+ "type": "boolean"
+ },
+ "note": {
+ "description": "present when eval_type was omitted",
+ "type": "string"
+ },
+ "passed": {
+ "description": "the ship verdict; false when nothing was judged",
+ "type": "boolean"
+ },
+ "provenance": {
+ "additionalProperties": {},
+ "description": "Iris version, ruleset and config hashes, thresholds, corpus version, time",
+ "properties": {
+ "configHash": {
+ "type": "string"
+ },
+ "corpusVersion": {
+ "type": "string"
+ },
+ "irisVersion": {
+ "type": "string"
+ },
+ "judgedAt": {
+ "type": "string"
+ },
+ "rulesetHash": {
+ "type": "string"
+ },
+ "thresholds": {
+ "additionalProperties": {},
+ "properties": {
+ "default": {
+ "type": "number"
+ },
+ "perRule": {
+ "additionalProperties": {},
+ "propertyNames": {
+ "type": "string"
+ },
+ "type": "object"
+ }
+ },
+ "required": [
+ "default"
+ ],
+ "type": "object"
+ }
+ },
+ "required": [
+ "irisVersion",
+ "rulesetHash",
+ "configHash",
+ "thresholds",
+ "corpusVersion",
+ "judgedAt"
+ ],
+ "type": "object"
+ },
+ "rule_results": {
+ "description": "per rule: verdict, message, kind, role, question, saw, evidence, uncertainty",
+ "items": {
+ "additionalProperties": {},
+ "properties": {
+ "budgetExceeded": {
+ "type": "boolean"
+ },
+ "category": {
+ "enum": [
+ "completeness",
+ "relevance",
+ "safety",
+ "cost",
+ "custom"
+ ],
+ "type": "string"
+ },
+ "classes": {
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "configInvalid": {
+ "type": "boolean"
+ },
+ "critical": {
+ "type": "boolean"
+ },
+ "criticalSource": {
+ "enum": [
+ "default",
+ "config"
+ ],
+ "type": "string"
+ },
+ "evidence": {
+ "items": {
+ "oneOf": [
+ {
+ "additionalProperties": {},
+ "properties": {
+ "end": {
+ "maximum": 9007199254740991,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "label": {
+ "type": "string"
+ },
+ "source": {
+ "type": "string"
+ },
+ "start": {
+ "maximum": 9007199254740991,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "type": {
+ "const": "span",
+ "type": "string"
+ }
+ },
+ "required": [
+ "type",
+ "source",
+ "start",
+ "end",
+ "label"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": {},
+ "properties": {
+ "count": {
+ "maximum": 9007199254740991,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "name": {
+ "type": "string"
+ },
+ "type": {
+ "const": "pattern",
+ "type": "string"
+ }
+ },
+ "required": [
+ "type",
+ "name",
+ "count"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": {},
+ "properties": {
+ "index": {
+ "maximum": 9007199254740991,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "label": {
+ "type": "string"
+ },
+ "toolName": {
+ "type": "string"
+ },
+ "type": {
+ "const": "toolCall",
+ "type": "string"
+ }
+ },
+ "required": [
+ "type",
+ "index",
+ "toolName",
+ "label"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": {},
+ "properties": {
+ "status": {
+ "enum": [
+ "resolved",
+ "dead",
+ "unverifiable",
+ "supported",
+ "unsupported"
+ ],
+ "type": "string"
+ },
+ "type": {
+ "const": "citation",
+ "type": "string"
+ },
+ "url": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "type",
+ "url",
+ "status"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": {},
+ "properties": {
+ "stat": {
+ "type": "string"
+ },
+ "threshold": {
+ "type": "number"
+ },
+ "thresholdSource": {
+ "enum": [
+ "default",
+ "config",
+ "call",
+ "rule"
+ ],
+ "type": "string"
+ },
+ "type": {
+ "const": "count",
+ "type": "string"
+ },
+ "unit": {
+ "type": "string"
+ },
+ "value": {
+ "type": "number"
+ }
+ },
+ "required": [
+ "type",
+ "stat",
+ "unit",
+ "value"
+ ],
+ "type": "object"
+ }
+ ]
+ },
+ "type": "array"
+ },
+ "kind": {
+ "enum": [
+ "measurement",
+ "detection",
+ "inference",
+ "judgment",
+ "policy",
+ "verification"
+ ],
+ "type": "string"
+ },
+ "message": {
+ "type": "string"
+ },
+ "passed": {
+ "type": "boolean"
+ },
+ "question": {
+ "enum": [
+ "safe_output",
+ "grounded",
+ "complete",
+ "relevant",
+ "task_completed",
+ "tool_use_correct",
+ "within_budget"
+ ],
+ "type": "string"
+ },
+ "role": {
+ "enum": [
+ "gate",
+ "veto",
+ "risk",
+ "advisory",
+ "term"
+ ],
+ "type": "string"
+ },
+ "ruleId": {
+ "type": "string"
+ },
+ "ruleName": {
+ "type": "string"
+ },
+ "ruleVersion": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "saw": {
+ "items": {
+ "enum": [
+ "output",
+ "input",
+ "expected",
+ "tool_calls",
+ "tool_outputs",
+ "tools_catalogue",
+ "cost",
+ "tokens",
+ "citations"
+ ],
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "score": {
+ "type": "number"
+ },
+ "skipClass": {
+ "enum": [
+ "not_applicable",
+ "defeated",
+ "config_invalid"
+ ],
+ "type": "string"
+ },
+ "skipReason": {
+ "type": "string"
+ },
+ "skipped": {
+ "type": "boolean"
+ },
+ "uncertainty": {
+ "oneOf": [
+ {
+ "additionalProperties": {},
+ "properties": {
+ "basis": {
+ "const": "published_accuracy",
+ "type": "string"
+ },
+ "corpus": {
+ "additionalProperties": {},
+ "properties": {
+ "fn": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "fp": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "labelling": {
+ "enum": [
+ "same-model",
+ "human-verified"
+ ],
+ "type": "string"
+ },
+ "n": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "release": {
+ "type": "string"
+ },
+ "tn": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "tp": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "version": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "n",
+ "tp",
+ "fp",
+ "fn",
+ "tn",
+ "version",
+ "release",
+ "labelling"
+ ],
+ "type": "object"
+ },
+ "fired": {
+ "type": "boolean"
+ },
+ "missRate": {
+ "additionalProperties": {},
+ "properties": {
+ "hi": {
+ "type": "number"
+ },
+ "lo": {
+ "type": "number"
+ },
+ "point": {
+ "type": "number"
+ }
+ },
+ "required": [
+ "point",
+ "lo",
+ "hi"
+ ],
+ "type": "object"
+ },
+ "ppv": {
+ "additionalProperties": {},
+ "properties": {
+ "hi": {
+ "type": "number"
+ },
+ "lo": {
+ "type": "number"
+ },
+ "point": {
+ "type": "number"
+ }
+ },
+ "required": [
+ "point",
+ "lo",
+ "hi"
+ ],
+ "type": "object"
+ },
+ "prior": {
+ "additionalProperties": {},
+ "properties": {
+ "pi": {
+ "type": "number"
+ },
+ "source": {
+ "enum": [
+ "default",
+ "config",
+ "estimated"
+ ],
+ "type": "string"
+ }
+ },
+ "required": [
+ "pi",
+ "source"
+ ],
+ "type": "object"
+ }
+ },
+ "required": [
+ "basis",
+ "fired",
+ "prior",
+ "corpus"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": {},
+ "properties": {
+ "basis": {
+ "const": "definition",
+ "type": "string"
+ },
+ "conformance": {
+ "additionalProperties": {},
+ "properties": {
+ "matched": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "n": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ }
+ },
+ "required": [
+ "n",
+ "matched"
+ ],
+ "type": "object"
+ }
+ },
+ "required": [
+ "basis",
+ "conformance"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": {},
+ "properties": {
+ "basis": {
+ "const": "self_consistency",
+ "type": "string"
+ },
+ "samples": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "scoreSd": {
+ "type": "number"
+ },
+ "voteFraction": {
+ "type": "number"
+ }
+ },
+ "required": [
+ "basis",
+ "samples",
+ "voteFraction",
+ "scoreSd"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": {},
+ "properties": {
+ "basis": {
+ "const": "local_labels",
+ "type": "string"
+ },
+ "n": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "precision": {
+ "additionalProperties": {},
+ "properties": {
+ "hi": {
+ "type": "number"
+ },
+ "lo": {
+ "type": "number"
+ },
+ "point": {
+ "type": "number"
+ }
+ },
+ "required": [
+ "point",
+ "lo",
+ "hi"
+ ],
+ "type": "object"
+ }
+ },
+ "required": [
+ "basis",
+ "precision",
+ "n"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": {},
+ "properties": {
+ "basis": {
+ "const": "policy",
+ "type": "string"
+ }
+ },
+ "required": [
+ "basis"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": {},
+ "properties": {
+ "basis": {
+ "const": "unmeasured",
+ "type": "string"
+ },
+ "why": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "basis",
+ "why"
+ ],
+ "type": "object"
+ }
+ ]
+ },
+ "value": {
+ "additionalProperties": {},
+ "properties": {
+ "stat": {
+ "type": "string"
+ },
+ "unit": {
+ "type": "string"
+ },
+ "value": {
+ "type": "number"
+ }
+ },
+ "required": [
+ "stat",
+ "unit",
+ "value"
+ ],
+ "type": "object"
+ }
+ },
+ "required": [
+ "ruleName",
+ "passed",
+ "score",
+ "message"
+ ],
+ "type": "object"
+ },
+ "type": "array"
+ },
+ "rules_evaluated": {
+ "description": "rules that judged",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "rules_skipped": {
+ "description": "rules that skipped",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "score": {
+ "description": "0..1 weighted quality over the rules that ran",
+ "type": "number"
+ },
+ "suggestions": {
+ "description": "what to change",
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "trace_id": {
+ "description": "the linked trace, when named",
+ "type": "string"
+ },
+ "verdict": {
+ "additionalProperties": {},
+ "description": "state, passed, basis (which layer decided), by (the rules), risk",
+ "properties": {
+ "basis": {
+ "enum": [
+ "policy_gate",
+ "detector_veto",
+ "critical_unknown",
+ "required_evidence_missing",
+ "risk_over_loss",
+ "score_below_threshold",
+ "clean",
+ "no_rules"
+ ],
+ "type": "string"
+ },
+ "by": {
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "confidence": {
+ "enum": [
+ "decisive",
+ "marginal"
+ ],
+ "type": "string"
+ },
+ "passed": {
+ "type": "boolean"
+ },
+ "risk": {
+ "anyOf": [
+ {
+ "additionalProperties": {},
+ "properties": {
+ "hi": {
+ "type": "number"
+ },
+ "lo": {
+ "type": "number"
+ },
+ "pBad": {
+ "type": "number"
+ }
+ },
+ "required": [
+ "pBad",
+ "lo",
+ "hi"
+ ],
+ "type": "object"
+ },
+ {
+ "type": "null"
+ }
+ ]
+ },
+ "state": {
+ "enum": [
+ "pass",
+ "fail",
+ "unknown"
+ ],
+ "type": "string"
+ }
+ },
+ "required": [
+ "state",
+ "passed",
+ "basis",
+ "by",
+ "risk"
+ ],
+ "type": "object"
+ }
+ },
+ "required": [
+ "id",
+ "eval_type",
+ "score",
+ "passed",
+ "rule_results",
+ "suggestions",
+ "rules_evaluated",
+ "rules_skipped",
+ "insufficient_data"
+ ],
+ "type": "object"
+}evaluate_with_llm_judge3 fields changedInput schema / properties / max_cost_usd / descriptionPrevious value: -"Cost cap in USD; defaults to IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL or 0.25"New value: +"Cost cap in USD for this call; defaults to IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL or 0.25. The worst case (two attempts, full max_output_tokens) is computed before the call and refused if it exceeds the cap"Input schema / properties / model / descriptionPrevious value: -"Model ID. Supported: anthropic = claude-opus-4-7 | claude-sonnet-4-6 | claude-haiku-4-5 | claude-haiku-4-5-20251001; openai = gpt-4o | gpt-4o-mini | o1-mini."New value: +"Model ID. Supported: anthropic = claude-opus-4-7 | claude-sonnet-4-6 | claude-haiku-4-5 | claude-haiku-4-5-20251001; openai = gpt-4o | gpt-4o-mini | o1-mini. Required — cost varies a hundredfold across models"Output schema / (root)Previous value: -nullNew value: +{
+ "$schema": "http://json-schema.org/draft-07/schema#",
+ "additionalProperties": {},
+ "properties": {
+ "cost_usd": {
+ "description": "the exact spend from the pricing table",
+ "type": [
+ "number",
+ "null"
+ ]
+ },
+ "dimensions": {
+ "additionalProperties": {},
+ "description": "per-dimension sub-scores for the template",
+ "propertyNames": {
+ "type": "string"
+ },
+ "type": "object"
+ },
+ "id": {
+ "description": "the evaluation id; read it back at iris://evaluations/{id}",
+ "type": "string"
+ },
+ "input_tokens": {
+ "description": "tokens sent, across both attempts when a retry ran",
+ "type": "number"
+ },
+ "latency_ms": {
+ "description": "wall time of the provider call(s)",
+ "type": "number"
+ },
+ "model": {
+ "description": "the model that judged",
+ "type": "string"
+ },
+ "output_tokens": {
+ "description": "tokens received, across both attempts when a retry ran",
+ "type": "number"
+ },
+ "passed": {
+ "description": "the judge's own pass verdict for the template",
+ "type": "boolean"
+ },
+ "provider": {
+ "description": "the provider called",
+ "enum": [
+ "anthropic",
+ "openai"
+ ],
+ "type": "string"
+ },
+ "rationale": {
+ "description": "the judge's reasoning, in its words",
+ "type": "string"
+ },
+ "raw_response_id": {
+ "description": "the provider's response id, for your own audit",
+ "type": "string"
+ },
+ "score": {
+ "description": "0..1 from the judge",
+ "type": "number"
+ },
+ "template": {
+ "description": "the template used",
+ "type": "string"
+ },
+ "trace_id": {
+ "description": "the linked trace, when one was named",
+ "type": "string"
+ }
+ },
+ "required": [
+ "id",
+ "score",
+ "passed",
+ "rationale",
+ "dimensions",
+ "model",
+ "provider",
+ "template",
+ "input_tokens",
+ "output_tokens",
+ "cost_usd",
+ "latency_ms"
+ ],
+ "type": "object"
+}get_traces3 fields changedInput schema / properties / agent_name / descriptionPrevious value: -"Filter by agent name — exact match (no wildcards in v0.4)"New value: +"Filter by agent name — exact match (no wildcards)"Input schema / properties / limit / descriptionPrevious value: -"Results per page (default 50, max 1000 — values >1000 return 400)"New value: +"Results per page (default 50, max 1000 — values above are rejected)"Output schema / (root)Previous value: -nullNew value: +{
+ "$schema": "http://json-schema.org/draft-07/schema#",
+ "additionalProperties": {},
+ "properties": {
+ "limit": {
+ "description": "the page size applied",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "offset": {
+ "description": "the offset applied",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "summary": {
+ "additionalProperties": {},
+ "description": "the dashboard aggregates for the last hour, when include_summary was true",
+ "properties": {},
+ "type": "object"
+ },
+ "total": {
+ "description": "how many traces match the filters, across every page",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "traces": {
+ "description": "the page of traces: trace_id, agent_name, framework, input, output, tool_calls, latency_ms, token_usage, cost_usd, metadata, timestamp",
+ "items": {
+ "additionalProperties": {},
+ "properties": {
+ "trace_id": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "trace_id"
+ ],
+ "type": "object"
+ },
+ "type": "array"
+ }
+ },
+ "required": [
+ "traces",
+ "total",
+ "limit",
+ "offset"
+ ],
+ "type": "object"
+}list_rules3 fields changedInput schema / properties / enabled_only / descriptionPrevious value: -"Return only enabled rules (excludes disabled ones)"New value: +"Return only enabled custom rules (a rule disabled with delete_rule stays in the store and does not fire)"Input schema / properties / eval_type / descriptionPrevious value: -"Filter to rules of a specific eval category"New value: +"Filter the custom rules to one eval category (exact match); built_in is never filtered"Output schema / (root)Previous value: -nullNew value: +{
+ "$schema": "http://json-schema.org/draft-07/schema#",
+ "additionalProperties": {},
+ "properties": {
+ "built_in": {
+ "description": "the shipped roster, never filtered: name, category, description, weight, kind, mechanism, needs, question, classes, version, the EFFECTIVE critical flag with criticalSource, and proof (published precision, recall, intervals and ppvAt from https://iris-eval.com/proof; null where the proof is a conformance check)",
+ "items": {
+ "additionalProperties": {},
+ "properties": {
+ "name": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "name"
+ ],
+ "type": "object"
+ },
+ "type": "array"
+ },
+ "enabled_count": {
+ "description": "of those, how many are enabled",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "quarantined": {
+ "description": "entries in the store this version could not validate; they do not fire and are never deleted by a deploy",
+ "items": {},
+ "type": "array"
+ },
+ "rules": {
+ "description": "the deployed custom rules after the filters: id, name, description, evalType, severity, definition, enabled, createdAt, updatedAt, version, sourceMomentId",
+ "items": {
+ "additionalProperties": {},
+ "properties": {
+ "id": {
+ "type": "string"
+ },
+ "name": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "id",
+ "name"
+ ],
+ "type": "object"
+ },
+ "type": "array"
+ },
+ "total": {
+ "description": "custom rules after the filters",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ }
+ },
+ "required": [
+ "rules",
+ "total",
+ "enabled_count",
+ "built_in",
+ "quarantined"
+ ],
+ "type": "object"
+}log_trace3 fields changedInput schema / properties / spans / descriptionPrevious value: -"Detailed execution spans (hierarchical span tree with timings, attributes, events)"New value: +"Detailed execution spans (hierarchical span tree with timings, attributes, events); a span without start_time takes the trace timestamp"Input schema / properties / tool_calls / descriptionPrevious value: -"Tool calls made during execution (per-call latency, errors, input/output)"New value: +"Tool calls made during execution, in order, each { tool_name, input?, output?, latency_ms?, error? } — what the trajectory rules judge; evaluate_output reuses them when given this trace_id"Output schema / (root)Previous value: -nullNew value: +{
+ "$schema": "http://json-schema.org/draft-07/schema#",
+ "additionalProperties": {},
+ "properties": {
+ "status": {
+ "const": "stored",
+ "description": "always \"stored\" on success",
+ "type": "string"
+ },
+ "trace_id": {
+ "description": "the stored trace id, 32 hex — pass it to evaluate_output, get_traces or delete_trace",
+ "type": "string"
+ }
+ },
+ "required": [
+ "trace_id",
+ "status"
+ ],
+ "type": "object"
+}verify_citations4 fields changedInput schema / properties / max_citations / descriptionPrevious value: -"Max citations to verify (extras skipped); default 20"New value: +"Max citations to verify (extras skipped, not errored); default 20, at most 50"Input schema / properties / max_cost_usd_total / descriptionPrevious value: -"Cap TOTAL judge cost across all citations in this call; default $1.00"New value: +"Cap TOTAL judge cost across all citations in this call; default 1.00 USD — the pipeline stops when the next call would exceed it"Input schema / properties / model / descriptionPrevious value: -"Judge model for per-citation verification. Supported: anthropic = claude-opus-4-7 | claude-sonnet-4-6 | claude-haiku-4-5-20251001; openai = gpt-4o | gpt-4o-mini | o1-mini."New value: +"Judge model for per-citation verification. Supported: anthropic = claude-opus-4-7 | claude-sonnet-4-6 | claude-haiku-4-5 | claude-haiku-4-5-20251001; openai = gpt-4o | gpt-4o-mini | o1-mini."Output schema / (root)Previous value: -nullNew value: +{
+ "$schema": "http://json-schema.org/draft-07/schema#",
+ "additionalProperties": {},
+ "properties": {
+ "citations": {
+ "description": "per citation: the citation (raw, kind, identifier, offsets), resolve_status ok | skipped | error, resolve_error, source (url, status, content_type, bytes_fetched, truncated), judge (supported, confidence, rationale, cost_usd, latency_ms, tokens)",
+ "items": {
+ "additionalProperties": {},
+ "properties": {
+ "resolve_status": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "resolve_status"
+ ],
+ "type": "object"
+ },
+ "type": "array"
+ },
+ "id": {
+ "description": "the evaluation id; read it back at iris://evaluations/{id}",
+ "type": "string"
+ },
+ "overall_score": {
+ "description": "supported / judged; null when nothing was judged",
+ "type": [
+ "number",
+ "null"
+ ]
+ },
+ "passed": {
+ "description": "true when every judged citation was supported, or nothing was judged and nothing failed",
+ "type": "boolean"
+ },
+ "total_citations_found": {
+ "description": "citations extracted from the output",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "total_cost_usd": {
+ "description": "the spend across every judge call",
+ "type": "number"
+ },
+ "total_judged": {
+ "description": "citations the judge ruled on",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "total_resolved": {
+ "description": "citations whose source was fetched",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "total_supported": {
+ "description": "citations the judge found supported",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "trace_id": {
+ "description": "the linked trace, when one was named",
+ "type": "string"
+ }
+ },
+ "required": [
+ "id",
+ "overall_score",
+ "passed",
+ "total_citations_found",
+ "total_resolved",
+ "total_judged",
+ "total_supported",
+ "total_cost_usd",
+ "citations"
+ ],
+ "type": "object"
+}v0.8.0delete_rule3 fields changedInput schema / additionalPropertiesAdded value: +falseInput schema / properties / enabledAdded value: +{
+ "description": "When present the rule is NOT deleted: false DISABLES it (kept in the store, stops firing immediately, history and provenance preserved); true RE-ENABLES a disabled rule. Omit to delete",
+ "type": "boolean"
+}Input schema / properties / rule_id / descriptionPrevious value: -"Rule id to delete (format: rule-<hex>); obtained from list_rules or deploy_rule response"New value: +"Rule id to delete or toggle (format: rule-<hex>); obtained from list_rules or deploy_rule response"delete_trace1 field changedInput schema / additionalPropertiesAdded value: +falsedeploy_rule20 fields changedInput schema / additionalPropertiesAdded value: +falseInput schema / properties / definition / additionalPropertiesAdded value: +falseInput schema / properties / definition / descriptionPrevious value: -"Check definition (regex, length, keyword, cost, or schema)"New value: +"Check definition (regex, length, keyword, cost, or schema). Accepts exactly type, config, weight and an optional name — an unknown key is rejected"Input schema / properties / definition / properties / config / descriptionAdded value: +"Check configuration; required keys depend on type (regex_match: pattern; min_length: min_length; max_length: max_length; contains_keywords/excludes_keywords: keywords; cost_threshold: max_cost; json_schema: none)"Input schema / properties / definition / properties / name / descriptionAdded value: +"Optional and IGNORED if given — the server overwrites it with the top-level `name` so the rule reports under one name everywhere"Input schema / properties / definition / properties / name / maxLengthAdded value: +80Input schema / properties / definition / properties / name / minLengthAdded value: +1Input schema / properties / definition / properties / type / descriptionAdded value: +"Check type — decides which config keys are required"Input schema / properties / definition / properties / weight / descriptionAdded value: +"Weight in the weighted score (default 1; must be > 0)"Input schema / properties / definition / properties / weight / exclusiveMinimumAdded value: +0Input schema / properties / definition / requiredPrevious value: -[
- "name",
- "type",
- "config"
-]New value: +[
+ "type",
+ "config"
+]Input schema / properties / evalType / descriptionPrevious value: -"Eval category this rule belongs to; determines when it fires"New value: +"camelCase alias of eval_type, accepted for compatibility — prefer eval_type (snake_case is canonical across the tools)"Input schema / properties / eval_typeAdded value: +{
+ "description": "Eval category this rule belongs to; the rule fires on evaluate_output calls whose eval_type equals it (and on eval_type=\"all\"). Canonical snake_case spelling — pass exactly one of eval_type / evalType",
+ "enum": [
+ "completeness",
+ "relevance",
+ "safety",
+ "cost",
+ "custom"
+ ],
+ "type": "string"
+}Input schema / properties / name / descriptionPrevious value: -"Human-readable rule name (used in eval results)"New value: +"Human-readable rule name (1-80 chars; used in eval results). Must be unique among deployed rules unless replace=true"Input schema / properties / name / maxLengthPrevious value: -120New value: +80Input schema / properties / replaceAdded value: +{
+ "default": false,
+ "description": "When a rule with this name is already deployed: false (default) rejects the call; true deletes the existing same-named rule(s) and deploys this one in their place (fresh id; audit rows preserved)",
+ "type": "boolean"
+}Input schema / properties / severity / descriptionPrevious value: -"Severity used for dashboard sort + audit alerts"New value: +"What a FAILURE of this rule means. low/medium: informational — contributes to the weighted score only (plus dashboard sort + audit alerts). high/critical: hard-fail — a failing evaluation of this rule forces the overall passed=false regardless of the weighted score"Input schema / properties / sourceMomentId / descriptionPrevious value: -"Optional Decision Moment ID the rule was derived from (preserves workflow-inversion provenance)"New value: +"camelCase alias of source_moment_id, accepted for compatibility — prefer source_moment_id"Input schema / properties / source_moment_idAdded value: +{
+ "description": "Optional Decision Moment ID the rule was derived from (preserves workflow-inversion provenance). Canonical snake_case — pass exactly one of source_moment_id / sourceMomentId",
+ "type": "string"
+}Input schema / requiredPrevious value: -[
- "name",
- "evalType",
- "definition"
-]New value: +[
+ "name",
+ "definition"
+]evaluate_output19 fields changedInput schema / additionalPropertiesAdded value: +falseInput schema / properties / cost_usd / descriptionPrevious value: -"Cost in USD — only consulted when eval_type=\"cost\" (compared against cost_threshold rules)"New value: +"Cost in USD — consulted by the cost bundle (eval_type=\"cost\" or \"all\") AND by any cost_threshold custom rule regardless of eval_type; omit it and such a rule skips rather than passes (a critical one is listed in critical_skipped)"Input schema / properties / custom_rules / descriptionPrevious value: -"Custom evaluation rules — fires REGARDLESS of eval_type; pass eval_type=\"custom\" if you want ONLY these"New value: +"Custom evaluation rules, max 10 per call (deploy persistent rule sets via deploy_rule instead) — fires REGARDLESS of eval_type; pass eval_type=\"custom\" if you want ONLY these. Each entry accepts exactly name, type, config, weight — an unknown key (e.g. a misspelled weight) is rejected"Input schema / properties / custom_rules / items / additionalPropertiesAdded value: +falseInput schema / properties / custom_rules / items / properties / config / descriptionAdded value: +"Check configuration; keys depend on type (pattern, min_length, keywords, max_cost, …)"Input schema / properties / custom_rules / items / properties / name / descriptionAdded value: +"Rule name as it will appear in rule_results"Input schema / properties / custom_rules / items / properties / name / minLengthAdded value: +1Input schema / properties / custom_rules / items / properties / type / descriptionAdded value: +"Check type — decides which config keys the rule reads"Input schema / properties / custom_rules / items / properties / weight / descriptionAdded value: +"Weight in the weighted score (default 1; must be > 0)"Input schema / properties / custom_rules / items / properties / weight / exclusiveMinimumAdded value: +0Input schema / properties / custom_rules / maxItemsAdded value: +10Input schema / properties / eval_type / defaultRemoved value: -"completeness"Input schema / properties / eval_type / descriptionPrevious value: -"Rule bundle to apply: completeness | relevance | safety | cost | custom — picks which built-in rules fire"New value: +"Rule bundle to apply: completeness | relevance | safety | cost | custom | all — picks which built-in rules fire. \"all\" runs every bundle in one call and adds a per-category breakdown. Defaults to \"all\" when omitted — every bundle runs, safety included, and the response carries a note saying the default ran"Input schema / properties / eval_type / enumPrevious value: -[
- "completeness",
- "relevance",
- "safety",
- "cost",
- "custom"
-]New value: +[
+ "completeness",
+ "relevance",
+ "safety",
+ "cost",
+ "custom",
+ "all"
+]Input schema / properties / expected / descriptionPrevious value: -"Expected output for comparison — REQUIRED when eval_type=\"relevance\" (used as keyword-overlap target)"New value: +"Expected output for comparison — consulted only by the completeness bundle's expected_coverage rule; NOT used by relevance (the relevance rules compare the output against `input`)"Input schema / properties / input / descriptionPrevious value: -"Original input for context — improves relevance scoring (keyword overlap vs input)"New value: +"Original input for context (the ask + any source material the agent was given) — REQUIRED when eval_type=\"relevance\" (keyword_overlap and topic_consistency compare the output against it and skip without it); also grounds the safety bundle's hallucination signals"Input schema / properties / token_usage / descriptionPrevious value: -"Token usage breakdown — only consulted when eval_type=\"cost\" (used for token-budget rules)"New value: +"Token usage breakdown — only consulted by the cost bundle (eval_type=\"cost\" or \"all\"; used for token-budget rules)"Input schema / properties / tool_callsAdded value: +{
+ "description": "What the agent DID — the tool calls it made, in order, each { tool_name, input?, output?, latency_ms?, error? } exactly as log_trace records them. Read by the trajectory rules — the rules that judge what the agent DID rather than what it wrote. Omit it and those rules SKIP rather than pass — an evaluation with no trajectory data reports \"not judged\", never \"clean\". When trace_id names a stored trace and this argument is omitted, the tool_calls stored on that trace are loaded and used, so a caller who already logged them need not resend them",
+ "items": {
+ "additionalProperties": false,
+ "properties": {
+ "error": {
+ "type": "string"
+ },
+ "input": {},
+ "latency_ms": {
+ "type": "number"
+ },
+ "output": {},
+ "tool_name": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "tool_name"
+ ],
+ "type": "object"
+ },
+ "type": "array"
+}Input schema / properties / trace_id / descriptionPrevious value: -"Link evaluation to a trace — surfaces this eval in the dashboard's trace drill-through"New value: +"Link evaluation to a trace — surfaces this eval in the dashboard's trace drill-through. Must be the id of a stored trace (from log_trace / get_traces); an unknown id is rejected before anything is evaluated"evaluate_with_llm_judge2 fields changedInput schema / additionalPropertiesAdded value: +falseInput schema / properties / trace_id / descriptionPrevious value: -"Link this evaluation to a trace"New value: +"Link this evaluation to a stored trace (id from log_trace / get_traces); an unknown id is rejected BEFORE the judge is called"get_traces16 fields changedInput schema / additionalPropertiesAdded value: +falseInput schema / properties / limit / maximumAdded value: +1000Input schema / properties / limit / minimumAdded value: +1Input schema / properties / limit / typePrevious value: -"number"New value: +"integer"Input schema / properties / max_score / descriptionPrevious value: -"Maximum eval score filter (0..1) — applied to LATEST eval per trace"New value: +"Maximum eval score filter (0..1; values outside are rejected) — applied to LATEST eval per trace"Input schema / properties / max_score / maximumAdded value: +1Input schema / properties / max_score / minimumAdded value: +0Input schema / properties / min_score / descriptionPrevious value: -"Minimum eval score filter (0..1) — applied to LATEST eval per trace, not all evals"New value: +"Minimum eval score filter (0..1; values outside are rejected) — applied to LATEST eval per trace, not all evals; must be <= max_score when both are set"Input schema / properties / min_score / maximumAdded value: +1Input schema / properties / min_score / minimumAdded value: +0Input schema / properties / offset / descriptionPrevious value: -"Zero-based pagination offset — skip first N results"New value: +"Zero-based pagination offset — skip first N results (non-negative integer)"Input schema / properties / offset / maximumAdded value: +9007199254740991Input schema / properties / offset / minimumAdded value: +0Input schema / properties / offset / typePrevious value: -"number"New value: +"integer"Input schema / properties / since / descriptionPrevious value: -"ISO timestamp lower bound — return traces with timestamp >= this"New value: +"ISO 8601 timestamp (or date) lower bound — return traces with timestamp >= this; anything that is not an ISO timestamp is rejected, never treated as \"no bound\""Input schema / properties / until / descriptionPrevious value: -"ISO timestamp upper bound — return traces with timestamp < this"New value: +"ISO 8601 timestamp (or date) upper bound — return traces with timestamp <= this; must not be earlier than `since`"list_rules1 field changedInput schema / additionalPropertiesAdded value: +falselog_trace2 fields changedInput schema / additionalPropertiesAdded value: +falseInput schema / properties / tool_calls / items / additionalPropertiesAdded value: +falseverify_citations2 fields changedInput schema / additionalPropertiesAdded value: +falseInput schema / properties / trace_id / descriptionPrevious value: -"Link verification result to a trace"New value: +"Link verification result to a stored trace (id from log_trace / get_traces); an unknown id is rejected before any fetch or judge call"v0.4.6delete_rule1 field changedInput schema / additionalPropertiesRemoved value: -falsedelete_trace1 field changedInput schema / additionalPropertiesRemoved value: -falsedeploy_rule3 fields changedInput schema / additionalPropertiesRemoved value: -falseInput schema / properties / definition / additionalPropertiesRemoved value: -falseInput schema / properties / definition / properties / config / propertyNamesAdded value: +{
+ "type": "string"
+}evaluate_output4 fields changedInput schema / additionalPropertiesRemoved value: -falseInput schema / properties / custom_rules / items / additionalPropertiesRemoved value: -falseInput schema / properties / custom_rules / items / properties / config / propertyNamesAdded value: +{
+ "type": "string"
+}Input schema / properties / token_usage / additionalPropertiesRemoved value: -falseevaluate_with_llm_judge2 fields changedInput schema / additionalPropertiesRemoved value: -falseInput schema / properties / timeout_ms / maximumAdded value: +9007199254740991get_traces1 field changedInput schema / additionalPropertiesRemoved value: -falselist_rules1 field changedInput schema / additionalPropertiesRemoved value: -falselog_trace8 fields changedInput schema / additionalPropertiesRemoved value: -falseInput schema / properties / metadata / propertyNamesAdded value: +{
+ "type": "string"
+}Input schema / properties / spans / items / additionalPropertiesRemoved value: -falseInput schema / properties / spans / items / properties / attributes / propertyNamesAdded value: +{
+ "type": "string"
+}Input schema / properties / spans / items / properties / events / items / additionalPropertiesRemoved value: -falseInput schema / properties / spans / items / properties / events / items / properties / attributes / propertyNamesAdded value: +{
+ "type": "string"
+}Input schema / properties / token_usage / additionalPropertiesRemoved value: -falseInput schema / properties / tool_calls / items / additionalPropertiesRemoved value: -falseverify_citations3 fields changedInput schema / additionalPropertiesRemoved value: -falseInput schema / properties / per_source_max_bytes / maximumAdded value: +9007199254740991Input schema / properties / per_source_timeout_ms / maximumAdded value: +9007199254740991v0.1.10delete_ruledelete_tracedeploy_ruleevaluate_output8 fields changedInput schema / properties / cost_usd / descriptionPrevious value: -"Cost for cost evaluation"New value: +"Cost in USD — only consulted when eval_type=\"cost\" (compared against cost_threshold rules)"Input schema / properties / custom_rules / descriptionPrevious value: -"Custom evaluation rules"New value: +"Custom evaluation rules — fires REGARDLESS of eval_type; pass eval_type=\"custom\" if you want ONLY these"Input schema / properties / eval_type / descriptionPrevious value: -"Type of evaluation"New value: +"Rule bundle to apply: completeness | relevance | safety | cost | custom — picks which built-in rules fire"Input schema / properties / expected / descriptionPrevious value: -"Expected output for comparison"New value: +"Expected output for comparison — REQUIRED when eval_type=\"relevance\" (used as keyword-overlap target)"Input schema / properties / input / descriptionPrevious value: -"Original input for context"New value: +"Original input for context — improves relevance scoring (keyword overlap vs input)"Input schema / properties / output / descriptionPrevious value: -"The output text to evaluate"New value: +"The output text to evaluate (the agent's response that gets scored against rules)"Input schema / properties / token_usage / descriptionPrevious value: -"Token usage for cost evaluation"New value: +"Token usage breakdown — only consulted when eval_type=\"cost\" (used for token-budget rules)"Input schema / properties / trace_id / descriptionPrevious value: -"Link evaluation to a trace"New value: +"Link evaluation to a trace — surfaces this eval in the dashboard's trace drill-through"evaluate_with_llm_judgeget_traces11 fields changedInput schema / properties / agent_name / descriptionPrevious value: -"Filter by agent name"New value: +"Filter by agent name — exact match (no wildcards in v0.4)"Input schema / properties / framework / descriptionPrevious value: -"Filter by framework"New value: +"Filter by agent framework — exact match (e.g., langchain, autogen)"Input schema / properties / include_summary / descriptionPrevious value: -"Include dashboard summary stats"New value: +"Include dashboard summary stats in same response — saves a round-trip when ingesting for dashboards"Input schema / properties / limit / descriptionPrevious value: -"Results per page"New value: +"Results per page (default 50, max 1000 — values >1000 return 400)"Input schema / properties / max_score / descriptionPrevious value: -"Maximum eval score filter"New value: +"Maximum eval score filter (0..1) — applied to LATEST eval per trace"Input schema / properties / min_score / descriptionPrevious value: -"Minimum eval score filter"New value: +"Minimum eval score filter (0..1) — applied to LATEST eval per trace, not all evals"Input schema / properties / offset / descriptionPrevious value: -"Pagination offset"New value: +"Zero-based pagination offset — skip first N results"Input schema / properties / since / descriptionPrevious value: -"ISO timestamp lower bound"New value: +"ISO timestamp lower bound — return traces with timestamp >= this"Input schema / properties / sort_by / descriptionPrevious value: -"Sort field"New value: +"Sort by timestamp | latency_ms | cost_usd (default timestamp)"Input schema / properties / sort_order / descriptionPrevious value: -"Sort order"New value: +"Sort order: asc | desc (default desc — most recent / highest first)"Input schema / properties / until / descriptionPrevious value: -"ISO timestamp upper bound"New value: +"ISO timestamp upper bound — return traces with timestamp < this"list_ruleslog_trace11 fields changedInput schema / properties / agent_name / descriptionPrevious value: -"Name of the agent"New value: +"Agent name — used for filtering in get_traces (e.g., \"customer-support-bot\")"Input schema / properties / cost_usd / descriptionPrevious value: -"Total cost in USD"New value: +"Total cost in USD — overrides per-span aggregation when provided (treated as authoritative)"Input schema / properties / framework / descriptionPrevious value: -"Agent framework name"New value: +"Agent framework identifier (e.g., langchain, autogen, custom)"Input schema / properties / input / descriptionPrevious value: -"Agent input text"New value: +"Agent input text — the user prompt or upstream input that produced this output"Input schema / properties / latency_ms / descriptionPrevious value: -"Total execution time in milliseconds"New value: +"Total execution time in milliseconds (end-to-end agent latency)"Input schema / properties / metadata / descriptionPrevious value: -"Arbitrary metadata"New value: +"Opaque key-value tags (e.g. {requestId, userId, env}) — queryable in dashboard, not via get_traces filters"Input schema / properties / output / descriptionPrevious value: -"Agent output text"New value: +"Agent output text — what the agent produced (pass to evaluate_output for scoring)"Input schema / properties / spans / descriptionPrevious value: -"Detailed execution spans"New value: +"Detailed execution spans (hierarchical span tree with timings, attributes, events)"Input schema / properties / timestamp / descriptionPrevious value: -"Trace timestamp (ISO 8601)"New value: +"Trace timestamp (ISO 8601); defaults to now() when omitted"Input schema / properties / token_usage / descriptionPrevious value: -"Token usage breakdown"New value: +"Token usage breakdown (prompt/completion/total — used for cost analysis)"Input schema / properties / tool_calls / descriptionPrevious value: -"Tool calls made during execution"New value: +"Tool calls made during execution (per-call latency, errors, input/output)"verify_citationsv0.1.8evaluate_outputget_traceslog_tracev0.1.7evaluate_outputget_traceslog_traceevaluate_outputget_traceslog_traceEach tool targets a distinct resource and action: trace creation/query/deletion, deterministic evaluation, LLM judging, citation verification, and rule lifecycle management are all clearly separated. The three evaluation-related tools are well-differentiated by deterministic vs. LLM vs. citation-specific behavior.
All tool names follow a consistent snake_case verb_noun pattern such as log_trace, get_traces, deploy_rule, delete_rule, and verify_citations. Even evaluate_with_llm_judge is a readable verb-object form with a modifier, and the naming style is uniform across the set.
Nine tools is well-scoped for the domain of agent tracing, evaluation, and custom rule management. Each tool earns its place, and there is no redundancy or bloat.
The trace lifecycle is complete (log/get/delete), evaluation has deterministic, LLM, and citation-verification paths, and rules support deploy/list/delete including replacement and toggling. The main gap is the absence of a dedicated tool to retrieve or list stored evaluations; results are returned at creation time and only the latest evaluation score is exposed through get_traces.
MCP observability. Query live traffic, errors, duration, and alerts from your AI agent.
MCP server for building and testing AI agents with multi-model experimentation and insights.
Hosted MCP server for LLM cost estimation, model comparison, and budget-aware routing.
AgentGuard — 20-tool AI safety MCP: policy preflight, risk scoring, audit logging, rate limits.
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/iris-eval/mcp-server'
If you have feedback or need assistance with the MCP directory API, please join our Discord server
