Iris is an MCP server for evaluating AI agent performance in production, providing observability, quality scoring, and cost monitoring — with no SDK or code changes required.
Log Agent Traces: Capture full execution details including hierarchical spans, tool calls, token usage, latency, cost in USD, metadata, and framework identification, stored in SQLite.
Evaluate Output Quality: Score outputs against 12 built-in rules across completeness, relevance, safety (PII detection, prompt injection, hallucination markers), and cost categories. Add custom rules using regex, keyword checks, length constraints, JSON schema validation, or cost thresholds. Link evaluations to a specific trace via trace_id.
Monitor Costs: Track aggregate costs across agents over any time window, set budget thresholds, and get flagged when agents overspend.
Query & Filter Traces: Retrieve stored traces filtered by agent name, framework, time range, and eval score range, with sorting and pagination.
Web Dashboard: Access a real-time dark-mode UI for trace visualization, evaluation results, and cost breakdowns.
Security & Transport: Supports stdio (default) or HTTP transport, with API key authentication, CORS restrictions, rate limiting, and input validation in HTTP mode.
Framework Integration: Works automatically with any MCP-compatible agent (Claude Desktop, Cursor, Windsurf).
Iris scores every agent run for quality, safety, and cost — on your machine, with no SDK and no account. Most agent projects check quality by running a few remembered prompts and eyeballing the output. Iris replaces that with numbers you can audit: your agent's runs land in a SQLite database on your disk, 15 built-in rules score them deterministically — PII, prompt injection, hallucination markers, cost thresholds, and the agent's own tool calls — free, with no LLM calls, and an optional LLM judge with a hard per-eval cost cap handles the semantic questions. Every rule is inspectable and editable, because a judge you can't audit is just vibes with a number on it. MIT licensed, no telemetry; your traces never leave your machine.
Requires Node.js 20 or later. Check with node --version.
No agent wiring, no config — one command:
npx @iris-eval/mcp-server --demoThis seeds a demo database — a handful of small agents with a week of runs — and serves the dashboard against it at http://localhost:6920 (your browser opens automatically on first run). The dashboard lands on Failures: what failed, worst and newest first. Worth clicking into — a PII leak caught by the safety rules, a flagged prompt-injection attempt, and a failed LLM-judge score with its rationale.
Demo data lives in its own database (demo.db in your Iris home directory — ~/.iris on macOS/Linux, %USERPROFILE%\.iris on Windows) and never mixes with your real traces. Remove all of it with one command:
npx @iris-eval/mcp-server --demo-clearRelated MCP server: runmeter
Add Iris to your MCP config. Works with Claude Desktop, Claude Code, Cursor, Windsurf, Continue, VS Code, Cline, Zed, Codex CLI, Gemini CLI — and any other MCP-compatible agent. One block, dashboard included:
{
"mcpServers": {
"iris-eval": {
"command": "npx",
"args": ["@iris-eval/mcp-server", "--dashboard"]
}
}
}Your agent discovers Iris's nine tools on connect, and the dashboard serves at http://localhost:6920. Now paste this to your agent:
Log that last task to Iris and evaluate the output.
The trace lands on the dashboard with its scores. Prefer the MCP server headless? Drop --dashboard from the args — you can open the same dashboard any time with npx @iris-eval/mcp-server --dashboard.
One thing worth knowing up front: MCP tools are called when the model decides to call them. Iris doesn't intercept your agent, so traces are logged when your agent asks it to log them — either because you told it to, or because your code calls the tools directly. Ask your agent to "log this to Iris and evaluate it" and it will. If you want capture that doesn't depend on the model choosing, POST /api/v1/traces does exactly that — your code sends the trace over plain HTTP, no model in the loop (see docs/http-ingest.md). The CLI and SDKs on the roadmap will be thin clients over the same endpoint.
The ingest endpoint lives on the dashboard port — 6920 by default, not the MCP transport port — and it exists only while the dashboard is running. Pass --dashboard (or set IRIS_DASHBOARD=true); --transport http on its own does not start it, and a request to the transport port returns 404. With the dashboard up, anything that can send an HTTP request can log a trace — and optionally run the deterministic evals in the same request. GET /api/v1/capabilities on the same port says what this server can judge, what each rule needs, the judge state with the steps that enable it, and the limits — the same object the MCP resource iris://capabilities serves — so an HTTP caller has the frame an MCP client gets at initialize:
curl -s -X POST "http://127.0.0.1:6920/api/v1/traces" \
-H "Content-Type: application/json" \
-d '{
"agent_name": "support-bot",
"input": "What is the refund policy?",
"output": "Refunds are available within 30 days of purchase.",
"evaluate": true,
"eval_type": "safety"
}'Returns 201 with the stored trace_id and the evaluation result (in --demo mode the endpoint refuses writes with 403, so demo data never mixes with yours). The endpoint accepts the same body as the log_trace tool and sits behind the same middleware stack as the rest of the dashboard: loopback bind and the DNS-rebinding guard by default, plus Bearer auth when you set one. Two plain facts about it: it accepts unauthenticated writes unless Iris was started with --api-key (or IRIS_API_KEY) — the loopback bind is what keeps it to your machine by default, so set a key before binding beyond loopback; and what it stores is verbatim — input and output land in iris.db exactly as sent, including any text no_pii goes on to flag. Full contract, field reference, and error semantics: docs/http-ingest.md.
npx @iris-eval/mcp-server --self-test # offline diagnostic; exit 0 = healthy, 1 = a check failed
npx @iris-eval/mcp-server --version # prints the bare version, e.g. 0.5.1--self-test first creates your Iris home if it is missing and checks that it is writable (exit 1, naming the path, if it is not), then runs its checks — storage round-trip, a planted SSN and a planted injection caught by the safety rules, dashboard boot, the DNS-rebinding guard — inside an isolated temp home, so your real database is never opened. Everything Iris writes lives under one directory, your Iris home: ~/.iris by default (%USERPROFILE%\.iris on Windows), or wherever IRIS_HOME points. That is where iris.db, config.json, custom-rules.json, audit.log, preferences.json and the demo files live; point IRIS_HOME at a scratch directory to try Iris without touching your real data.
Edit your MCP config file:
macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json
Add the JSON config above, then restart Claude Desktop.
claude mcp add --transport stdio iris-eval -- npx @iris-eval/mcp-serverThen restart the session (/clear or relaunch) for tools to load.
Windows note: Do not use
cmd /cwrapper — it causes path parsing issues. Thenpxcommand works directly.
Add to your workspace .cursor/mcp.json or global MCP settings using the JSON config above.
Add to .vscode/mcp.json in your workspace (note: VS Code uses servers, not mcpServers):
{
"servers": {
"iris-eval": {
"command": "npx",
"args": ["@iris-eval/mcp-server"]
}
}
}Open Cline's MCP Servers panel → Configure MCP Servers, and add the mcpServers JSON config above to cline_mcp_settings.json.
Add to Zed settings.json:
{
"context_servers": {
"iris-eval": {
"command": {
"path": "npx",
"args": ["@iris-eval/mcp-server"]
}
}
}
}Add to ~/.codex/config.toml:
[mcp_servers.iris-eval]
command = "npx"
args = ["@iris-eval/mcp-server"]Add the mcpServers JSON config above to ~/.gemini/settings.json.
Iris is a standard stdio MCP server — one npx @iris-eval/mcp-server command, no SDK, no code changes. If your client supports MCP, it supports Iris. Client config formats change; when in doubt, check your client's MCP docs and point it at that command.
# Global install (recommended for persistent data and faster startup)
npm install -g @iris-eval/mcp-server
iris-mcp --dashboard
# Docker — two servers, two ports: 3000 = MCP HTTP transport,
# 6920 = dashboard (which also serves the POST /api/v1/traces ingest endpoint)
docker run -p 3000:3000 -p 6920:6920 -v iris-data:/data ghcr.io/iris-eval/mcp-serverTip: Global install (
npm install -g) stores traces persistently at~/.iris/iris.db. Withnpx, traces persist in the same location, but startup is slower due to package resolution.
Trace Logging | Hierarchical span trees with per-tool-call latency, token usage, and cost in USD. Stored in SQLite, queryable instantly. |
Output Evaluation | 15 built-in rules across 4 categories: completeness, relevance, safety, cost. PII detection (19 patterns: SSN, credit card, phone, email, IBAN, DOB, MRN, IP, API key, passport, plus AWS/Slack/SendGrid/GitHub/Google/npm/DigitalOcean tokens, PEM private-key blocks and seed phrases), prompt injection (37 patterns, phrase + structural), stub-output detection, hallucination detection (25 context-grounded fabrication/contradiction signals — pass |
LLM-as-Judge | Optional semantic scoring via Anthropic or OpenAI — bring your own API key. Five templates. Hard per-eval cost cap ( |
Cost Visibility | Aggregate cost across all agents over any time window. Set budget thresholds. Get flagged when agents overspend. |
Web Dashboard | Real-time dark-mode UI that lands on the failures, worst and newest first — trace visualization, eval results, cost breakdowns, and a command palette (⌘K) that searches your own rules, traces, and evals. |
Local-first | Everything lives in SQLite on your disk. No account, no sign-up, no telemetry. Outbound HTTP happens only where you opt in: your own LLM-judge key, citation fetching, or an OTel exporter you configure. |
Where this is going next: the roadmap.
Every built-in rule has a published precision, recall and F1 with 95% confidence intervals, measured on a labelled corpus that lives in this repository (proof/corpus/) and regenerates with one command — npm run proof — offline, with no key and no model in the loop. CI re-runs the measurement on every pull request and fails if the committed numbers differ from what the code produces, so a rule cannot change without its numbers changing with it. The numbers are on iris-eval.com/proof and in proof/RESULTS.md; how the corpus was made, what it is not, and how to read an interval are in docs/proof.md. The corpus is synthetic and model-labelled — a human blind label is pending, and the page says so; node proof/blind-sample.mjs draws the reproducible sample that will settle it.
Iris registers nine tools that any MCP-compatible agent can invoke — full rule + trace lifecycle + LLM-as-judge + semantic citation verification:
log_trace — Log an agent execution with spans, tool calls, token usage, and cost
evaluate_output — Score output quality against completeness, relevance, safety, and cost rules (heuristic, deterministic, free)
get_traces — Query stored traces with filtering, pagination, and time-range support
list_rules — Enumerate deployed custom eval rules (read-only)
deploy_rule — Register a new custom eval rule so it fires on every evaluate_output of that category
delete_rule — Remove a deployed custom rule (destructive, idempotent)
delete_trace — Remove a single stored trace by ID (destructive, tenant-scoped)
evaluate_with_llm_judge — Semantic eval via LLM (Anthropic or OpenAI). Five templates: accuracy, helpfulness, safety, correctness, faithfulness. Cost-capped, per-eval pricing disclosed. Bring your own API key (IRIS_ANTHROPIC_API_KEY or IRIS_OPENAI_API_KEY) — Iris doesn't proxy or relay LLM calls.
verify_citations — Extract citations from output (numbered, author-year, URLs, DOIs), fetch sources behind an SSRF-guarded + domain-allowlisted resolver, and use an LLM judge to check whether each source actually supports the cited claim. Opt-in outbound HTTP. Same BYOK requirement as evaluate_with_llm_judge.
Enable the LLM judge (optional; the deterministic rules never need it)
Get an API key from Anthropic or OpenAI.
Put it in the environment of the process that runs Iris, not only your shell. Claude Code, Claude Desktop, Cursor and most MCP clients: the "env" block of the iris-eval entry in your MCP config — "iris-eval": { "command": "npx", "args": ["-y", "@iris-eval/mcp-server"], "env": { "IRIS_ANTHROPIC_API_KEY": "sk-ant-..." } } (IRIS_OPENAI_API_KEY for an OpenAI key). Docker: -e IRIS_ANTHROPIC_API_KEY=... on the run command. HTTP or CI: export it before starting iris-mcp.
Restart the MCP session. A running process never sees a variable set after it started.
Confirm from inside your client: read iris://capabilities — judge.enabled must be true there. A key exported in your shell is not passed to the process your client spawns unless its config lists it. On a machine, npx @iris-eval/mcp-server --self-test prints the judge line for that shell, and GET /api/v1/health reports judge.enabled on a running dashboard.
Spend guard: each call is capped by IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL (default 0.25 USD) and refused before any spend if the worst case would exceed it. Iris calls the provider directly with your key and never proxies it.
When IRIS_OTEL_ENDPOINT is configured, log_trace calls also emit a best-effort OTLP/HTTP JSON export to any OpenTelemetry collector (Jaeger, Grafana Tempo, Datadog OTLP, Honeycomb, etc). See docs/otel-integration.md.
passed is decidedevaluate_output returns both a score and a passed flag — they answer different questions:
score (0..1) is the weighted average across the rules that ran — a quality gradient.
passed is the ship/no-ship verdict: true only when the score clears the pass threshold (default 0.7) and no critical rule failed.
Genuine safety violations hard-fail. By default no_pii, no_injection_patterns, and no_blocklist_words are critical rules: if one fails, the eval reports passed: false no matter how well the other rules scored, and the response names the culprits in critical_failures. A leaked SSN can't be averaged away. Which built-in rules are critical is a deployment setting (eval.criticalRules / eval.nonCriticalRules); every rule result carries the effective critical flag and criticalSource, and list_rules reports the roster this server applies. Custom rules deployed with severity: "high" or "critical" hard-fail the same way; low/medium severities only affect the score. One boundary to know, stated the same way on every surface: a critical rule that skipped (missing context, a broken definition, or a regex killed at the sandbox budget) has not judged the output and does not veto — every such rule is named in critical_skipped. A gate that must fail closed treats a non-empty critical_skipped as unknown, not clean, and may treat any budgetExceeded skip in rule_results the same way.
For CI gates: if you omit eval_type, every bundle runs — completeness, relevance, safety, cost and any custom rules — and the response says eval_type: "all" with a note that the default ran, plus a per-bundle categories map. A bundle with nothing to judge (cost without cost_usd, relevance without input) reports passed: null there — not evaluated, not failing — and never counts toward the verdict. The response always echoes the eval_type that ran, so your gate can verify coverage; key on passed for the verdict and name a bundle only when you want a narrower run.
Two ways to add a rule. Inline rules ride along on one evaluate_output call (custom_rules, up to 10 per call); they fire alongside whatever eval_type bundle you chose, or alone with eval_type: "custom". Deployed rules are registered once with deploy_rule, persist in custom-rules.json under your Iris home, and fire on every future evaluate_output of their evalType. The definition is the same shape either way:
Field | Required | What it is |
| yes | 1–80 characters; appears as |
| yes | one of |
| yes | the keys for that type: |
| no | weight in the score; default |
deploy_rule wraps the definition with name, an optional description, evalType (completeness · relevance · safety · cost · custom) and severity. Severity says what a failure means: low/medium only lower the score; high/critical hard-fail the evaluation — passed: false, the rule named in critical_failures — whatever the weighted score says. A rule that skips (a cost_threshold rule with no cost_usd, or a regex killed at the 100 ms sandbox budget) has not judged the output and is listed in critical_skipped instead. Deploy a critical rule that forbids internal hostnames in anything the agent says:
{
"name": "no_internal_hostnames",
"description": "Output must not mention internal hostnames.",
"evalType": "safety",
"severity": "critical",
"definition": {
"name": "no_internal_hostnames",
"type": "regex_no_match",
"config": { "pattern": "\\b[a-z0-9-]+\\.internal\\.example\\b", "flags": "i" }
}
}The response is the persisted rule — keep the id for delete_rule:
{ "rule": { "id": "rule-588823d0", "name": "no_internal_hostnames", "evalType": "safety", "severity": "critical", "enabled": true, "version": 1, "definition": { "…": "…" } } }From the very next evaluate_output with eval_type: "safety", an output that mentions db-primary.internal.example comes back passed: false with critical_failures: ["no_internal_hostnames"] — even though all five built-in safety rules passed and the weighted score is 0.895. Regex patterns must pass a ReDoS check at deploy time and always run in a sandbox worker under a hard 100 ms deadline. list_rules shows what is deployed; the dashboard's rule composer builds the same shape from a failure you clicked on. Full reference, scoring per type, and worked examples: docs/custom-rules.md.
Full tool schemas and configuration: iris-eval.com
Iris runs entirely on your machine today, and everything it does is free and MIT licensed with no limits and no account.
Hosted storage, shared team history and alerting are under consideration, not under construction. There is no pricing, and nothing to buy. If shared history would be useful to you, the waitlist is how we find out whether it's worth building — it commits you to nothing.
Two commitments hold regardless: nothing that is free today will move behind a paywall, and no compliance certification will be claimed before it is held.
Claude Desktop setup — MCP config for stdio and HTTP modes
TypeScript — MCP SDK client — connect and invoke tools
HTTP transport (TS + Python) — full client code for REST-style integration
LangChain instrumentation (Python, conceptual) — scaffold showing the shape; needs your agent code to be runnable
CrewAI instrumentation (Python, conceptual) — scaffold; same caveat
GitHub Issues — Bug reports and feature requests
GitHub Discussions — Questions and ideas
Contributing Guide — How to contribute
HTTP Ingest — Deterministic trace capture via POST /api/v1/traces
Roadmap — What's coming next
Versioning policy — What each version number promises, and what has to be true before 1.0
Flag | Default | Description |
|
| Transport type: |
|
| HTTP transport port |
|
| SQLite database path |
|
| Config file path |
| — | API key for HTTP authentication (transport and dashboard, including |
|
| Enable web dashboard. Also the only way the |
|
| Dashboard port |
|
| Dashboard bind address. Loopback by default — the dashboard is unauthenticated unless |
|
| Seed a demo database (separate from your real traces) and serve the dashboard against it |
|
| Delete the demo database and exit |
|
| Run the offline install diagnostic in an isolated temp home, then exit (0 = healthy, 1 = a check failed) |
|
| Delete every stored trace, span and evaluation from the configured database, compact the file and truncate the write-ahead log so the deleted text does not linger on disk, then exit. Deployed rules, the audit log and preferences are kept. Not reversible. Stop any running Iris server first — the file is compacted in place. Refuses to combine with |
| — | Print the bare version (e.g. |
Every variable --help documents. CLI flags take precedence over environment variables when both are set.
Variable | Description |
| Transport type ( |
| HTTP transport bind address (default |
| HTTP transport port (1-65535, default |
| Directory for all per-user files: |
| SQLite database path (overrides |
| Log level: |
|
|
| Dashboard port (1-65535, default |
| Dashboard bind address (default |
| API key for HTTP authentication |
| Comma-separated origin allowlist. Dashboard: CORS headers (supports globs, e.g. |
| Set to |
| Required by |
| Required by |
| Hard cost cap per LLM judge call (default |
| Set to |
| Comma-separated hostname allowlist for |
| Enable best-effort OTLP/HTTP JSON trace export to this collector URL |
|
|
| Comma-separated |
| Per-export timeout (default |
| Website waitlist API only — required when the iris-eval.com site is deployed; the server never reads it |
When using HTTP transport, Iris includes:
API key authentication with timing-safe comparison (Bearer for API clients; browser sign-in to the dashboard via ?key=)
CORS restricted to localhost by default
Rate limiting (600 req/min dashboard API, 20 req/min MCP)
Helmet security headers
Zod input validation on all routes
ReDoS-safe regex for custom eval rules
1MB request body limits
# Production deployment
iris-mcp --transport http --port 3000 --api-key "$(openssl rand -hex 32)" --dashboardWith a key set, API clients — MCP clients, capture SDKs, POST /api/v1/traces — send Authorization: Bearer <key>. To open the dashboard in a browser, append the key once to any dashboard URL, http://localhost:6920/?key=<api key>: Iris exchanges it for an HttpOnly, SameSite=Lax session cookie and redirects to the same page with the key removed from the address bar. A page opened without a session shows a sign-in form that does the same exchange. The key is never stored in the browser, and sessions live only in the server process.
Everything Iris stores lives under your Iris home (~/.iris, or IRIS_HOME). iris.db keeps every trace's input and output verbatim — including any text no_pii goes on to flag; detection does not redact unless you ask it to: storage.redact: "critical_spans" in config.json stores each evaluation's output with the spans a critical detector flagged replaced by [REDACTED:<pattern>] (off by default; the evidence offsets still index the text the caller saw). At startup, and every retention.sweepIntervalHours (default 24, 0 disables the timer) after that, traces and evaluations older than retention.days (default 30, 0 disables, set in config.json) are deleted and the write-ahead log is checkpointed. Deleting a trace — by delete_trace or by the sweep — erases the text of every evaluation linked to it (the output, the expected text, the suggestions and the rule messages) and stamps erased_at; the verdict, the scores and the evidence offsets stay. To remove everything now, stop the server and run --purge: it deletes every stored trace, span and evaluation, compacts the database and truncates the write-ahead log so the text is gone from disk, and keeps your deployed rules, audit log and preferences.
npx @iris-eval/mcp-server --self-testIt checks storage, the deterministic evals, and the dashboard in an isolated temp home and prints a per-step verdict — the failure output names the broken step. Exit code 0 means the install is healthy.
ERR_MODULE_NOT_FOUNDYou may have a cached older version. Clear the npx cache and retry:
npx --yes @iris-eval/mcp-server@latestOr install globally to avoid cache issues entirely:
npm install -g @iris-eval/mcp-server@latestnpm install --ignore-scripts broke the SQLite bindingIris stores traces with better-sqlite3, a native module that fetches or compiles its binding in an install script. If that script was skipped — --ignore-scripts on the command line, ignore-scripts=true in an .npmrc (common on corporate machines), or a registry mirror that strips postinstall — startup fails with a long "Could not locate the bindings file" dump listing a dozen paths it tried. Rebuild that one module:
npm rebuild better-sqlite3
# for a global install:
npm rebuild -g better-sqlite3MCP tools only load at session start. After adding iris-eval, restart the session with /clear or relaunch the terminal.
npx @iris-eval/mcp-server --versionThe first startup log line also carries it (Starting Iris MCP server vX.Y.Z), and --self-test prints it in its summary. For a global install, npm ls -g @iris-eval/mcp-server shows the installed version.
# If using npx (clears cache and fetches latest)
npx --yes @iris-eval/mcp-server@latest
# If installed globally
npm update -g @iris-eval/mcp-serverIris requires Node.js 20 or later. Node 18 reached EOL in April 2025 and is not supported.
node --version # Must be v20.x or v22.x+cmd /c not neededClaude Code's /doctor may suggest wrapping npx with cmd /c. This is not needed and causes path parsing issues. Use npx directly:
# Correct
claude mcp add --transport stdio iris-eval -- npx @iris-eval/mcp-server
# Wrong (causes /c to be parsed as a path)
claude mcp add --transport stdio iris-eval -- cmd /c "npx @iris-eval/mcp-server"If Iris is useful to you, consider starring the repo — it helps others find it.
MIT Licensed.
Remove a deployed custom rule — or, with enabled, disable or re-enable it without removing it — effective on the next evaluate_output call.
What it does. Without enabled: deletes the rule from ~/.iris/custom-rules.json, appends a rule.delete audit entry and unregisters it from the running engine; deleted is false when no rule has that id (already gone, or not this tenant's), and no audit row is written twice. With enabled: the rule stays with its history and provenance; false stops it firing at once and keeps it off across restarts, true brings it back under the same id; a rule.toggle audit entry is written unless the flag was already in that state. Past evaluations that referenced the rule are untouched either way.
When not to use it. On built-in rules: they are not in the store and cannot be deleted or disabled. To delete a trace (delete_trace). To replace a rule: deploy_rule with the same name and replace: true.
Returns. JSON with deleted (true when a rule was removed; always false on a toggle); rule_id (the id that was asked for); toggled (toggle only: true when the rule exists (also when it was already in the requested state)); enabled (toggle only: the rule's state after the call); rule (toggle only: the rule as stored).
Errors. IRIS_STORAGE_ERROR when the store cannot be written. A malformed rule_id (not rule-) or an unknown argument is refused before the handler runs. Every failure returns {"error":{"code","message","recovery":[]}} with isError true; follow recovery before retrying.
Siblings. deploy_rule — add or replace a rule; list_rules — find the id; evaluate_output — where the rule fires.
| Name | Required | Description | Default |
|---|---|---|---|
| enabled | No | When present the rule is NOT deleted: false DISABLES it (kept in the store, stops firing immediately, history and provenance preserved); true RE-ENABLES a disabled rule. Omit to delete | |
| rule_id | Yes | Rule id to delete or toggle (format: rule-<hex>); obtained from list_rules or deploy_rule response |
| Name | Required | Description |
|---|---|---|
| rule | No | toggle only: the rule as stored |
| deleted | Yes | true when a rule was removed; always false on a toggle |
| enabled | No | toggle only: the rule's state after the call |
| rule_id | Yes | the id that was asked for |
| toggled | No | toggle only: true when the rule exists (also when it was already in the requested state) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Goes far beyond the destructiveHint annotation by disclosing the exact file modified, audit entries written, engine unregistration, persistence across restarts, no impact on past evaluations, error shape, and malformed-rule_id rejection. This gives the agent a rich behavioral model.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but tightly organized under clear labeled sections ('What it does', 'When not to use it', 'Returns', 'Errors', 'Siblings'), and the first sentence front-loads the core behavior. Every sentence contributes necessary information for a complex destructive/toggle tool.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Covers side effects, timing, return values, error recovery, exclusions, and sibling routing. Even with an output schema noted, the operational nuances described here are essential and nothing critical is missing for an agent to call this tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3, but the description adds critical operational meaning: omitting enabled means delete, enabled: false stops firing immediately while preserving history, enabled: true restores the same id, and return fields differ between delete and toggle modes. This materially improves correct invocation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: 'Remove a deployed custom rule — or, with enabled, disable or re-enable it'. It clearly distinguishes this tool from siblings by naming delete_trace, deploy_rule, and the built-in rule exclusion, so an agent can select it correctly without ambiguity.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Contains an explicit 'When not to use it' section naming alternatives and conditions: built-in rules cannot be modified, trace deletion belongs to delete_trace, and rule replacement belongs to deploy_rule with replace: true. This is ideal routing guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Remove one stored trace by id; its spans go with it, and every evaluation linked to it keeps its verdict and loses its text.
What it does. Deletes the trace row for the caller's tenant. Spans cascade. Evaluations linked to it keep their verdict, scores, criticality and evidence offsets; their output text, expected text, suggestions and rule messages are erased in the same transaction and erased_at is stamped, so no text from the trace survives in any evaluation. deleted is false when no trace has that id — already removed, or not this tenant's — and that is not an error. No audit entry is written: traces are user data, not policy.
When not to use it. To expire old data in bulk (retention.days; the sweep runs at boot and every retention.sweepIntervalHours). To delete evaluations: they are not deleted per row; retention and --purge cover them. To pause anything: traces are immutable, there is nothing to pause.
Returns. JSON with deleted (true when a trace row was removed; false when no trace with that id existed for this tenant); trace_id (the id that was asked for).
Errors. IRIS_STORAGE_ERROR when the delete cannot run. A malformed trace_id (not 32 lowercase hex) is refused before the handler runs. Every failure returns {"error":{"code","message","recovery":[]}} with isError true; follow recovery before retrying.
Siblings. log_trace — store a trace; get_traces — find the trace to delete; delete_rule — the equivalent for custom rules.
| Name | Required | Description | Default |
|---|---|---|---|
| trace_id | Yes | Trace id to delete (32-hex lowercase; obtained from log_trace response or get_traces) |
| Name | Required | Description |
|---|---|---|
| deleted | Yes | true when a trace row was removed; false when no trace with that id existed for this tenant |
| trace_id | Yes | the id that was asked for |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already signal destructiveness, but the description goes far beyond that: spans cascade, evaluations keep verdicts but lose text, erased_at is stamped, deleted=false for absent/foreign traces is not an error, and no audit entry is written. It also discloses error behavior and malformed-id refusal, which is valuable execution context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but well-structured with clear sections, and the core action is front-loaded. There is some redundancy between the opening sentence and the 'What it does' section, but the richness of behavioral detail justifies most of the length.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a destructive delete tool with an output schema, the description covers inputs, side effects, return semantics, error format, and exclusions. Nothing an agent needs to decide whether to call it and interpret the result is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and the single parameter trace_id is already fully documented with format and provenance. The description adds no new semantic details about the parameter itself, so baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a precise verb-resource statement: 'Remove one stored trace by id', and immediately distinguishes the tool from siblings like delete_rule. It names log_trace and get_traces as the sourcing/finding counterparts, so an agent can orient without opening schemas.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly covers when to use the tool and, more importantly, when not to use it: bulk retention expiry, deleting evaluations, or pausing traces. It also names the sibling delete_rule as the equivalent for custom rules, giving the agent a clear decision boundary.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Deploy a custom rule that fires on every future evaluate_output call of its bundle — persisted, active immediately, audited.
What it does. Writes the rule to ~/.iris/custom-rules.json, appends a rule.deploy audit entry and registers it with the running engine, so it fires on the very next call and survives restarts. eval_type says WHEN it fires (that bundle, and eval_type="all"); severity says what a failure DOES: low and medium only lower the weighted score, high and critical force passed to false and list the rule in critical_failures. definition.type picks the check (regex_match, regex_no_match, min_length, max_length, contains_keywords, excludes_keywords, json_schema, cost_threshold) and definition.config carries its keys (pattern; min_length; max_length; keywords; max_cost). Any bundle and type combine. Names are unique: a taken name is refused unless replace is true, which retires the earlier rule(s) first and reports them. Argument names are snake_case; the camelCase aliases evalType and sourceMomentId are accepted — pass one spelling of each.
When not to use it. To try a rule first: POST /api/v1/rules/custom/preview on the dashboard replays a definition against stored traces without deploying. For a one-off check on one call: the custom_rules argument of evaluate_output. To pause a rule: delete_rule with enabled: false.
Returns. JSON with rule (the rule as persisted: id (rule-, keep it for delete_rule), name, description, evalType, severity, definition, enabled, createdAt, updatedAt, version, sourceMomentId); replaced (with replace: true, the earlier rule(s) of the same name that were retired); warning (with replace: true, one sentence naming what was retired).
Errors. IRIS_DUPLICATE_RULE when the name is deployed and replace is false (the message names the existing id). IRIS_INVALID_RULE_CONFIG when the definition is rejected — a regex that fails the ReDoS check or exceeds 1000 characters, a missing config key — naming the field; nothing is deployed. IRIS_STORAGE_ERROR when the store cannot be written. An unknown key in definition, a name over 80 characters, a non-positive weight or both spellings of an alias are refused before the handler runs. Every failure returns {"error":{"code","message","recovery":[]}} with isError true; follow recovery before retrying.
Siblings. list_rules — see what is deployed and the built-in roster; delete_rule — remove, disable or re-enable; evaluate_output — where the rule fires.
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes | Human-readable rule name (1-80 chars; used in eval results). Must be unique among deployed rules unless replace=true | |
| replace | No | When a rule with this name is already deployed: false (default) rejects the call; true deletes the existing same-named rule(s) and deploys this one in their place (fresh id; audit rows preserved) | |
| evalType | No | camelCase alias of eval_type, accepted for compatibility — prefer eval_type (snake_case is canonical across the tools) | |
| severity | No | What a FAILURE of this rule means. low/medium: informational — contributes to the weighted score only (plus dashboard sort + audit alerts). high/critical: hard-fail — a failing evaluation of this rule forces the overall passed=false regardless of the weighted score | medium |
| eval_type | No | Eval category this rule belongs to; the rule fires on evaluate_output calls whose eval_type equals it (and on eval_type="all"). Canonical snake_case spelling — pass exactly one of eval_type / evalType | |
| definition | Yes | Check definition (regex, length, keyword, cost, or schema). Accepts exactly type, config, weight and an optional name — an unknown key is rejected | |
| description | No | What this rule checks for and why it matters | |
| sourceMomentId | No | camelCase alias of source_moment_id, accepted for compatibility — prefer source_moment_id | |
| source_moment_id | No | Optional Decision Moment ID the rule was derived from (preserves workflow-inversion provenance). Canonical snake_case — pass exactly one of source_moment_id / sourceMomentId |
| Name | Required | Description |
|---|---|---|
| rule | Yes | the rule as persisted: id (rule-<hex>, keep it for delete_rule), name, description, evalType, severity, definition, enabled, createdAt, updatedAt, version, sourceMomentId |
| warning | No | with replace: true, one sentence naming what was retired |
| replaced | No | with replace: true, the earlier rule(s) of the same name that were retired |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only indicate it is not read-only, not idempotent, and not destructive. The description substantially exceeds them by disclosing file writes, audit entries, engine registration, restart persistence, unique-name refusal, replacement semantics, and detailed error behavior. No contradictions with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but deliberately structured with clear sections: what it does, when not to use it, returns, errors, and siblings. It front-loads the core behavior in the first sentence and every paragraph covers a distinct needed concern.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the complex nested schema and 9 parameters, the description covers side effects, return shape, error codes, recovery behavior, and sibling relationships. An agent has enough information to decide when to call it, construct a valid definition, and interpret outcomes.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, but the description adds meaning beyond the schema: alias handling, the 'all' eval_type behavior, severity's hard-fail impact, definition.config key mapping, and rejection of unknown keys. It compensates richly for any ambiguity left by the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The opening sentence states a specific action and resource: deploying a custom rule that fires on future evaluate_output calls, with persistence, immediate activation, and auditing. It clearly distinguishes this from siblings like list_rules and delete_rule.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Includes an explicit 'When not to use it' section naming preview, evaluate_output's custom_rules argument, and delete_rule for pausing. This gives an agent concrete routing guidance for alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Score an agent output against the deterministic rule bundles: the ship verdict with its basis, every rule result with evidence and uncertainty, and what was not judged.
What it does. In-process, no network, no key. eval_type picks one bundle (completeness, relevance, safety, cost, custom) or all (the default): every bundle plus deployed and inline custom rules, with a per-bundle breakdown in categories. Inputs decide what can be judged: input is REQUIRED when eval_type="relevance" (keyword_overlap and topic_consistency compare the output against it and skip without it) and grounds the hallucination signals; tool_calls, or a trace_id whose stored tool_calls are reused, feed the trajectory rules; cost_usd and token_usage feed the cost rules; expected feeds only expected_coverage. A rule without its input SKIPS, is named, and never counts as a pass. custom_rules always fire. One row is stored, linked to trace_id when given.
When not to use it. To validate arbitrary JSON Schema (the json_schema custom type asserts an output's shape only). To screen inputs before they reach an agent: no_injection_patterns inspects the agent's OUTPUT text for injection-shaped content — attack phrasing and structural directives the output echoes or complies with — and never reads the input, so it is not an input firewall. For semantic judgment, evaluate_with_llm_judge and verify_citations need a key you supply.
Returns. JSON with id (the evaluation id, readable at iris://evaluations/{id}); trace_id (the linked trace, when named); verdict (state, passed, basis (which layer decided), by (the rules), risk); coverage (per question: judged, unjudged and why, or not_applicable; plus the inputs carried); provenance (Iris version, ruleset and config hashes, thresholds, corpus version, time); erased_at (set once the linked trace was deleted); eval_type (the bundle that ran); score (0..1 weighted quality over the rules that ran); passed (the ship verdict; false when nothing was judged); rule_results (per rule: verdict, message, kind, role, question, saw, evidence, uncertainty); suggestions (what to change); rules_evaluated (rules that judged); rules_skipped (rules that skipped); insufficient_data (true when no rule could judge); critical_failures (critical rules that failed and vetoed passed); critical_skipped (critical rules that could not judge; treat as unknown); categories (per-bundle verdicts for eval_type all); note (present when eval_type was omitted).
Errors. IRIS_UNKNOWN_TRACE when trace_id names no stored trace — checked first, nothing scored or written. IRIS_STORAGE_ERROR when the row cannot be written. Unknown arguments or keys are refused before the handler runs, naming the valid ones; a regex rule over its budget or with a broken config reports skipped, not an error. Every failure returns {"error":{"code","message","recovery":[]}} with isError true; follow recovery before retrying.
Siblings. log_trace — record the execution first; evaluate_with_llm_judge — semantic scoring on your key; verify_citations — citation grounding on your key; list_rules — the roster, needs and published accuracy.
| Name | Required | Description | Default |
|---|---|---|---|
| input | No | Original input for context (the ask + any source material the agent was given) — REQUIRED when eval_type="relevance" (keyword_overlap and topic_consistency compare the output against it and skip without it); also grounds the safety bundle's hallucination signals | |
| output | Yes | The output text to evaluate (the agent's response that gets scored against rules) | |
| cost_usd | No | Cost in USD — consulted by the cost bundle (eval_type="cost" or "all") AND by any cost_threshold custom rule regardless of eval_type; omit it and such a rule skips rather than passes (a critical one is listed in critical_skipped) | |
| expected | No | Expected output for comparison — consulted only by the completeness bundle's expected_coverage rule; NOT used by relevance (the relevance rules compare the output against `input`) | |
| trace_id | No | Link evaluation to a trace — surfaces this eval in the dashboard's trace drill-through and lets the tool reuse the trace's stored tool_calls. Must be the id of a stored trace (from log_trace / get_traces); an unknown id is rejected before anything is evaluated | |
| eval_type | No | Rule bundle to apply: completeness | relevance | safety | cost | custom | all — picks which built-in rules fire. "all" runs every bundle in one call and adds a per-category breakdown. Defaults to "all" when omitted — every bundle runs, safety included, and the response carries a note saying the default ran | |
| tool_calls | No | What the agent DID — the tool calls it made, in order, each { tool_name, input?, output?, latency_ms?, error? } exactly as log_trace records them. Read by the trajectory rules — the rules that judge what the agent DID rather than what it wrote. Omit it and those rules SKIP rather than pass — an evaluation with no trajectory data reports "not judged", never "clean". When trace_id names a stored trace and this argument is omitted, the tool_calls stored on that trace are loaded and used, so a caller who already logged them need not resend them | |
| token_usage | No | Token usage breakdown — only consulted by the cost bundle (eval_type="cost" or "all"; used for token-budget rules) | |
| custom_rules | No | Custom evaluation rules, max 10 per call (deploy persistent rule sets via deploy_rule instead) — fires REGARDLESS of eval_type; pass eval_type="custom" if you want ONLY these. Each entry accepts exactly name, type, config, weight — an unknown key (e.g. a misspelled weight) is rejected |
| Name | Required | Description |
|---|---|---|
| id | Yes | the evaluation id, readable at iris://evaluations/{id} |
| note | No | present when eval_type was omitted |
| score | Yes | 0..1 weighted quality over the rules that ran |
| passed | Yes | the ship verdict; false when nothing was judged |
| verdict | No | state, passed, basis (which layer decided), by (the rules), risk |
| coverage | No | per question: judged, unjudged and why, or not_applicable; plus the inputs carried |
| trace_id | No | the linked trace, when named |
| erased_at | No | set once the linked trace was deleted |
| eval_type | Yes | the bundle that ran |
| categories | No | per-bundle verdicts for eval_type all |
| provenance | No | Iris version, ruleset and config hashes, thresholds, corpus version, time |
| suggestions | Yes | what to change |
| rule_results | Yes | per rule: verdict, message, kind, role, question, saw, evidence, uncertainty |
| rules_skipped | Yes | rules that skipped |
| rules_evaluated | Yes | rules that judged |
| critical_skipped | No | critical rules that could not judge; treat as unknown |
| critical_failures | No | critical rules that failed and vetoed passed |
| insufficient_data | Yes | true when no rule could judge |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description heavily exceeds the annotations' coverage: it states 'In-process, no network, no key', says 'One row is stored, linked to trace_id when given', describes skip semantics for missing inputs, and enumerates error codes (IRIS_UNKNOWN_TRACE, IRIS_STORAGE_ERROR) with recovery guidance. It does not contradict annotations such as idempotentHint or destructiveHint.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but tightly sectioned with clear headings: What it does / When not to use it / Returns / Errors / Siblings. The core purpose is front-loaded, and each section carries non-redundant information needed for a 9-parameter tool with a rich output schema. No filler or tautology is present.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite already having an output schema, the description still summarizes all key return fields, error conditions, default behavior, and skip conditions, making the tool's full contract explicit. It also names siblings and distinguishes their scope, so nothing essential for correct invocation is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is already 100%, but the description adds critical behavior beyond the schema: input is 'REQUIRED when eval_type="relevance"', cost_usd is read by the cost bundle and any cost_threshold custom rule, expected 'feeds only expected_coverage', and tool_calls 'SKIP rather than pass' if omitted. custom_rules 'always fire' regardless of eval_type. This goes far beyond the standalone parameter descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with 'Score an agent output against the deterministic rule bundles' — a specific verb, object, and judgment mechanism. It further distinguishes itself from siblings by naming them in the Siblings section ('evaluate_with_llm_judge — semantic scoring on your key; verify_citations — citation grounding'). An agent can quickly tell this is the deterministic rule-based evaluator.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is an explicit 'When not to use it' section that names three alternative cases — JSON Schema validation, input screening, and semantic judgment — and points to the correct sibling for each. It also explains eval_type bundle selection, the default ('Defaults to "all"'), and when to use custom_rules alone with eval_type='custom'.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Score an output with an LLM judge on your own provider key: a 0..1 score, a rationale, per-dimension sub-scores and the exact spend.
What it does. Calls Anthropic or OpenAI directly with the key in this process's environment (IRIS_ANTHROPIC_API_KEY or IRIS_OPENAI_API_KEY); Iris never proxies. template picks the question: accuracy, helpfulness, safety, correctness (needs expected) or faithfulness (needs source_material); input improves helpfulness and safety. model is required; provider is inferred from it. The worst-case spend — both attempts, full max_output_tokens — is computed BEFORE the call and refused if it exceeds max_cost_usd (default IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL or 0.25). temperature defaults to 0; a rate-limited call is retried once. One evaluation row is stored with the provider response id, tokens, cost and latency, linked to trace_id when given. The judge's own accuracy is measurable on a key you supply and is not yet published (see iris://proof).
When not to use it. For length, keyword, PII, injection or cost checks: evaluate_output is free and deterministic. Without a key: the call returns IRIS_JUDGE_NOT_ENABLED with the enable steps — do not search for them. On very large outputs without raising max_cost_usd: the pre-check refuses.
Returns. JSON with id (the evaluation id; read it back at iris://evaluations/{id}); trace_id (the linked trace, when one was named); score (0..1 from the judge); passed (the verdict: the score against the template's threshold, which is pass_threshold below. Not the model's own boolean — that is self_reported_pass); pass_threshold (the threshold the score was read against, so you can check the arithmetic); self_reported_pass (what the model said about passing, when it said anything. Recorded, never obeyed); disagreement (true when the model's own boolean disagrees with the threshold verdict — its rubric and its judgement have come apart on this output); rationale (the judge's reasoning, in its words); dimensions (per-dimension sub-scores for the template); model (the model that judged); provider (the provider called); template (the template used); input_tokens (tokens sent, across both attempts when a retry ran); output_tokens (tokens received, across both attempts when a retry ran); cost_usd (the exact spend from the pricing table); latency_ms (wall time of the provider call(s)); raw_response_id (the provider's response id, for your own audit).
Errors. IRIS_JUDGE_NOT_ENABLED (no key for the provider reached this process; recovery carries the steps). IRIS_JUDGE_UNKNOWN_MODEL (valid lists the models). IRIS_UNKNOWN_TRACE, checked before any spend. IRIS_BUDGET_EXCEEDED (nothing spent; the message carries both numbers). IRIS_PROVIDER_ERROR with kind auth, rate_limit, bad_request, server_error, timeout or malformed_response, and retryable set. Every failure returns {"error":{"code","message","recovery":[]}} with isError true; follow recovery before retrying.
Siblings. evaluate_output — the free deterministic path; verify_citations — citation grounding, the narrower judge; log_trace — record the execution first.
| Name | Required | Description | Default |
|---|---|---|---|
| input | No | User question / prompt that produced the output (improves accuracy for helpfulness/safety) | |
| model | Yes | Model ID. Supported: anthropic = claude-opus-4-7 | claude-sonnet-4-6 | claude-haiku-4-5 | claude-haiku-4-5-20251001; openai = gpt-4o | gpt-4o-mini | o1-mini. Required — cost varies a hundredfold across models | |
| output | Yes | The agent output text to evaluate | |
| expected | No | Reference answer (required for correctness template) | |
| provider | No | Auto-detected from model when omitted | |
| template | Yes | Judge dimension: accuracy (factual correctness), helpfulness (does it address the ask), safety (harm potential), correctness (vs reference answer — requires `expected`), faithfulness (RAG grounding — requires `source_material`). | |
| trace_id | No | Link this evaluation to a stored trace (id from log_trace / get_traces); an unknown id is rejected BEFORE the judge is called | |
| timeout_ms | No | Per-request timeout; default 60_000 | |
| temperature | No | Sampling temperature; default 0 (deterministic) | |
| max_cost_usd | No | Cost cap in USD for this call; defaults to IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL or 0.25. The worst case (two attempts, full max_output_tokens) is computed before the call and refused if it exceeds the cap | |
| source_material | No | Provided RAG sources (required for faithfulness template) | |
| max_output_tokens | No | Judge output token cap; default 512 |
| Name | Required | Description |
|---|---|---|
| id | Yes | the evaluation id; read it back at iris://evaluations/{id} |
| model | Yes | the model that judged |
| score | Yes | 0..1 from the judge |
| passed | Yes | the verdict: the score against the template's threshold, which is pass_threshold below. Not the model's own boolean — that is self_reported_pass |
| cost_usd | Yes | the exact spend from the pricing table |
| provider | Yes | the provider called |
| template | Yes | the template used |
| trace_id | No | the linked trace, when one was named |
| rationale | Yes | the judge's reasoning, in its words |
| dimensions | Yes | per-dimension sub-scores for the template |
| latency_ms | Yes | wall time of the provider call(s) |
| disagreement | No | true when the model's own boolean disagrees with the threshold verdict — its rubric and its judgement have come apart on this output |
| input_tokens | Yes | tokens sent, across both attempts when a retry ran |
| output_tokens | Yes | tokens received, across both attempts when a retry ran |
| pass_threshold | Yes | the threshold the score was read against, so you can check the arithmetic |
| raw_response_id | No | the provider's response id, for your own audit |
| self_reported_pass | No | what the model said about passing, when it said anything. Recorded, never obeyed |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description goes far beyond annotations, disclosing that Iris never proxies, that spend is pre-computed and capped before the call, that rate-limited calls retry once, that temperature defaults to 0, and that a row is stored with provider response id/tokens/cost/latency. It even explains that self_reported_pass is recorded but never obeyed, and that the pre-check refuses before spending. No contradiction with annotations (readOnlyHint=false, destructiveHint=false) — the write/store behavior is consistent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long, but every section earns its place: What it does, When not to use it, Returns, Errors, Siblings. It is structured with headers and front-loaded with the essential purpose and cost/behavior caveats. Some redundancy exists (the template requirements are stated both in prose and in schema descriptions, and the return field list is exhaustive), but the structure makes it navigable and scannable. Not a 5 because it could be tightened by trimming the full return-field enumeration that the output schema already documents.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 12-parameter, externally-calling, cost-bearing tool with an output schema, the description covers everything an agent needs: what happens before the call (budget pre-check, trace validation), what happens during (retry, direct provider call), what is returned (mapped to the output schema), what errors look like (codes + recovery), and explicit when-not-to-use conditions. The error-code enumeration is especially valuable because the output schema exists but the error shape is partially described. This is as complete as a description of this complexity practically needs to be.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3, but the description adds substantive meaning beyond the schema: it explains template-purpose mapping in prose, clarifies model cost variance ('hundredfold'), states how max_cost_usd is applied ('worst case computed before the call'), and defines temperature default determinism. It doesn't fully re-explain every parameter, but it enriches the key decision parameters enough to push above baseline.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb ('Score an output'), a precise resource ('with an LLM judge on your own provider key'), and enumerates concrete outputs (0..1 score, rationale, sub-scores, spend). It clearly distinguishes itself from siblings by naming exactly what it is not (evaluate_output, verify_citations) and describing its provider-direct behavior. An agent can identify what this tool does without guessing.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description contains an explicit 'When not to use it' section that names alternatives (evaluate_output for deterministic checks) and conditions (no key -> not enabled; large outputs -> budget refusal). It also explains template selection requirements (correctness needs expected, faithfulness needs source_material) and how provider/model relate. This is comprehensive routing guidance beyond what the schema provides.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Query stored traces with filters, pagination and sorting; optionally include the dashboard summary in the same response.
What it does. Read-only, local storage only. Filters are exact-match (agent_name, framework), inclusive time bounds (since, until — an ISO 8601 timestamp or date) and a score range applied to the LATEST evaluation of each trace (min_score, max_score, 0..1). limit is 1..1000 (default 50), offset counts from 0, sort_by is timestamp, latency_ms or cost_usd, sort_order asc or desc (default: newest first). include_summary adds the one-hour dashboard aggregates. A crossed range (min above max, since after until) is refused naming both values rather than returning an empty page that reads as "no such traces".
When not to use it. To score a trace (evaluate_output). To create one (log_trace). As a live stream: this is a query, and Iris has no event stream — poll with backoff.
Returns. JSON with traces (the page of traces: trace_id, agent_name, framework, input, output, tool_calls, latency_ms, token_usage, cost_usd, metadata, timestamp); total (how many traces match the filters, across every page); limit (the page size applied); offset (the offset applied); summary (the dashboard aggregates for the last hour, when include_summary was true).
Errors. IRIS_STORAGE_ERROR when the database cannot be read. An out-of-range or crossed bound is refused before the handler runs, naming the values. An empty result is total 0, not an error. Every failure returns {"error":{"code","message","recovery":[]}} with isError true; follow recovery before retrying.
Siblings. log_trace — record an execution; evaluate_output — score one output; delete_trace — remove one trace.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Results per page (default 50, max 1000 — values above are rejected) | |
| since | No | ISO 8601 timestamp (or date) lower bound — return traces with timestamp >= this; anything that is not an ISO timestamp is rejected, never treated as "no bound" | |
| until | No | ISO 8601 timestamp (or date) upper bound — return traces with timestamp <= this; must not be earlier than `since` | |
| offset | No | Zero-based pagination offset — skip first N results (non-negative integer) | |
| sort_by | No | Sort by timestamp | latency_ms | cost_usd (default timestamp) | timestamp |
| framework | No | Filter by agent framework — exact match (e.g., langchain, autogen) | |
| max_score | No | Maximum eval score filter (0..1; values outside are rejected) — applied to LATEST eval per trace | |
| min_score | No | Minimum eval score filter (0..1; values outside are rejected) — applied to LATEST eval per trace, not all evals; must be <= max_score when both are set | |
| agent_name | No | Filter by agent name — exact match (no wildcards) | |
| sort_order | No | Sort order: asc | desc (default desc — most recent / highest first) | desc |
| include_summary | No | Include dashboard summary stats in same response — saves a round-trip when ingesting for dashboards |
| Name | Required | Description |
|---|---|---|
| limit | Yes | the page size applied |
| total | Yes | how many traces match the filters, across every page |
| offset | Yes | the offset applied |
| traces | Yes | the page of traces: trace_id, agent_name, framework, input, output, tool_calls, latency_ms, token_usage, cost_usd, metadata, timestamp |
| summary | No | the dashboard aggregates for the last hour, when include_summary was true |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint and idempotentHint, and the description reinforces and extends this by stating read-only, local-storage-only behavior. It also discloses exact-match semantics, inclusive time bounds, LATEST-eval score filtering, crossed-range refusal, error shape, empty-result behavior, and the absence of an event stream — far beyond what annotations alone convey.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but well structured with clear headings (What it does, When not to use it, Returns, Errors, Siblings) and a front-loaded summary sentence. Each section earns its place by covering important behavior, errors, and routing for an 11-parameter tool, so the length is appropriate to the complexity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity, 11 parameters, and rich annotations, the description is complete: it covers filters, pagination, sorting, optional summary, return shape, error codes, empty results, and sibling differentiation. An agent has everything needed to invoke it correctly and handle failures.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, but the description adds material meaning beyond the schema: exact-match vs wildcard behavior, inclusive bounds, score applied to the LATEST evaluation, default sort order, and the crossed-range validation rule that names both offending values. This is genuinely helpful parameter semantics rather than schema repetition.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The opening line states a specific verb and resource — 'Query stored traces with filters, pagination and sorting' — and the tool is immediately distinguished from siblings in the Siblings section. The description also names what it is not for (scoring, creating, live streaming), so an agent can tell it apart without inspecting other tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is an explicit 'When not to use it' section that names alternatives (evaluate_output, log_trace) and explains that this is a query, not a live stream, with polling guidance. This gives clear routing rules both for when to use the tool and when to choose a sibling.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
The rule inventory: the built-in roster with what each rule needs, the criticality this server applies and its published accuracy, plus every deployed custom rule.
What it does. Read-only, no network. built_in is the shipped roster and is never narrowed by the filters. For each rule: kind (measurement, detection, inference, judgment, policy, verification), mechanism, needs (the inputs it reads — absent means the rule skips, never passes), question, classes, version, weight, the EFFECTIVE critical flag with criticalSource (default, or config when eval.criticalRules / eval.nonCriticalRules changed it on this server — read it before trusting a passed: true), and proof: precision and recall with 95% intervals and the positive predictive value at four prevalences, the numbers published at https://iris-eval.com/proof. rules is the custom-rule store, filterable by eval_type and enabled_only; total and enabled_count count custom rules. quarantined lists store entries this version could not validate; they do not fire.
When not to use it. To count traces (get_traces). To add, remove or pause a rule (deploy_rule, delete_rule). Built-in rules are not in the store and cannot be deployed, deleted or disabled.
Returns. JSON with rules (the deployed custom rules after the filters: id, name, description, evalType, severity, definition, enabled, createdAt, updatedAt, version, sourceMomentId); total (custom rules after the filters); enabled_count (of those, how many are enabled); built_in (the shipped roster, never filtered: name, category, description, weight, kind, mechanism, needs, question, classes, version, the EFFECTIVE critical flag with criticalSource, and proof (published precision, recall, intervals and ppvAt from https://iris-eval.com/proof; null where the proof is a conformance check)); quarantined (entries in the store this version could not validate; they do not fire and are never deleted by a deploy).
Errors. IRIS_INTERNAL_ERROR if the store file cannot be read. A missing store file is an empty list, not an error. Every failure returns {"error":{"code","message","recovery":[]}} with isError true; follow recovery before retrying.
Siblings. deploy_rule — add a custom rule; delete_rule — remove, disable or re-enable one; evaluate_output — run the rules.
| Name | Required | Description | Default |
|---|---|---|---|
| eval_type | No | Filter the custom rules to one eval category (exact match); built_in is never filtered | |
| enabled_only | No | Return only enabled custom rules (a rule disabled with delete_rule stays in the store and does not fire) |
| Name | Required | Description |
|---|---|---|
| rules | Yes | the deployed custom rules after the filters: id, name, description, evalType, severity, definition, enabled, createdAt, updatedAt, version, sourceMomentId |
| total | Yes | custom rules after the filters |
| built_in | Yes | the shipped roster, never filtered: name, category, description, weight, kind, mechanism, needs, question, classes, version, the EFFECTIVE critical flag with criticalSource, and proof (published precision, recall, intervals and ppvAt from https://iris-eval.com/proof; null where the proof is a conformance check) |
| quarantined | Yes | entries in the store this version could not validate; they do not fire and are never deleted by a deploy |
| enabled_count | Yes | of those, how many are enabled |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare the tool read-only and idempotent, and the description adds substantial behavioral context: no network access, built_in is never filtered, filters apply only to custom rules, quarantined rules do not fire, and a missing store file is treated as an empty list rather than an error. The error response shape is also disclosed, which goes well beyond the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but exceptionally well structured with clear sections: What it does, When not to use it, Returns, Errors, and Siblings. Every section answers a likely agent question, and the most important scoping information is front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers purpose, parameter behavior, return structure, error handling, non-usage cases, and sibling relationships. Given the tool's complexity, this is complete enough for an agent to invoke it correctly and interpret its response without additional probing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Although schema coverage is 100%, the description adds important semantic detail: eval_type only filters custom rules and never built_in, and enabled_only interacts with delete_rule semantics (disabled rules stay in the store but do not fire). This goes beyond the schema's short descriptions and clarifies edge behavior.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly defines the tool as a rule inventory covering both built-in and custom rules, listing exact fields returned and the distinction between the two stores. It also explicitly names sibling tools to contrast with, so an agent can select this tool without ambiguity.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The 'When not to use it' section names the exact alternatives (get_traces, deploy_rule, delete_rule) and explains that built-in rules cannot be modified. This gives an agent concrete routing guidance and exclusion criteria, which is more than most tool descriptions provide.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Store one agent execution — input, output, tool calls, spans, cost, latency, token usage — and get the trace_id every later call keys on.
What it does. Writes one trace row to local SQLite and mints a fresh trace_id; nothing is deduplicated, so resubmitting the same payload stores a second trace. Only agent_name is required. Store what you have: tool_calls so the trajectory rules can later judge what the agent did, cost_usd and token_usage so the cost rules can, input and output so everything else can. When IRIS_OTEL_ENDPOINT is set the trace is also exported to that collector, best-effort and asynchronous; the local write never waits on it. Traces are immutable: there is no update path. In stdio mode nothing authenticates the caller; over HTTP a Bearer token is required only when an API key is configured.
When not to use it. For a transient log line (use your logger). To score an output: log first, then call evaluate_output with the trace_id, which also lets it reuse the stored tool_calls. To change a stored trace: delete_trace and log again.
Returns. JSON with trace_id (the stored trace id, 32 hex — pass it to evaluate_output, get_traces or delete_trace); status (always "stored" on success).
Errors. IRIS_STORAGE_ERROR when the database cannot be written. An unknown argument or a malformed span or tool_calls entry is refused before the handler runs, naming the valid keys. Every failure returns {"error":{"code","message","recovery":[]}} with isError true; follow recovery before retrying.
Siblings. evaluate_output — score the stored output; get_traces — query what was logged; delete_trace — remove one trace.
| Name | Required | Description | Default |
|---|---|---|---|
| input | No | Agent input text — the user prompt or upstream input that produced this output | |
| spans | No | Detailed execution spans (hierarchical span tree with timings, attributes, events); a span without start_time takes the trace timestamp | |
| output | No | Agent output text — what the agent produced (pass to evaluate_output for scoring) | |
| cost_usd | No | Total cost in USD — overrides per-span aggregation when provided (treated as authoritative) | |
| metadata | No | Opaque key-value tags (e.g. {requestId, userId, env}) — queryable in dashboard, not via get_traces filters | |
| framework | No | Agent framework identifier (e.g., langchain, autogen, custom) | |
| timestamp | No | Trace timestamp (ISO 8601); defaults to now() when omitted | |
| agent_name | Yes | Agent name — used for filtering in get_traces (e.g., "customer-support-bot") | |
| latency_ms | No | Total execution time in milliseconds (end-to-end agent latency) | |
| tool_calls | No | Tool calls made during execution, in order, each { tool_name, input?, output?, latency_ms?, error? } — what the trajectory rules judge; evaluate_output reuses them when given this trace_id | |
| token_usage | No | Token usage breakdown (prompt/completion/total — used for cost analysis) |
| Name | Required | Description |
|---|---|---|
| status | Yes | always "stored" on success |
| trace_id | Yes | the stored trace id, 32 hex — pass it to evaluate_output, get_traces or delete_trace |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only indicate readOnlyHint=false, openWorldHint=false, idempotentHint=false, destructiveHint=false. The description goes far beyond those hints: it discloses that traces are immutable with no update path, that resubmission stores a second trace (no deduplication), that the local write never waits on the best-effort async OTel export, and that authentication is optional in stdio mode but requires a Bearer token over HTTP when an API key is configured. It also explains error behavior and return shape. No contradiction with annotations; the description carries the full burden and succeeds.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but every section earns its place and is heavily front-loaded: the first sentence states the core action and the returned key. The 'What it does', 'When not to use it', 'Returns', 'Errors', and 'Siblings' labels give it scannable structure. It is verbose, but the density of decision-relevant information justifies the length. Loses one point for minor redundancy (e.g., restating evaluate_output reuse in both the What and param description).
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given an output schema exists (describing the return JSON), 11 parameters with 100% schema coverage, and a rich nested schema, the description is complete. It covers the operation's side effects (immutable write, second trace on resubmit), error contract, authentication nuance, and sibling routing. An agent has everything needed to decide when to call log_trace and what to expect.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents all 11 parameters in detail (e.g., agent_name is used for filtering in get_traces, cost_usd overrides per-span aggregation, metadata is queryable in dashboard but not via get_traces filters). The description adds cross-parameter context (tool_calls for trajectory rules, cost_usd/token_usage for cost rules, input/output for everything else), but much of that is already in the field descriptions. Baseline 3 is appropriate: the description complements rather than compensates.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a precise verb-object pair, 'Store one agent execution', and enumerates exactly what a trace captures (input, output, tool calls, spans, cost, latency, token usage) and what it returns (trace_id). It clearly distinguishes log_trace from siblings like evaluate_output and get_traces by naming them and their purposes. This is a specific, actionable statement of what the tool does.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The 'When not to use it' section explicitly routes the agent: use a logger for transient log lines, call evaluate_output after logging to score, and use delete_trace then log again to modify a stored trace. It also gives a conditional alternative (IRIS_OTEL_ENDPOINT export behavior). This is exemplary guidance for selecting log_trace versus alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Extract the citations in an output, fetch the sources (opt-in, SSRF-guarded) and ask an LLM judge on your key whether each source supports its claim.
What it does. Three phases. Extraction, no network: [N] references, (Author, Year), bare URLs and DOIs. Fetch of URL and DOI citations only when allow_fetch is true or IRIS_CITATION_ALLOW_FETCH=1, through a scheme allowlist, private and cloud-metadata address blocking, an optional hostname allowlist (domain_allowlist, merged with IRIS_CITATION_DOMAINS), a per-source timeout and byte cap, and at most three re-checked redirects. Then one judge call per resolved citation on your own key, reading the first part of each source, capped in total by max_cost_usd_total. Up to max_citations are verified; extras are skipped, not errored. overall_score is supported / judged and null when nothing was judged. Per-citation failures (bad scheme, blocked address, timeout, too large, cost cap, fetch disabled) are reported on the citation, never scored as unsupported. One evaluation row is stored.
When not to use it. When the output has no citations: the score is null, and evaluate_output's hallucination signals are the cheap check. Without a key (IRIS_ANTHROPIC_API_KEY or IRIS_OPENAI_API_KEY): the call returns IRIS_JUDGE_NOT_ENABLED with the enable steps. With fetch enabled and an open allowlist on untrusted output: you are running a user-directed fetcher — set IRIS_CITATION_DOMAINS.
Returns. JSON with id (the evaluation id; read it back at iris://evaluations/{id}); trace_id (the linked trace, when one was named); overall_score (supported / judged; null when nothing was judged); passed (true when every judged citation was supported; false when any judged citation was not; NULL when nothing was judged — no verdict, because nothing was verified. Until 0.10.0 that last case returned true.); total_unsupported (judged citations the judge ruled unsupported — the number the verdict turns on); total_citations_found (citations extracted from the output); total_resolved (citations whose source was fetched); total_judged (citations the judge ruled on); total_supported (citations the judge found supported); total_cost_usd (the spend across every judge call); citations (per citation: the citation (raw, kind, identifier, offsets), resolve_status ok | skipped | error, resolve_error, source (url, status, content_type, bytes_fetched, truncated), judge (supported, confidence, rationale, cost_usd, latency_ms, tokens)).
Errors. IRIS_JUDGE_NOT_ENABLED, IRIS_JUDGE_UNKNOWN_MODEL and IRIS_UNKNOWN_TRACE before any fetch or spend. IRIS_JUDGE_FAILED when citations resolved but the judge failed on every one — an error, not a passing verdict; nothing is stored. Every failure returns {"error":{"code","message","recovery":[]}} with isError true; follow recovery before retrying.
Siblings. evaluate_with_llm_judge — general semantic scoring; evaluate_output — the free deterministic path, including the hallucination signals; log_trace — record the execution first.
| Name | Required | Description | Default |
|---|---|---|---|
| model | Yes | Judge model for per-citation verification. Supported: anthropic = claude-opus-4-7 | claude-sonnet-4-6 | claude-haiku-4-5 | claude-haiku-4-5-20251001; openai = gpt-4o | gpt-4o-mini | o1-mini. | |
| output | Yes | The agent output containing citations to verify | |
| provider | No | Auto-detected from model when omitted | |
| trace_id | No | Link verification result to a stored trace (id from log_trace / get_traces); an unknown id is rejected before any fetch or judge call | |
| allow_fetch | No | Permit outbound HTTP to resolve URLs/DOIs. Defaults to IRIS_CITATION_ALLOW_FETCH=1; false otherwise. SSRF-guarded regardless. | |
| max_citations | No | Max citations to verify (extras skipped, not errored); default 20, at most 50 | |
| domain_allowlist | No | Restrict fetches to hostnames in this list (suffix match allowed). Merged with IRIS_CITATION_DOMAINS env. | |
| max_cost_usd_total | No | Cap TOTAL judge cost across all citations in this call; default 1.00 USD — the pipeline stops when the next call would exceed it | |
| per_source_max_bytes | No | Per-URL body cap; default 5MB | |
| per_source_timeout_ms | No | Per-URL fetch timeout; default 10_000 |
| Name | Required | Description |
|---|---|---|
| id | Yes | the evaluation id; read it back at iris://evaluations/{id} |
| passed | Yes | true when every judged citation was supported; false when any judged citation was not; NULL when nothing was judged — no verdict, because nothing was verified. Until 0.10.0 that last case returned true. |
| trace_id | No | the linked trace, when one was named |
| citations | Yes | per citation: the citation (raw, kind, identifier, offsets), resolve_status ok | skipped | error, resolve_error, source (url, status, content_type, bytes_fetched, truncated), judge (supported, confidence, rationale, cost_usd, latency_ms, tokens) |
| total_judged | Yes | citations the judge ruled on |
| overall_score | Yes | supported / judged; null when nothing was judged |
| total_cost_usd | Yes | the spend across every judge call |
| total_resolved | Yes | citations whose source was fetched |
| total_supported | Yes | citations the judge found supported |
| total_unsupported | Yes | judged citations the judge ruled unsupported — the number the verdict turns on |
| total_citations_found | Yes | citations extracted from the output |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond annotations, the description discloses: fetch is opt-in via allow_fetch or env var, SSRF protections (scheme allowlist, blocked private/cloud-metadata addresses, hostname allowlist, timeouts, byte caps, redirect limits), cost caps, per-citation failures never scored as unsupported, one evaluation row stored, and the null-verdict behavior including a 0.10.0 change note. It also explains errors like IRIS_JUDGE_FAILED. Annotations are minimal (readOnly=false, openWorld=true, idempotent=false, destructive=false), so the description carries the burden and does so thoroughly.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but earns its length: it uses labeled sections (What it does, When not to use it, Returns, Errors, Siblings) and front-loads the headline behavior. Some redundancy exists (e.g., 'When not to use it' repeats the key requirement mentioned in the opener), and the Returns section is dense, but the structure makes it navigable.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 10-parameter, complex tool with an output schema, the description covers the pipeline phases, error codes, env var interactions, security posture, edge cases, and return fields. The output schema exists, so the description need not spell out return shapes, but it still summarizes them and adds the critical behavioral notes about passed/NULL and stored evaluations. Nothing essential is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema documents every parameter meaning. The description adds operational context that interacts with parameters — e.g., allow_fetch defaulting from env, domain_allowlist merging with env, max_citations extras being skipped, max_cost_usd_total behavior — but the core semantic meaning per parameter is already in the schema. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb phrase — 'Extract the citations in an output, fetch the sources (opt-in, SSRF-guarded) and ask an LLM judge on your key whether each source supports its claim.' This precisely states the tool's operation and resource. It differentiates from siblings by name ('verify_citations') and behavior, and the Siblings section explicitly distinguishes it from evaluate_with_llm_judge and evaluate_output.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description includes an explicit 'When not to use it' section naming concrete conditions: no citations, no key, and the risk of an open allowlist on untrusted output. It even points to evaluate_output's hallucination signals as the cheap alternative. This is exemplary guidance for when to use the tool vs alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
v0.10.0evaluate_with_llm_judge5 fields changedOutput schema / properties / disagreementAdded value: +{
+ "description": "true when the model's own boolean disagrees with the threshold verdict — its rubric and its judgement have come apart on this output",
+ "type": "boolean"
+}Output schema / properties / pass_thresholdAdded value: +{
+ "description": "the threshold the score was read against, so you can check the arithmetic",
+ "type": "number"
+}Output schema / properties / passed / descriptionPrevious value: -"the judge's own pass verdict for the template"New value: +"the verdict: the score against the template's threshold, which is pass_threshold below. Not the model's own boolean — that is self_reported_pass"Output schema / properties / self_reported_passAdded value: +{
+ "description": "what the model said about passing, when it said anything. Recorded, never obeyed",
+ "type": "boolean"
+}Output schema / requiredPrevious value: -[
- "id",
- "score",
- "passed",
- "rationale",
- "dimensions",
- "model",
- "provider",
- "template",
- "input_tokens",
- "output_tokens",
- "cost_usd",
- "latency_ms"
-]New value: +[
+ "id",
+ "score",
+ "passed",
+ "pass_threshold",
+ "rationale",
+ "dimensions",
+ "model",
+ "provider",
+ "template",
+ "input_tokens",
+ "output_tokens",
+ "cost_usd",
+ "latency_ms"
+]verify_citations4 fields changedOutput schema / properties / passed / descriptionPrevious value: -"true when every judged citation was supported, or nothing was judged and nothing failed"New value: +"true when every judged citation was supported; false when any judged citation was not; NULL when nothing was judged — no verdict, because nothing was verified. Until 0.10.0 that last case returned true."Output schema / properties / passed / typePrevious value: -"boolean"New value: +[
+ "boolean",
+ "null"
+]Output schema / properties / total_unsupportedAdded value: +{
+ "description": "judged citations the judge ruled unsupported — the number the verdict turns on",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+}Output schema / requiredPrevious value: -[
- "id",
- "overall_score",
- "passed",
- "total_citations_found",
- "total_resolved",
- "total_judged",
- "total_supported",
- "total_cost_usd",
- "citations"
-]New value: +[
+ "id",
+ "overall_score",
+ "passed",
+ "total_unsupported",
+ "total_citations_found",
+ "total_resolved",
+ "total_judged",
+ "total_supported",
+ "total_cost_usd",
+ "citations"
+]v0.9.0delete_rule1 field changedOutput schema / (root)Previous value: -nullNew value: +{
+ "$schema": "http://json-schema.org/draft-07/schema#",
+ "additionalProperties": {},
+ "properties": {
+ "deleted": {
+ "description": "true when a rule was removed; always false on a toggle",
+ "type": "boolean"
+ },
+ "enabled": {
+ "description": "toggle only: the rule's state after the call",
+ "type": "boolean"
+ },
+ "rule": {
+ "additionalProperties": {},
+ "description": "toggle only: the rule as stored",
+ "properties": {},
+ "type": "object"
+ },
+ "rule_id": {
+ "description": "the id that was asked for",
+ "type": "string"
+ },
+ "toggled": {
+ "description": "toggle only: true when the rule exists (also when it was already in the requested state)",
+ "type": "boolean"
+ }
+ },
+ "required": [
+ "deleted",
+ "rule_id"
+ ],
+ "type": "object"
+}delete_trace1 field changedOutput schema / (root)Previous value: -nullNew value: +{
+ "$schema": "http://json-schema.org/draft-07/schema#",
+ "additionalProperties": {},
+ "properties": {
+ "deleted": {
+ "description": "true when a trace row was removed; false when no trace with that id existed for this tenant",
+ "type": "boolean"
+ },
+ "trace_id": {
+ "description": "the id that was asked for",
+ "type": "string"
+ }
+ },
+ "required": [
+ "deleted",
+ "trace_id"
+ ],
+ "type": "object"
+}deploy_rule1 field changedOutput schema / (root)Previous value: -nullNew value: +{
+ "$schema": "http://json-schema.org/draft-07/schema#",
+ "additionalProperties": {},
+ "properties": {
+ "replaced": {
+ "description": "with replace: true, the earlier rule(s) of the same name that were retired",
+ "items": {
+ "additionalProperties": {},
+ "properties": {
+ "id": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "id"
+ ],
+ "type": "object"
+ },
+ "type": "array"
+ },
+ "rule": {
+ "additionalProperties": {},
+ "description": "the rule as persisted: id (rule-<hex>, keep it for delete_rule), name, description, evalType, severity, definition, enabled, createdAt, updatedAt, version, sourceMomentId",
+ "properties": {
+ "id": {
+ "type": "string"
+ },
+ "name": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "id",
+ "name"
+ ],
+ "type": "object"
+ },
+ "warning": {
+ "description": "with replace: true, one sentence naming what was retired",
+ "type": "string"
+ }
+ },
+ "required": [
+ "rule"
+ ],
+ "type": "object"
+}evaluate_output2 fields changedInput schema / properties / trace_id / descriptionPrevious value: -"Link evaluation to a trace — surfaces this eval in the dashboard's trace drill-through. Must be the id of a stored trace (from log_trace / get_traces); an unknown id is rejected before anything is evaluated"New value: +"Link evaluation to a trace — surfaces this eval in the dashboard's trace drill-through and lets the tool reuse the trace's stored tool_calls. Must be the id of a stored trace (from log_trace / get_traces); an unknown id is rejected before anything is evaluated"Output schema / (root)Previous value: -nullNew value: +{
+ "$schema": "http://json-schema.org/draft-07/schema#",
+ "additionalProperties": {},
+ "properties": {
+ "categories": {
+ "additionalProperties": {
+ "additionalProperties": {},
+ "properties": {
+ "critical_failures": {
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "critical_skipped": {
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "insufficient_data": {
+ "type": "boolean"
+ },
+ "passed": {
+ "type": [
+ "boolean",
+ "null"
+ ]
+ },
+ "rules_evaluated": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "rules_skipped": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "score": {
+ "type": [
+ "number",
+ "null"
+ ]
+ }
+ },
+ "required": [
+ "score",
+ "passed",
+ "rules_evaluated",
+ "rules_skipped",
+ "insufficient_data"
+ ],
+ "type": "object"
+ },
+ "description": "per-bundle verdicts for eval_type all",
+ "propertyNames": {
+ "type": "string"
+ },
+ "type": "object"
+ },
+ "coverage": {
+ "additionalProperties": {},
+ "description": "per question: judged, unjudged and why, or not_applicable; plus the inputs carried",
+ "properties": {
+ "dormant": {
+ "items": {
+ "additionalProperties": {},
+ "properties": {
+ "name": {
+ "type": "string"
+ },
+ "reason": {
+ "type": "string"
+ },
+ "ruleId": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "ruleId",
+ "name",
+ "reason"
+ ],
+ "type": "object"
+ },
+ "type": "array"
+ },
+ "inputs": {
+ "additionalProperties": {
+ "type": "boolean"
+ },
+ "propertyNames": {
+ "type": "string"
+ },
+ "type": "object"
+ },
+ "questions": {
+ "items": {
+ "additionalProperties": {},
+ "properties": {
+ "id": {
+ "enum": [
+ "safe_output",
+ "grounded",
+ "complete",
+ "relevant",
+ "task_completed",
+ "tool_use_correct",
+ "within_budget"
+ ],
+ "type": "string"
+ },
+ "status": {
+ "enum": [
+ "judged",
+ "unjudged",
+ "not_applicable"
+ ],
+ "type": "string"
+ },
+ "why": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "id",
+ "status"
+ ],
+ "type": "object"
+ },
+ "type": "array"
+ }
+ },
+ "required": [
+ "inputs",
+ "questions"
+ ],
+ "type": "object"
+ },
+ "critical_failures": {
+ "description": "critical rules that failed and vetoed passed",
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "critical_skipped": {
+ "description": "critical rules that could not judge; treat as unknown",
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "erased_at": {
+ "description": "set once the linked trace was deleted",
+ "type": "string"
+ },
+ "eval_type": {
+ "description": "the bundle that ran",
+ "enum": [
+ "completeness",
+ "relevance",
+ "safety",
+ "cost",
+ "custom",
+ "all"
+ ],
+ "type": "string"
+ },
+ "id": {
+ "description": "the evaluation id, readable at iris://evaluations/{id}",
+ "type": "string"
+ },
+ "insufficient_data": {
+ "description": "true when no rule could judge",
+ "type": "boolean"
+ },
+ "note": {
+ "description": "present when eval_type was omitted",
+ "type": "string"
+ },
+ "passed": {
+ "description": "the ship verdict; false when nothing was judged",
+ "type": "boolean"
+ },
+ "provenance": {
+ "additionalProperties": {},
+ "description": "Iris version, ruleset and config hashes, thresholds, corpus version, time",
+ "properties": {
+ "configHash": {
+ "type": "string"
+ },
+ "corpusVersion": {
+ "type": "string"
+ },
+ "irisVersion": {
+ "type": "string"
+ },
+ "judgedAt": {
+ "type": "string"
+ },
+ "rulesetHash": {
+ "type": "string"
+ },
+ "thresholds": {
+ "additionalProperties": {},
+ "properties": {
+ "default": {
+ "type": "number"
+ },
+ "perRule": {
+ "additionalProperties": {},
+ "propertyNames": {
+ "type": "string"
+ },
+ "type": "object"
+ }
+ },
+ "required": [
+ "default"
+ ],
+ "type": "object"
+ }
+ },
+ "required": [
+ "irisVersion",
+ "rulesetHash",
+ "configHash",
+ "thresholds",
+ "corpusVersion",
+ "judgedAt"
+ ],
+ "type": "object"
+ },
+ "rule_results": {
+ "description": "per rule: verdict, message, kind, role, question, saw, evidence, uncertainty",
+ "items": {
+ "additionalProperties": {},
+ "properties": {
+ "budgetExceeded": {
+ "type": "boolean"
+ },
+ "category": {
+ "enum": [
+ "completeness",
+ "relevance",
+ "safety",
+ "cost",
+ "custom"
+ ],
+ "type": "string"
+ },
+ "classes": {
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "configInvalid": {
+ "type": "boolean"
+ },
+ "critical": {
+ "type": "boolean"
+ },
+ "criticalSource": {
+ "enum": [
+ "default",
+ "config"
+ ],
+ "type": "string"
+ },
+ "evidence": {
+ "items": {
+ "oneOf": [
+ {
+ "additionalProperties": {},
+ "properties": {
+ "end": {
+ "maximum": 9007199254740991,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "label": {
+ "type": "string"
+ },
+ "source": {
+ "type": "string"
+ },
+ "start": {
+ "maximum": 9007199254740991,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "type": {
+ "const": "span",
+ "type": "string"
+ }
+ },
+ "required": [
+ "type",
+ "source",
+ "start",
+ "end",
+ "label"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": {},
+ "properties": {
+ "count": {
+ "maximum": 9007199254740991,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "name": {
+ "type": "string"
+ },
+ "type": {
+ "const": "pattern",
+ "type": "string"
+ }
+ },
+ "required": [
+ "type",
+ "name",
+ "count"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": {},
+ "properties": {
+ "index": {
+ "maximum": 9007199254740991,
+ "minimum": 0,
+ "type": "integer"
+ },
+ "label": {
+ "type": "string"
+ },
+ "toolName": {
+ "type": "string"
+ },
+ "type": {
+ "const": "toolCall",
+ "type": "string"
+ }
+ },
+ "required": [
+ "type",
+ "index",
+ "toolName",
+ "label"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": {},
+ "properties": {
+ "status": {
+ "enum": [
+ "resolved",
+ "dead",
+ "unverifiable",
+ "supported",
+ "unsupported"
+ ],
+ "type": "string"
+ },
+ "type": {
+ "const": "citation",
+ "type": "string"
+ },
+ "url": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "type",
+ "url",
+ "status"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": {},
+ "properties": {
+ "stat": {
+ "type": "string"
+ },
+ "threshold": {
+ "type": "number"
+ },
+ "thresholdSource": {
+ "enum": [
+ "default",
+ "config",
+ "call",
+ "rule"
+ ],
+ "type": "string"
+ },
+ "type": {
+ "const": "count",
+ "type": "string"
+ },
+ "unit": {
+ "type": "string"
+ },
+ "value": {
+ "type": "number"
+ }
+ },
+ "required": [
+ "type",
+ "stat",
+ "unit",
+ "value"
+ ],
+ "type": "object"
+ }
+ ]
+ },
+ "type": "array"
+ },
+ "kind": {
+ "enum": [
+ "measurement",
+ "detection",
+ "inference",
+ "judgment",
+ "policy",
+ "verification"
+ ],
+ "type": "string"
+ },
+ "message": {
+ "type": "string"
+ },
+ "passed": {
+ "type": "boolean"
+ },
+ "question": {
+ "enum": [
+ "safe_output",
+ "grounded",
+ "complete",
+ "relevant",
+ "task_completed",
+ "tool_use_correct",
+ "within_budget"
+ ],
+ "type": "string"
+ },
+ "role": {
+ "enum": [
+ "gate",
+ "veto",
+ "risk",
+ "advisory",
+ "term"
+ ],
+ "type": "string"
+ },
+ "ruleId": {
+ "type": "string"
+ },
+ "ruleName": {
+ "type": "string"
+ },
+ "ruleVersion": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "saw": {
+ "items": {
+ "enum": [
+ "output",
+ "input",
+ "expected",
+ "tool_calls",
+ "tool_outputs",
+ "tools_catalogue",
+ "cost",
+ "tokens",
+ "citations"
+ ],
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "score": {
+ "type": "number"
+ },
+ "skipClass": {
+ "enum": [
+ "not_applicable",
+ "defeated",
+ "config_invalid"
+ ],
+ "type": "string"
+ },
+ "skipReason": {
+ "type": "string"
+ },
+ "skipped": {
+ "type": "boolean"
+ },
+ "uncertainty": {
+ "oneOf": [
+ {
+ "additionalProperties": {},
+ "properties": {
+ "basis": {
+ "const": "published_accuracy",
+ "type": "string"
+ },
+ "corpus": {
+ "additionalProperties": {},
+ "properties": {
+ "fn": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "fp": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "labelling": {
+ "enum": [
+ "same-model",
+ "human-verified"
+ ],
+ "type": "string"
+ },
+ "n": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "release": {
+ "type": "string"
+ },
+ "tn": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "tp": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "version": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "n",
+ "tp",
+ "fp",
+ "fn",
+ "tn",
+ "version",
+ "release",
+ "labelling"
+ ],
+ "type": "object"
+ },
+ "fired": {
+ "type": "boolean"
+ },
+ "missRate": {
+ "additionalProperties": {},
+ "properties": {
+ "hi": {
+ "type": "number"
+ },
+ "lo": {
+ "type": "number"
+ },
+ "point": {
+ "type": "number"
+ }
+ },
+ "required": [
+ "point",
+ "lo",
+ "hi"
+ ],
+ "type": "object"
+ },
+ "ppv": {
+ "additionalProperties": {},
+ "properties": {
+ "hi": {
+ "type": "number"
+ },
+ "lo": {
+ "type": "number"
+ },
+ "point": {
+ "type": "number"
+ }
+ },
+ "required": [
+ "point",
+ "lo",
+ "hi"
+ ],
+ "type": "object"
+ },
+ "prior": {
+ "additionalProperties": {},
+ "properties": {
+ "pi": {
+ "type": "number"
+ },
+ "source": {
+ "enum": [
+ "default",
+ "config",
+ "estimated"
+ ],
+ "type": "string"
+ }
+ },
+ "required": [
+ "pi",
+ "source"
+ ],
+ "type": "object"
+ }
+ },
+ "required": [
+ "basis",
+ "fired",
+ "prior",
+ "corpus"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": {},
+ "properties": {
+ "basis": {
+ "const": "definition",
+ "type": "string"
+ },
+ "conformance": {
+ "additionalProperties": {},
+ "properties": {
+ "matched": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "n": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ }
+ },
+ "required": [
+ "n",
+ "matched"
+ ],
+ "type": "object"
+ }
+ },
+ "required": [
+ "basis",
+ "conformance"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": {},
+ "properties": {
+ "basis": {
+ "const": "self_consistency",
+ "type": "string"
+ },
+ "samples": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "scoreSd": {
+ "type": "number"
+ },
+ "voteFraction": {
+ "type": "number"
+ }
+ },
+ "required": [
+ "basis",
+ "samples",
+ "voteFraction",
+ "scoreSd"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": {},
+ "properties": {
+ "basis": {
+ "const": "local_labels",
+ "type": "string"
+ },
+ "n": {
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "precision": {
+ "additionalProperties": {},
+ "properties": {
+ "hi": {
+ "type": "number"
+ },
+ "lo": {
+ "type": "number"
+ },
+ "point": {
+ "type": "number"
+ }
+ },
+ "required": [
+ "point",
+ "lo",
+ "hi"
+ ],
+ "type": "object"
+ }
+ },
+ "required": [
+ "basis",
+ "precision",
+ "n"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": {},
+ "properties": {
+ "basis": {
+ "const": "policy",
+ "type": "string"
+ }
+ },
+ "required": [
+ "basis"
+ ],
+ "type": "object"
+ },
+ {
+ "additionalProperties": {},
+ "properties": {
+ "basis": {
+ "const": "unmeasured",
+ "type": "string"
+ },
+ "why": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "basis",
+ "why"
+ ],
+ "type": "object"
+ }
+ ]
+ },
+ "value": {
+ "additionalProperties": {},
+ "properties": {
+ "stat": {
+ "type": "string"
+ },
+ "unit": {
+ "type": "string"
+ },
+ "value": {
+ "type": "number"
+ }
+ },
+ "required": [
+ "stat",
+ "unit",
+ "value"
+ ],
+ "type": "object"
+ }
+ },
+ "required": [
+ "ruleName",
+ "passed",
+ "score",
+ "message"
+ ],
+ "type": "object"
+ },
+ "type": "array"
+ },
+ "rules_evaluated": {
+ "description": "rules that judged",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "rules_skipped": {
+ "description": "rules that skipped",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "score": {
+ "description": "0..1 weighted quality over the rules that ran",
+ "type": "number"
+ },
+ "suggestions": {
+ "description": "what to change",
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "trace_id": {
+ "description": "the linked trace, when named",
+ "type": "string"
+ },
+ "verdict": {
+ "additionalProperties": {},
+ "description": "state, passed, basis (which layer decided), by (the rules), risk",
+ "properties": {
+ "basis": {
+ "enum": [
+ "policy_gate",
+ "detector_veto",
+ "critical_unknown",
+ "required_evidence_missing",
+ "risk_over_loss",
+ "score_below_threshold",
+ "clean",
+ "no_rules"
+ ],
+ "type": "string"
+ },
+ "by": {
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ },
+ "confidence": {
+ "enum": [
+ "decisive",
+ "marginal"
+ ],
+ "type": "string"
+ },
+ "passed": {
+ "type": "boolean"
+ },
+ "risk": {
+ "anyOf": [
+ {
+ "additionalProperties": {},
+ "properties": {
+ "hi": {
+ "type": "number"
+ },
+ "lo": {
+ "type": "number"
+ },
+ "pBad": {
+ "type": "number"
+ }
+ },
+ "required": [
+ "pBad",
+ "lo",
+ "hi"
+ ],
+ "type": "object"
+ },
+ {
+ "type": "null"
+ }
+ ]
+ },
+ "state": {
+ "enum": [
+ "pass",
+ "fail",
+ "unknown"
+ ],
+ "type": "string"
+ }
+ },
+ "required": [
+ "state",
+ "passed",
+ "basis",
+ "by",
+ "risk"
+ ],
+ "type": "object"
+ }
+ },
+ "required": [
+ "id",
+ "eval_type",
+ "score",
+ "passed",
+ "rule_results",
+ "suggestions",
+ "rules_evaluated",
+ "rules_skipped",
+ "insufficient_data"
+ ],
+ "type": "object"
+}evaluate_with_llm_judge3 fields changedInput schema / properties / max_cost_usd / descriptionPrevious value: -"Cost cap in USD; defaults to IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL or 0.25"New value: +"Cost cap in USD for this call; defaults to IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL or 0.25. The worst case (two attempts, full max_output_tokens) is computed before the call and refused if it exceeds the cap"Input schema / properties / model / descriptionPrevious value: -"Model ID. Supported: anthropic = claude-opus-4-7 | claude-sonnet-4-6 | claude-haiku-4-5 | claude-haiku-4-5-20251001; openai = gpt-4o | gpt-4o-mini | o1-mini."New value: +"Model ID. Supported: anthropic = claude-opus-4-7 | claude-sonnet-4-6 | claude-haiku-4-5 | claude-haiku-4-5-20251001; openai = gpt-4o | gpt-4o-mini | o1-mini. Required — cost varies a hundredfold across models"Output schema / (root)Previous value: -nullNew value: +{
+ "$schema": "http://json-schema.org/draft-07/schema#",
+ "additionalProperties": {},
+ "properties": {
+ "cost_usd": {
+ "description": "the exact spend from the pricing table",
+ "type": [
+ "number",
+ "null"
+ ]
+ },
+ "dimensions": {
+ "additionalProperties": {},
+ "description": "per-dimension sub-scores for the template",
+ "propertyNames": {
+ "type": "string"
+ },
+ "type": "object"
+ },
+ "id": {
+ "description": "the evaluation id; read it back at iris://evaluations/{id}",
+ "type": "string"
+ },
+ "input_tokens": {
+ "description": "tokens sent, across both attempts when a retry ran",
+ "type": "number"
+ },
+ "latency_ms": {
+ "description": "wall time of the provider call(s)",
+ "type": "number"
+ },
+ "model": {
+ "description": "the model that judged",
+ "type": "string"
+ },
+ "output_tokens": {
+ "description": "tokens received, across both attempts when a retry ran",
+ "type": "number"
+ },
+ "passed": {
+ "description": "the judge's own pass verdict for the template",
+ "type": "boolean"
+ },
+ "provider": {
+ "description": "the provider called",
+ "enum": [
+ "anthropic",
+ "openai"
+ ],
+ "type": "string"
+ },
+ "rationale": {
+ "description": "the judge's reasoning, in its words",
+ "type": "string"
+ },
+ "raw_response_id": {
+ "description": "the provider's response id, for your own audit",
+ "type": "string"
+ },
+ "score": {
+ "description": "0..1 from the judge",
+ "type": "number"
+ },
+ "template": {
+ "description": "the template used",
+ "type": "string"
+ },
+ "trace_id": {
+ "description": "the linked trace, when one was named",
+ "type": "string"
+ }
+ },
+ "required": [
+ "id",
+ "score",
+ "passed",
+ "rationale",
+ "dimensions",
+ "model",
+ "provider",
+ "template",
+ "input_tokens",
+ "output_tokens",
+ "cost_usd",
+ "latency_ms"
+ ],
+ "type": "object"
+}get_traces3 fields changedInput schema / properties / agent_name / descriptionPrevious value: -"Filter by agent name — exact match (no wildcards in v0.4)"New value: +"Filter by agent name — exact match (no wildcards)"Input schema / properties / limit / descriptionPrevious value: -"Results per page (default 50, max 1000 — values >1000 return 400)"New value: +"Results per page (default 50, max 1000 — values above are rejected)"Output schema / (root)Previous value: -nullNew value: +{
+ "$schema": "http://json-schema.org/draft-07/schema#",
+ "additionalProperties": {},
+ "properties": {
+ "limit": {
+ "description": "the page size applied",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "offset": {
+ "description": "the offset applied",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "summary": {
+ "additionalProperties": {},
+ "description": "the dashboard aggregates for the last hour, when include_summary was true",
+ "properties": {},
+ "type": "object"
+ },
+ "total": {
+ "description": "how many traces match the filters, across every page",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "traces": {
+ "description": "the page of traces: trace_id, agent_name, framework, input, output, tool_calls, latency_ms, token_usage, cost_usd, metadata, timestamp",
+ "items": {
+ "additionalProperties": {},
+ "properties": {
+ "trace_id": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "trace_id"
+ ],
+ "type": "object"
+ },
+ "type": "array"
+ }
+ },
+ "required": [
+ "traces",
+ "total",
+ "limit",
+ "offset"
+ ],
+ "type": "object"
+}list_rules3 fields changedInput schema / properties / enabled_only / descriptionPrevious value: -"Return only enabled rules (excludes disabled ones)"New value: +"Return only enabled custom rules (a rule disabled with delete_rule stays in the store and does not fire)"Input schema / properties / eval_type / descriptionPrevious value: -"Filter to rules of a specific eval category"New value: +"Filter the custom rules to one eval category (exact match); built_in is never filtered"Output schema / (root)Previous value: -nullNew value: +{
+ "$schema": "http://json-schema.org/draft-07/schema#",
+ "additionalProperties": {},
+ "properties": {
+ "built_in": {
+ "description": "the shipped roster, never filtered: name, category, description, weight, kind, mechanism, needs, question, classes, version, the EFFECTIVE critical flag with criticalSource, and proof (published precision, recall, intervals and ppvAt from https://iris-eval.com/proof; null where the proof is a conformance check)",
+ "items": {
+ "additionalProperties": {},
+ "properties": {
+ "name": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "name"
+ ],
+ "type": "object"
+ },
+ "type": "array"
+ },
+ "enabled_count": {
+ "description": "of those, how many are enabled",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "quarantined": {
+ "description": "entries in the store this version could not validate; they do not fire and are never deleted by a deploy",
+ "items": {},
+ "type": "array"
+ },
+ "rules": {
+ "description": "the deployed custom rules after the filters: id, name, description, evalType, severity, definition, enabled, createdAt, updatedAt, version, sourceMomentId",
+ "items": {
+ "additionalProperties": {},
+ "properties": {
+ "id": {
+ "type": "string"
+ },
+ "name": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "id",
+ "name"
+ ],
+ "type": "object"
+ },
+ "type": "array"
+ },
+ "total": {
+ "description": "custom rules after the filters",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ }
+ },
+ "required": [
+ "rules",
+ "total",
+ "enabled_count",
+ "built_in",
+ "quarantined"
+ ],
+ "type": "object"
+}log_trace3 fields changedInput schema / properties / spans / descriptionPrevious value: -"Detailed execution spans (hierarchical span tree with timings, attributes, events)"New value: +"Detailed execution spans (hierarchical span tree with timings, attributes, events); a span without start_time takes the trace timestamp"Input schema / properties / tool_calls / descriptionPrevious value: -"Tool calls made during execution (per-call latency, errors, input/output)"New value: +"Tool calls made during execution, in order, each { tool_name, input?, output?, latency_ms?, error? } — what the trajectory rules judge; evaluate_output reuses them when given this trace_id"Output schema / (root)Previous value: -nullNew value: +{
+ "$schema": "http://json-schema.org/draft-07/schema#",
+ "additionalProperties": {},
+ "properties": {
+ "status": {
+ "const": "stored",
+ "description": "always \"stored\" on success",
+ "type": "string"
+ },
+ "trace_id": {
+ "description": "the stored trace id, 32 hex — pass it to evaluate_output, get_traces or delete_trace",
+ "type": "string"
+ }
+ },
+ "required": [
+ "trace_id",
+ "status"
+ ],
+ "type": "object"
+}verify_citations4 fields changedInput schema / properties / max_citations / descriptionPrevious value: -"Max citations to verify (extras skipped); default 20"New value: +"Max citations to verify (extras skipped, not errored); default 20, at most 50"Input schema / properties / max_cost_usd_total / descriptionPrevious value: -"Cap TOTAL judge cost across all citations in this call; default $1.00"New value: +"Cap TOTAL judge cost across all citations in this call; default 1.00 USD — the pipeline stops when the next call would exceed it"Input schema / properties / model / descriptionPrevious value: -"Judge model for per-citation verification. Supported: anthropic = claude-opus-4-7 | claude-sonnet-4-6 | claude-haiku-4-5-20251001; openai = gpt-4o | gpt-4o-mini | o1-mini."New value: +"Judge model for per-citation verification. Supported: anthropic = claude-opus-4-7 | claude-sonnet-4-6 | claude-haiku-4-5 | claude-haiku-4-5-20251001; openai = gpt-4o | gpt-4o-mini | o1-mini."Output schema / (root)Previous value: -nullNew value: +{
+ "$schema": "http://json-schema.org/draft-07/schema#",
+ "additionalProperties": {},
+ "properties": {
+ "citations": {
+ "description": "per citation: the citation (raw, kind, identifier, offsets), resolve_status ok | skipped | error, resolve_error, source (url, status, content_type, bytes_fetched, truncated), judge (supported, confidence, rationale, cost_usd, latency_ms, tokens)",
+ "items": {
+ "additionalProperties": {},
+ "properties": {
+ "resolve_status": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "resolve_status"
+ ],
+ "type": "object"
+ },
+ "type": "array"
+ },
+ "id": {
+ "description": "the evaluation id; read it back at iris://evaluations/{id}",
+ "type": "string"
+ },
+ "overall_score": {
+ "description": "supported / judged; null when nothing was judged",
+ "type": [
+ "number",
+ "null"
+ ]
+ },
+ "passed": {
+ "description": "true when every judged citation was supported, or nothing was judged and nothing failed",
+ "type": "boolean"
+ },
+ "total_citations_found": {
+ "description": "citations extracted from the output",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "total_cost_usd": {
+ "description": "the spend across every judge call",
+ "type": "number"
+ },
+ "total_judged": {
+ "description": "citations the judge ruled on",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "total_resolved": {
+ "description": "citations whose source was fetched",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "total_supported": {
+ "description": "citations the judge found supported",
+ "maximum": 9007199254740991,
+ "minimum": -9007199254740991,
+ "type": "integer"
+ },
+ "trace_id": {
+ "description": "the linked trace, when one was named",
+ "type": "string"
+ }
+ },
+ "required": [
+ "id",
+ "overall_score",
+ "passed",
+ "total_citations_found",
+ "total_resolved",
+ "total_judged",
+ "total_supported",
+ "total_cost_usd",
+ "citations"
+ ],
+ "type": "object"
+}v0.8.0delete_rule3 fields changedInput schema / additionalPropertiesAdded value: +falseInput schema / properties / enabledAdded value: +{
+ "description": "When present the rule is NOT deleted: false DISABLES it (kept in the store, stops firing immediately, history and provenance preserved); true RE-ENABLES a disabled rule. Omit to delete",
+ "type": "boolean"
+}Input schema / properties / rule_id / descriptionPrevious value: -"Rule id to delete (format: rule-<hex>); obtained from list_rules or deploy_rule response"New value: +"Rule id to delete or toggle (format: rule-<hex>); obtained from list_rules or deploy_rule response"delete_trace1 field changedInput schema / additionalPropertiesAdded value: +falsedeploy_rule20 fields changedInput schema / additionalPropertiesAdded value: +falseInput schema / properties / definition / additionalPropertiesAdded value: +falseInput schema / properties / definition / descriptionPrevious value: -"Check definition (regex, length, keyword, cost, or schema)"New value: +"Check definition (regex, length, keyword, cost, or schema). Accepts exactly type, config, weight and an optional name — an unknown key is rejected"Input schema / properties / definition / properties / config / descriptionAdded value: +"Check configuration; required keys depend on type (regex_match: pattern; min_length: min_length; max_length: max_length; contains_keywords/excludes_keywords: keywords; cost_threshold: max_cost; json_schema: none)"Input schema / properties / definition / properties / name / descriptionAdded value: +"Optional and IGNORED if given — the server overwrites it with the top-level `name` so the rule reports under one name everywhere"Input schema / properties / definition / properties / name / maxLengthAdded value: +80Input schema / properties / definition / properties / name / minLengthAdded value: +1Input schema / properties / definition / properties / type / descriptionAdded value: +"Check type — decides which config keys are required"Input schema / properties / definition / properties / weight / descriptionAdded value: +"Weight in the weighted score (default 1; must be > 0)"Input schema / properties / definition / properties / weight / exclusiveMinimumAdded value: +0Input schema / properties / definition / requiredPrevious value: -[
- "name",
- "type",
- "config"
-]New value: +[
+ "type",
+ "config"
+]Input schema / properties / evalType / descriptionPrevious value: -"Eval category this rule belongs to; determines when it fires"New value: +"camelCase alias of eval_type, accepted for compatibility — prefer eval_type (snake_case is canonical across the tools)"Input schema / properties / eval_typeAdded value: +{
+ "description": "Eval category this rule belongs to; the rule fires on evaluate_output calls whose eval_type equals it (and on eval_type=\"all\"). Canonical snake_case spelling — pass exactly one of eval_type / evalType",
+ "enum": [
+ "completeness",
+ "relevance",
+ "safety",
+ "cost",
+ "custom"
+ ],
+ "type": "string"
+}Input schema / properties / name / descriptionPrevious value: -"Human-readable rule name (used in eval results)"New value: +"Human-readable rule name (1-80 chars; used in eval results). Must be unique among deployed rules unless replace=true"Input schema / properties / name / maxLengthPrevious value: -120New value: +80Input schema / properties / replaceAdded value: +{
+ "default": false,
+ "description": "When a rule with this name is already deployed: false (default) rejects the call; true deletes the existing same-named rule(s) and deploys this one in their place (fresh id; audit rows preserved)",
+ "type": "boolean"
+}Input schema / properties / severity / descriptionPrevious value: -"Severity used for dashboard sort + audit alerts"New value: +"What a FAILURE of this rule means. low/medium: informational — contributes to the weighted score only (plus dashboard sort + audit alerts). high/critical: hard-fail — a failing evaluation of this rule forces the overall passed=false regardless of the weighted score"Input schema / properties / sourceMomentId / descriptionPrevious value: -"Optional Decision Moment ID the rule was derived from (preserves workflow-inversion provenance)"New value: +"camelCase alias of source_moment_id, accepted for compatibility — prefer source_moment_id"Input schema / properties / source_moment_idAdded value: +{
+ "description": "Optional Decision Moment ID the rule was derived from (preserves workflow-inversion provenance). Canonical snake_case — pass exactly one of source_moment_id / sourceMomentId",
+ "type": "string"
+}Input schema / requiredPrevious value: -[
- "name",
- "evalType",
- "definition"
-]New value: +[
+ "name",
+ "definition"
+]evaluate_output19 fields changedInput schema / additionalPropertiesAdded value: +falseInput schema / properties / cost_usd / descriptionPrevious value: -"Cost in USD — only consulted when eval_type=\"cost\" (compared against cost_threshold rules)"New value: +"Cost in USD — consulted by the cost bundle (eval_type=\"cost\" or \"all\") AND by any cost_threshold custom rule regardless of eval_type; omit it and such a rule skips rather than passes (a critical one is listed in critical_skipped)"Input schema / properties / custom_rules / descriptionPrevious value: -"Custom evaluation rules — fires REGARDLESS of eval_type; pass eval_type=\"custom\" if you want ONLY these"New value: +"Custom evaluation rules, max 10 per call (deploy persistent rule sets via deploy_rule instead) — fires REGARDLESS of eval_type; pass eval_type=\"custom\" if you want ONLY these. Each entry accepts exactly name, type, config, weight — an unknown key (e.g. a misspelled weight) is rejected"Input schema / properties / custom_rules / items / additionalPropertiesAdded value: +falseInput schema / properties / custom_rules / items / properties / config / descriptionAdded value: +"Check configuration; keys depend on type (pattern, min_length, keywords, max_cost, …)"Input schema / properties / custom_rules / items / properties / name / descriptionAdded value: +"Rule name as it will appear in rule_results"Input schema / properties / custom_rules / items / properties / name / minLengthAdded value: +1Input schema / properties / custom_rules / items / properties / type / descriptionAdded value: +"Check type — decides which config keys the rule reads"Input schema / properties / custom_rules / items / properties / weight / descriptionAdded value: +"Weight in the weighted score (default 1; must be > 0)"Input schema / properties / custom_rules / items / properties / weight / exclusiveMinimumAdded value: +0Input schema / properties / custom_rules / maxItemsAdded value: +10Input schema / properties / eval_type / defaultRemoved value: -"completeness"Input schema / properties / eval_type / descriptionPrevious value: -"Rule bundle to apply: completeness | relevance | safety | cost | custom — picks which built-in rules fire"New value: +"Rule bundle to apply: completeness | relevance | safety | cost | custom | all — picks which built-in rules fire. \"all\" runs every bundle in one call and adds a per-category breakdown. Defaults to \"all\" when omitted — every bundle runs, safety included, and the response carries a note saying the default ran"Input schema / properties / eval_type / enumPrevious value: -[
- "completeness",
- "relevance",
- "safety",
- "cost",
- "custom"
-]New value: +[
+ "completeness",
+ "relevance",
+ "safety",
+ "cost",
+ "custom",
+ "all"
+]Input schema / properties / expected / descriptionPrevious value: -"Expected output for comparison — REQUIRED when eval_type=\"relevance\" (used as keyword-overlap target)"New value: +"Expected output for comparison — consulted only by the completeness bundle's expected_coverage rule; NOT used by relevance (the relevance rules compare the output against `input`)"Input schema / properties / input / descriptionPrevious value: -"Original input for context — improves relevance scoring (keyword overlap vs input)"New value: +"Original input for context (the ask + any source material the agent was given) — REQUIRED when eval_type=\"relevance\" (keyword_overlap and topic_consistency compare the output against it and skip without it); also grounds the safety bundle's hallucination signals"Input schema / properties / token_usage / descriptionPrevious value: -"Token usage breakdown — only consulted when eval_type=\"cost\" (used for token-budget rules)"New value: +"Token usage breakdown — only consulted by the cost bundle (eval_type=\"cost\" or \"all\"; used for token-budget rules)"Input schema / properties / tool_callsAdded value: +{
+ "description": "What the agent DID — the tool calls it made, in order, each { tool_name, input?, output?, latency_ms?, error? } exactly as log_trace records them. Read by the trajectory rules — the rules that judge what the agent DID rather than what it wrote. Omit it and those rules SKIP rather than pass — an evaluation with no trajectory data reports \"not judged\", never \"clean\". When trace_id names a stored trace and this argument is omitted, the tool_calls stored on that trace are loaded and used, so a caller who already logged them need not resend them",
+ "items": {
+ "additionalProperties": false,
+ "properties": {
+ "error": {
+ "type": "string"
+ },
+ "input": {},
+ "latency_ms": {
+ "type": "number"
+ },
+ "output": {},
+ "tool_name": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "tool_name"
+ ],
+ "type": "object"
+ },
+ "type": "array"
+}Input schema / properties / trace_id / descriptionPrevious value: -"Link evaluation to a trace — surfaces this eval in the dashboard's trace drill-through"New value: +"Link evaluation to a trace — surfaces this eval in the dashboard's trace drill-through. Must be the id of a stored trace (from log_trace / get_traces); an unknown id is rejected before anything is evaluated"evaluate_with_llm_judge2 fields changedInput schema / additionalPropertiesAdded value: +falseInput schema / properties / trace_id / descriptionPrevious value: -"Link this evaluation to a trace"New value: +"Link this evaluation to a stored trace (id from log_trace / get_traces); an unknown id is rejected BEFORE the judge is called"get_traces16 fields changedInput schema / additionalPropertiesAdded value: +falseInput schema / properties / limit / maximumAdded value: +1000Input schema / properties / limit / minimumAdded value: +1Input schema / properties / limit / typePrevious value: -"number"New value: +"integer"Input schema / properties / max_score / descriptionPrevious value: -"Maximum eval score filter (0..1) — applied to LATEST eval per trace"New value: +"Maximum eval score filter (0..1; values outside are rejected) — applied to LATEST eval per trace"Input schema / properties / max_score / maximumAdded value: +1Input schema / properties / max_score / minimumAdded value: +0Input schema / properties / min_score / descriptionPrevious value: -"Minimum eval score filter (0..1) — applied to LATEST eval per trace, not all evals"New value: +"Minimum eval score filter (0..1; values outside are rejected) — applied to LATEST eval per trace, not all evals; must be <= max_score when both are set"Input schema / properties / min_score / maximumAdded value: +1Input schema / properties / min_score / minimumAdded value: +0Input schema / properties / offset / descriptionPrevious value: -"Zero-based pagination offset — skip first N results"New value: +"Zero-based pagination offset — skip first N results (non-negative integer)"Input schema / properties / offset / maximumAdded value: +9007199254740991Input schema / properties / offset / minimumAdded value: +0Input schema / properties / offset / typePrevious value: -"number"New value: +"integer"Input schema / properties / since / descriptionPrevious value: -"ISO timestamp lower bound — return traces with timestamp >= this"New value: +"ISO 8601 timestamp (or date) lower bound — return traces with timestamp >= this; anything that is not an ISO timestamp is rejected, never treated as \"no bound\""Input schema / properties / until / descriptionPrevious value: -"ISO timestamp upper bound — return traces with timestamp < this"New value: +"ISO 8601 timestamp (or date) upper bound — return traces with timestamp <= this; must not be earlier than `since`"list_rules1 field changedInput schema / additionalPropertiesAdded value: +falselog_trace2 fields changedInput schema / additionalPropertiesAdded value: +falseInput schema / properties / tool_calls / items / additionalPropertiesAdded value: +falseverify_citations2 fields changedInput schema / additionalPropertiesAdded value: +falseInput schema / properties / trace_id / descriptionPrevious value: -"Link verification result to a trace"New value: +"Link verification result to a stored trace (id from log_trace / get_traces); an unknown id is rejected before any fetch or judge call"v0.4.6delete_rule1 field changedInput schema / additionalPropertiesRemoved value: -falsedelete_trace1 field changedInput schema / additionalPropertiesRemoved value: -falsedeploy_rule3 fields changedInput schema / additionalPropertiesRemoved value: -falseInput schema / properties / definition / additionalPropertiesRemoved value: -falseInput schema / properties / definition / properties / config / propertyNamesAdded value: +{
+ "type": "string"
+}evaluate_output4 fields changedInput schema / additionalPropertiesRemoved value: -falseInput schema / properties / custom_rules / items / additionalPropertiesRemoved value: -falseInput schema / properties / custom_rules / items / properties / config / propertyNamesAdded value: +{
+ "type": "string"
+}Input schema / properties / token_usage / additionalPropertiesRemoved value: -falseevaluate_with_llm_judge2 fields changedInput schema / additionalPropertiesRemoved value: -falseInput schema / properties / timeout_ms / maximumAdded value: +9007199254740991get_traces1 field changedInput schema / additionalPropertiesRemoved value: -falselist_rules1 field changedInput schema / additionalPropertiesRemoved value: -falselog_trace8 fields changedInput schema / additionalPropertiesRemoved value: -falseInput schema / properties / metadata / propertyNamesAdded value: +{
+ "type": "string"
+}Input schema / properties / spans / items / additionalPropertiesRemoved value: -falseInput schema / properties / spans / items / properties / attributes / propertyNamesAdded value: +{
+ "type": "string"
+}Input schema / properties / spans / items / properties / events / items / additionalPropertiesRemoved value: -falseInput schema / properties / spans / items / properties / events / items / properties / attributes / propertyNamesAdded value: +{
+ "type": "string"
+}Input schema / properties / token_usage / additionalPropertiesRemoved value: -falseInput schema / properties / tool_calls / items / additionalPropertiesRemoved value: -falseverify_citations3 fields changedInput schema / additionalPropertiesRemoved value: -falseInput schema / properties / per_source_max_bytes / maximumAdded value: +9007199254740991Input schema / properties / per_source_timeout_ms / maximumAdded value: +9007199254740991v0.1.10delete_ruledelete_tracedeploy_ruleevaluate_output8 fields changedInput schema / properties / cost_usd / descriptionPrevious value: -"Cost for cost evaluation"New value: +"Cost in USD — only consulted when eval_type=\"cost\" (compared against cost_threshold rules)"Input schema / properties / custom_rules / descriptionPrevious value: -"Custom evaluation rules"New value: +"Custom evaluation rules — fires REGARDLESS of eval_type; pass eval_type=\"custom\" if you want ONLY these"Input schema / properties / eval_type / descriptionPrevious value: -"Type of evaluation"New value: +"Rule bundle to apply: completeness | relevance | safety | cost | custom — picks which built-in rules fire"Input schema / properties / expected / descriptionPrevious value: -"Expected output for comparison"New value: +"Expected output for comparison — REQUIRED when eval_type=\"relevance\" (used as keyword-overlap target)"Input schema / properties / input / descriptionPrevious value: -"Original input for context"New value: +"Original input for context — improves relevance scoring (keyword overlap vs input)"Input schema / properties / output / descriptionPrevious value: -"The output text to evaluate"New value: +"The output text to evaluate (the agent's response that gets scored against rules)"Input schema / properties / token_usage / descriptionPrevious value: -"Token usage for cost evaluation"New value: +"Token usage breakdown — only consulted when eval_type=\"cost\" (used for token-budget rules)"Input schema / properties / trace_id / descriptionPrevious value: -"Link evaluation to a trace"New value: +"Link evaluation to a trace — surfaces this eval in the dashboard's trace drill-through"evaluate_with_llm_judgeget_traces11 fields changedInput schema / properties / agent_name / descriptionPrevious value: -"Filter by agent name"New value: +"Filter by agent name — exact match (no wildcards in v0.4)"Input schema / properties / framework / descriptionPrevious value: -"Filter by framework"New value: +"Filter by agent framework — exact match (e.g., langchain, autogen)"Input schema / properties / include_summary / descriptionPrevious value: -"Include dashboard summary stats"New value: +"Include dashboard summary stats in same response — saves a round-trip when ingesting for dashboards"Input schema / properties / limit / descriptionPrevious value: -"Results per page"New value: +"Results per page (default 50, max 1000 — values >1000 return 400)"Input schema / properties / max_score / descriptionPrevious value: -"Maximum eval score filter"New value: +"Maximum eval score filter (0..1) — applied to LATEST eval per trace"Input schema / properties / min_score / descriptionPrevious value: -"Minimum eval score filter"New value: +"Minimum eval score filter (0..1) — applied to LATEST eval per trace, not all evals"Input schema / properties / offset / descriptionPrevious value: -"Pagination offset"New value: +"Zero-based pagination offset — skip first N results"Input schema / properties / since / descriptionPrevious value: -"ISO timestamp lower bound"New value: +"ISO timestamp lower bound — return traces with timestamp >= this"Input schema / properties / sort_by / descriptionPrevious value: -"Sort field"New value: +"Sort by timestamp | latency_ms | cost_usd (default timestamp)"Input schema / properties / sort_order / descriptionPrevious value: -"Sort order"New value: +"Sort order: asc | desc (default desc — most recent / highest first)"Input schema / properties / until / descriptionPrevious value: -"ISO timestamp upper bound"New value: +"ISO timestamp upper bound — return traces with timestamp < this"list_ruleslog_trace11 fields changedInput schema / properties / agent_name / descriptionPrevious value: -"Name of the agent"New value: +"Agent name — used for filtering in get_traces (e.g., \"customer-support-bot\")"Input schema / properties / cost_usd / descriptionPrevious value: -"Total cost in USD"New value: +"Total cost in USD — overrides per-span aggregation when provided (treated as authoritative)"Input schema / properties / framework / descriptionPrevious value: -"Agent framework name"New value: +"Agent framework identifier (e.g., langchain, autogen, custom)"Input schema / properties / input / descriptionPrevious value: -"Agent input text"New value: +"Agent input text — the user prompt or upstream input that produced this output"Input schema / properties / latency_ms / descriptionPrevious value: -"Total execution time in milliseconds"New value: +"Total execution time in milliseconds (end-to-end agent latency)"Input schema / properties / metadata / descriptionPrevious value: -"Arbitrary metadata"New value: +"Opaque key-value tags (e.g. {requestId, userId, env}) — queryable in dashboard, not via get_traces filters"Input schema / properties / output / descriptionPrevious value: -"Agent output text"New value: +"Agent output text — what the agent produced (pass to evaluate_output for scoring)"Input schema / properties / spans / descriptionPrevious value: -"Detailed execution spans"New value: +"Detailed execution spans (hierarchical span tree with timings, attributes, events)"Input schema / properties / timestamp / descriptionPrevious value: -"Trace timestamp (ISO 8601)"New value: +"Trace timestamp (ISO 8601); defaults to now() when omitted"Input schema / properties / token_usage / descriptionPrevious value: -"Token usage breakdown"New value: +"Token usage breakdown (prompt/completion/total — used for cost analysis)"Input schema / properties / tool_calls / descriptionPrevious value: -"Tool calls made during execution"New value: +"Tool calls made during execution (per-call latency, errors, input/output)"verify_citationsv0.1.8evaluate_outputget_traceslog_tracev0.1.7evaluate_outputget_traceslog_traceevaluate_outputget_traceslog_traceEach tool targets a distinct resource and action: trace creation/query/deletion, deterministic evaluation, LLM judging, citation verification, and rule lifecycle management are all clearly separated. The three evaluation-related tools are well-differentiated by deterministic vs. LLM vs. citation-specific behavior.
All tool names follow a consistent snake_case verb_noun pattern such as log_trace, get_traces, deploy_rule, delete_rule, and verify_citations. Even evaluate_with_llm_judge is a readable verb-object form with a modifier, and the naming style is uniform across the set.
Nine tools is well-scoped for the domain of agent tracing, evaluation, and custom rule management. Each tool earns its place, and there is no redundancy or bloat.
The trace lifecycle is complete (log/get/delete), evaluation has deterministic, LLM, and citation-verification paths, and rules support deploy/list/delete including replacement and toggling. The main gap is the absence of a dedicated tool to retrieve or list stored evaluations; results are returned at creation time and only the latest evaluation score is exposed through get_traces.
MCP observability. Query live traffic, errors, duration, and alerts from your AI agent.
MCP server for building and testing AI agents with multi-model experimentation and insights.
Hosted MCP server for LLM cost estimation, model comparison, and budget-aware routing.
AgentGuard — 20-tool AI safety MCP: policy preflight, risk scoring, audit logging, rate limits.
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/iris-eval/mcp-server'
If you have feedback or need assistance with the MCP directory API, please join our Discord server
