Which integrations are available for this server?

Generates CI templates for Bitbucket pipelines to run Judges evaluations. Allows running code reviews on pull requests, uploading SARIF results to GitHub Code Scanning, and running a self-hosted GitHub App for PR reviews. Provides a GitHub Action to run Judges evaluations in CI pipelines, with options for diff-only, fail-on-findings, and SARIF upload. Generates CI templates for GitLab pipelines to run Judges evaluations. Installs a pre-commit hook to run Judges evaluations before commits.

How do I use Judges Panel?

1. Click on "Install Server". 2. Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state. 3. In the chat, type @ followed by the MCP server name and your instructions, e.g., "@Judges Panel Evaluate src/app.ts for all 45 dimensions" That's it! The server will respond to your query, and you can continue using it as needed. Here is a step-by-step guide with screenshots.

Judges Panel

by KevinRabun

Overview Schema Related Servers Score Discussions

TypeScript

Local

Judges Panel

An MCP (Model Context Protocol) server that provides a panel of 45 specialized judges to evaluate AI-generated code — acting as an independent quality gate regardless of which project is being reviewed. Combines deterministic pattern matching & AST analysis (instant, offline, zero LLM calls) with LLM-powered deep-review prompts that let your AI assistant perform expert-persona analysis across all 45 domains.

Highlights:

Includes an App Builder Workflow (3-step) demo for release decisions, plain-language risk summaries, and prioritized fixes — see Try the Demo.
Includes V2 context-aware evaluation with policy profiles, evidence calibration, specialty feedback, confidence scoring, and uncertainty reporting.
Includes public repository URL reporting to clone a repo, run the full tribunal, and output a consolidated markdown report.
200+ deterministic auto-fix patches (see src/patches/index.ts) plus LLM-powered deep review.

🧪 Many commands in printHelp are experimental/roadmap. By default, we show GA commands only. Set JUDGES_SHOW_EXPERIMENTAL=1 to reveal stubs; these may not be wired yet.

npm npm downloads License: MIT Tests

🔰 Packages
CLI: @kevinrabun/judges-cli → binary judges (use npx @kevinrabun/judges-cli eval --file app.ts).
MCP/API: @kevinrabun/judges → programmatic API + MCP server (npm install @kevinrabun/judges).
VS Code extension: see vscode-extension/.
GitHub Action: uses: KevinRabun/judges@main (see CI quickstart).

Quickstart

CLI (one-off)

# Using the CLI package (recommended)
npx @kevinrabun/judges-cli eval --file src/app.ts

# Show GA commands only (default)
npx @kevinrabun/judges-cli --help

# Show experimental/roadmap commands
echo "JUDGES_SHOW_EXPERIMENTAL=1" >> $GITHUB_ENV
npx @kevinrabun/judges-cli --help

# License scan (supply-chain & license compliance)
npx @kevinrabun/judges-cli license-scan --dir .

CLI vs API: If you want to embed Judges in your app (MCP/API), install @kevinrabun/judges. For the command-line, use @kevinrabun/judges-cli (binary judges).

GitHub Action

name: Judges
on: [pull_request, push]
jobs:
  judges:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: KevinRabun/judges@main
        with:
          path: .
          diff-only: true           # evaluate only changed lines in PRs (default true)
          fail-on-findings: true    # fail on critical/high findings
          upload-sarif: true        # upload SARIF to GitHub Code Scanning

Programmatic API (MCP server included)

npm install @kevinrabun/judges

import { evaluateCode } from "@kevinrabun/judges/api";
const verdict = evaluateCode("const password = 'ProdSecret';", "typescript");
console.log(verdict.overallVerdict, verdict.overallScore);

MCP server

The MCP server runs on stdio and is started by your MCP client (VS Code, Claude Desktop, etc.). Configure it in your MCP settings (e.g. mcp.json):

{
  "servers": {
    "judges": {
      "type": "stdio",
      "command": "npx",
      "args": ["-y", "@kevinrabun/judges"]
    }
  }
}

Or run the server directly:

npx @kevinrabun/judges
# Starts the MCP server on stdio

Config file: .judgesrc.json (supports ${ENV_VAR} substitution via expandEnvPlaceholders). See Configuration.

Related MCP server: CodeBase Optimizer

Why Judges?

AI code generators (Copilot, Cursor, Claude, ChatGPT, etc.) write code fast — but they routinely produce insecure defaults, missing auth, hardcoded secrets, and poor error handling. Human reviewers catch some of this, but nobody reviews 45 dimensions consistently.

	ESLint / Biome	SonarQube	Semgrep / CodeQL	Judges
Scope	Style + some bugs	Bugs + code smells	Security patterns	45 domains: security, cost, compliance, a11y, API design, cloud, UX, …
AI-generated code focus	No	No	Partial	Purpose-built for AI output failure modes
Setup	Config per project	Server + scanner	Cloud or local	One command: `npx @kevinrabun/judges-cli eval file.ts`
Auto-fix patches	Some	No	No	200+ deterministic patches — instant, offline
Non-technical output	No	Dashboard	No	Plain-language findings with What/Why/Next
MCP native	No	No	No	Yes — works inside Copilot, Claude, Cursor
SARIF output	No	Yes	Yes	Yes — upload to GitHub Code Scanning
Cost	Free	$$$$	Free/paid	Free / MIT

Judges doesn't replace linters — it covers the dimensions linters don't: authentication strategy, data sovereignty, cost patterns, accessibility, framework-specific anti-patterns, and architectural issues across multiple files.

Quick Start

Prereqs: Node.js >=18 (>=20 recommended), npx available. The judges CLI binary ships with @kevinrabun/judges-cli (preferred) and also works via npx @kevinrabun/judges.
Packages:
CLI: npm install -g @kevinrabun/judges-cli (or npx @kevinrabun/judges-cli ...)
MCP/API: npm install @kevinrabun/judges

Use @kevinrabun/judges for the MCP server and programmatic API. Use @kevinrabun/judges-cli when you want the judges terminal command.

Try it now (no clone needed)

# Install the CLI globally
npm install -g @kevinrabun/judges-cli

# Evaluate any file
judges eval src/app.ts

# Pipe from stdin
cat api.py | judges eval --language python

# Single judge
judges eval --judge cybersecurity server.ts

# SARIF output for CI
judges eval --file app.ts --format sarif > results.sarif

# HTML report with severity filters and dark/light theme
judges eval --file app.ts --format html > report.html

# Fail CI on findings (exit code 1)
judges eval --fail-on-findings src/api.ts

# Suppress known findings via baseline
judges eval --baseline baseline.json src/api.ts

# Use a named preset
judges eval --preset security-only src/api.ts

# Use a config file
judges eval --config .judgesrc.json src/api.ts

# Set a minimum score threshold (exit 1 if below)
judges eval --min-score 80 src/api.ts

# One-line summary for scripts
judges eval --summary src/api.ts

# Agentic skills (orchestrated judge sets)
judges skill ai-code-review --file src/app.ts
judges skill security-review --file src/api.ts --format json
judges skill release-gate --file src/app.ts
judges skills   # list available skills

> Full catalog: [`docs/skills.md`](docs/skills.md)


# List all 45 judges
judges list

Additional CLI Commands

# Interactive project setup wizard
judges init

# Preview auto-fix patches (dry run)
judges fix src/app.ts

# Apply patches directly
judges fix src/app.ts --apply

# License compliance scan (copyleft/unknown detection)
judges license-scan --format json --risk high

# Watch mode — re-evaluate on file save
judges watch src/

# Project-level report (local directory)
judges report . --format html --output report.html

# Evaluate a unified diff (pipe from git diff)
git diff HEAD~1 | judges diff

# Analyze dependencies for supply-chain risks
judges deps --path . --format json

# Run GitHub App server (zero-config PR reviews)
judges app serve --port 4567

# Run GitHub PR review (gh CLI required)
judges review --pr 123 --repo owner/name --diff-only

# Auto-tune presets and configs
judges tune --dir . --apply

# Create a baseline file to suppress known findings
judges baseline create --file src/api.ts -o baseline.json

# Generate CI template files
judges ci-templates --provider github
judges ci-templates --provider gitlab
judges ci-templates --provider azure
judges ci-templates --provider bitbucket

# Generate per-judge rule documentation
judges docs
judges docs --judge cybersecurity
judges docs --output docs/

# Install shell completions
judges completions bash   # eval "$(judges completions bash)"
judges completions zsh
judges completions fish
judges completions powershell

# Install pre-commit hook
judges hook install

# Uninstall pre-commit hook
judges hook uninstall

🔎 Tip: The CLI help now defaults to GA commands only. To see experimental/roadmap commands, run:
JUDGES_SHOW_EXPERIMENTAL=1 judges --help

GitHub App (self-hosted webhook)

Run a zero-config PR reviewer as a GitHub App:

# Run the webhook server locally
judges app serve --port 4567

Required env vars:

JUDGES_APP_ID – GitHub App ID
JUDGES_PRIVATE_KEY or JUDGES_PRIVATE_KEY_PATH – PEM private key
JUDGES_WEBHOOK_SECRET – signature verification secret

Optional:

JUDGES_MIN_SEVERITY (default: medium)
JUDGES_MAX_COMMENTS (default: 25)
JUDGES_TEST_DRY_RUN=1 to avoid live network calls during tests

For local testing, you can expose http://localhost:4567/webhook via ngrok http 4567 and configure the GitHub App webhook URL accordingly.

Use in GitHub Actions

Add Judges to your CI pipeline with zero configuration:

# .github/workflows/judges.yml
name: Judges Code Review
on: [pull_request]

jobs:
  judges:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      security-events: write  # only if using upload-sarif
    steps:
      - uses: actions/checkout@v4
      - uses: KevinRabun/judges@main
        with:
          path: src/api.ts        # file or directory
          format: text             # text | json | sarif | markdown
          upload-sarif: true       # upload to GitHub Code Scanning
          fail-on-findings: true   # fail CI on critical/high findings

Outputs available for downstream steps: verdict, score, findings, critical, high, sarif-file.

Use with Docker (no Node.js required)

# Build the image
docker build -t judges .

# Evaluate a local file
docker run --rm -v $(pwd):/code judges eval --file /code/app.ts

# Pipe from stdin
cat api.py | docker run --rm -i judges eval --language python

# List judges
docker run --rm judges list

Or use as an MCP server

1. Install and Build

git clone https://github.com/KevinRabun/judges.git
cd judges
npm install
npm run build

2. Try the Demo

Run the included demo to see all 45 judges evaluate a purposely flawed API server:

npm run demo

This evaluates examples/sample-vulnerable-api.ts — a file intentionally packed with security holes, performance anti-patterns, and code quality issues — and prints a full verdict with per-judge scores and findings.

The demo now also includes an App Builder Workflow (3-step) section. In a single run, you get both tribunal output and workflow output:

Release decision (Ship now / Ship with caution / Do not ship)
Plain-language summaries of top risks
Prioritized remediation tasks and AI-fixable P0/P1 items

Sample workflow output (truncated):

╔══════════════════════════════════════════════════════════════╗
║             App Builder Workflow Demo (3-Step)             ║
╚══════════════════════════════════════════════════════════════╝

  Decision       : Do not ship
  Verdict        : FAIL (47/100)
  Risk Counts    : Critical 24 | High 27 | Medium 55

  Step 2 — Plain-Language Findings:
  - [CRITICAL] DATA-001: Hardcoded password detected
      What: ...
      Why : ...
      Next: ...

  Step 3 — Prioritized Tasks:
  - P0 | DEVELOPER | Effort L | DATA-001
      Task: ...
      Done: ...

  AI-Fixable Now (P0/P1):
  - P0 DATA-001: ...

Sample tribunal output (truncated):

╔══════════════════════════════════════════════════════════════╗
║           Judges Panel — Full Tribunal Demo                 ║
╚══════════════════════════════════════════════════════════════╝

  Overall Verdict : FAIL
  Overall Score   : 43/100
  Critical Issues : 15
  High Issues     : 17
  Total Findings  : 83
  Judges Run      : 33

  Per-Judge Breakdown:
  ────────────────────────────────────────────────────────────────
  ❌ Judge Data Security              0/100    7 finding(s)
  ❌ Judge Cybersecurity              0/100    7 finding(s)
  ❌ Judge Cost Effectiveness        52/100    5 finding(s)
  ⚠️  Judge Scalability              65/100    4 finding(s)
  ❌ Judge Cloud Readiness           61/100    4 finding(s)
  ❌ Judge Software Practices        45/100    6 finding(s)
  ❌ Judge Accessibility              0/100    8 finding(s)
  ❌ Judge API Design                 0/100    9 finding(s)
  ❌ Judge Reliability               54/100    3 finding(s)
  ❌ Judge Observability             45/100    5 finding(s)
  ❌ Judge Performance               27/100    5 finding(s)
  ❌ Judge Compliance                 0/100    4 finding(s)
  ⚠️  Judge Testing                  90/100    1 finding(s)
  ⚠️  Judge Documentation            70/100    4 finding(s)
  ⚠️  Judge Internationalization     65/100    4 finding(s)
  ⚠️  Judge Dependency Health        90/100    1 finding(s)
  ❌ Judge Concurrency               44/100    4 finding(s)
  ❌ Judge Ethics & Bias             65/100    2 finding(s)
  ❌ Judge Maintainability           52/100    4 finding(s)
  ❌ Judge Error Handling            27/100    3 finding(s)
  ❌ Judge Authentication             0/100    4 finding(s)
  ❌ Judge Database                   0/100    5 finding(s)
  ❌ Judge Caching                   62/100    3 finding(s)
  ❌ Judge Configuration Mgmt         0/100    3 finding(s)
  ⚠️  Judge Backwards Compat         80/100    2 finding(s)
  ⚠️  Judge Portability              72/100    2 finding(s)
  ❌ Judge UX                        52/100    4 finding(s)
  ❌ Judge Logging Privacy            0/100    4 finding(s)
  ❌ Judge Rate Limiting             27/100    4 finding(s)
  ⚠️  Judge CI/CD                    80/100    2 finding(s)

3. Run the Tests

npm test

Runs automated tests covering all judges, AST parsers, markdown formatters, and edge cases.

4. Connect to Your Editor

VS Code (recommended — zero config)

Install the Judges Panel extension from the Marketplace. It provides:

Inline diagnostics & quick-fixes on every file save
@judges chat participant — type @judges in Copilot Chat, or just ask for a "judges panel review" and Copilot routes automatically
Auto-configured MCP server — all 45 expert-persona prompts available to Copilot with zero setup

code --install-extension kevinrabun.judges-panel

VS Code — manual MCP config

If you prefer explicit workspace config (or want teammates without the extension to benefit), create .vscode/mcp.json:

{
  "servers": {
    "judges": {
      "command": "npx",
      "args": ["-y", "@kevinrabun/judges"]
    }
  }
}

Claude Desktop

Add to claude_desktop_config.json:

{
  "mcpServers": {
    "judges": {
      "command": "npx",
      "args": ["-y", "@kevinrabun/judges"]
    }
  }
}

Cursor / other MCP clients

Use the same npx command for any MCP-compatible client:

{
  "command": "npx",
  "args": ["-y", "@kevinrabun/judges"]
}

5. Use Judges in GitHub Copilot PR Reviews

Yes — users can include Judges as part of GitHub-based review workflows, with one important caveat:

The hosted copilot-pull-request-reviewer on GitHub does not currently let you directly attach arbitrary local MCP servers the same way VS Code does.
The practical pattern is to run Judges in CI on each PR, publish a report/check, and have Copilot + human reviewers use that output during review.

Option A (recommended): PR workflow check + report artifact

Create .github/workflows/judges-pr-review.yml:

name: Judges PR Review

on:
  pull_request:
    types: [opened, synchronize, reopened]

jobs:
  judges:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Setup Node
        uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm

      - name: Install
        run: npm ci

      - name: Generate Judges report
        run: |
          npx tsx -e "import { generateRepoReportFromLocalPath } from './src/reports/public-repo-report.ts';
          const result = generateRepoReportFromLocalPath({
            repoPath: process.cwd(),
            outputPath: 'judges-pr-report.md',
            maxFiles: 600,
            maxFindingsInReport: 150,
          });
          console.log('Overall:', result.overallVerdict, result.averageScore);"

      - name: Upload report artifact
        uses: actions/upload-artifact@v4
        with:
          name: judges-pr-report
          path: judges-pr-report.md

This gives every PR a reproducible Judges output your team (and Copilot) can reference.

Option B: Add Copilot custom instructions in-repo

Add .github/instructions/judges.instructions.md with guidance such as:

When reviewing pull requests:
1. Read the latest Judges report artifact/check output first.
2. Prioritize CRITICAL and HIGH findings in remediation guidance.
3. If findings conflict, defer to security/compliance-related Judges.
4. Include rule IDs (e.g., DATA-001, CYBER-004) in suggested fixes.

This helps keep Copilot feedback aligned with Judges findings.

CLI Reference

All commands support --help for usage details.

`judges eval`

Evaluate a file with all 45 judges or a single judge.

Flag	Description
`--file <path>` / positional	File to evaluate
`--judge <id>` / `-j <id>`	Single judge mode
`--language <lang>` / `-l <lang>`	Language hint (auto-detected from extension)
`--format <fmt>` / `-f <fmt>`	Output format: `text`, `json`, `sarif`, `markdown`, `html`, `pdf`, `junit`, `codeclimate`, `github-actions`
`--output <path>` / `-o <path>`	Write output to file
`--fail-on-findings`	Exit with code 1 if verdict is FAIL
`--baseline <path>` / `-b <path>`	JSON baseline file — suppress known findings
`--summary`	Print a single summary line (ideal for scripts)
`--config <path>`	Load a `.judgesrc` / `.judgesrc.json` config file
`--preset <name>`	Use a named preset (see Named Presets for all 22 options)
`--min-score <n>`	Exit with code 1 if overall score is below this threshold
`--verbose`	Print timing and debug information
`--quiet`	Suppress non-essential output
`--no-color`	Disable ANSI colors

`judges init`

Interactive wizard that generates project configuration:

.judgesrc.json — rule customization, disabled judges, severity thresholds
.github/workflows/judges.yml — GitHub Actions CI workflow
.gitlab-ci.judges.yml — GitLab CI pipeline (optional)
azure-pipelines.judges.yml — Azure Pipelines (optional)

`judges fix`

Preview or apply auto-fix patches from deterministic findings.

Flag	Description
positional	File to fix
`--apply`	Write patches to disk (default: dry run)
`--judge <id>`	Limit to a single judge's findings

`judges watch`

Continuously re-evaluate files on save.

Flag	Description
positional	File or directory to watch (default: `.`)
`--judge <id>`	Single judge mode
`--fail-on-findings`	Exit non-zero if any evaluation fails

`judges report`

Run a full project-level tribunal on a local directory.

Flag	Description
positional	Directory path (default: `.`)
`--format <fmt>`	Output format: `text`, `json`, `html`, `markdown`
`--output <path>`	Write report to file
`--max-files <n>`	Maximum files to analyze (default: 600)
`--max-file-bytes <n>`	Skip files larger than this (default: 300000)

`judges hook`

Manage a Git pre-commit hook that runs Judges on staged files.

judges hook install    # add pre-commit hook
judges hook uninstall  # remove pre-commit hook

Detects Husky (.husky/pre-commit) and falls back to .git/hooks/pre-commit. Uses marker-based injection so it won't clobber existing hooks.

`judges diff`

Evaluate only the changed lines from a unified diff (e.g., git diff output).

Flag	Description
`--file <path>`	Read diff from file instead of stdin
`--format <fmt>`	Output format: `text`, `json`, `sarif`, `junit`, `codeclimate`
`--output <path>`	Write output to file

git diff HEAD~1 | judges diff
judges diff --file changes.patch --format sarif

`judges deps`

Analyze project dependencies for supply-chain risks.

Flag	Description
`--path <dir>`	Project root to scan (default: `.`)
`--format <fmt>`	Output format: `text`, `json`

judges deps --path .
judges deps --path ./backend --format json

`judges baseline`

Create a baseline file to suppress known findings in future evaluations.

judges baseline create --file src/api.ts
judges baseline create --file src/api.ts -o .judges-baseline.json

`judges ci-templates`

Generate CI/CD configuration templates for popular providers.

judges ci-templates --provider github   # .github/workflows/judges.yml
judges ci-templates --provider gitlab   # .gitlab-ci.judges.yml
judges ci-templates --provider azure    # azure-pipelines.judges.yml
judges ci-templates --provider bitbucket # bitbucket-pipelines.yml (snippet)

`judges docs`

Generate per-judge rule documentation in Markdown.

Flag	Description
`--judge <id>`	Generate docs for a single judge
`--output <dir>`	Write individual `.md` files per judge

judges docs                          # all judges to stdout
judges docs --judge cybersecurity    # single judge
judges docs --output docs/judges/    # write files to directory

`judges completions`

Generate shell completion scripts.

eval "$(judges completions bash)"        # Bash
eval "$(judges completions zsh)"         # Zsh
judges completions fish | source         # Fish
judges completions powershell            # PowerShell (Register-ArgumentCompleter)

Named Presets

Use --preset to apply pre-configured evaluation settings:

Preset	Description
`strict`	All severities, all judges — maximum thoroughness
`lenient`	Only high and critical findings — fast and focused
`security-only`	Security-focused — disables non-security judges (cost, scalability, docs, a11y, i18n, UX, etc.)
`startup`	Skip compliance, sovereignty, i18n judges — move fast
`compliance`	Only compliance, data-sovereignty, authentication — regulatory focus
`performance`	Only performance, scalability, caching, cost-effectiveness
`react`	Tuned for React/Next.js apps — enables accessibility, XSS protection
`express`	Tuned for Express.js APIs — middleware security, auth, CORS, rate limiting
`fastapi`	Tuned for Python FastAPI — input validation, async patterns, API security
`django`	Tuned for Django apps — template security, ORM misuse, CSRF
`spring-boot`	Tuned for Java Spring Boot — injection, configuration, actuator security
`rails`	Tuned for Ruby on Rails — mass assignment, CSRF, SQL injection
`nextjs`	Tuned for Next.js — server/client security, API routes, SSR/ISR
`terraform`	Tuned for Terraform/OpenTofu IaC — infrastructure security, compliance
`kubernetes`	Tuned for K8s manifests — security contexts, RBAC, resource limits
`onboarding`	Smart defaults for first-time adoption — suppresses noisy rules
`fintech`	Financial services — PCI DSS, cryptography, authentication, audit
`healthtech`	Healthcare — HIPAA compliance, data sovereignty, encryption, audit trails
`saas`	Multi-tenant SaaS — tenant isolation, rate limiting, scalability
`government`	Government/public sector — compliance, sovereignty, authentication
`open-source`	Open-source projects — documentation, backwards compatibility, security, dependency health
`ai-review`	AI-generated code review — hallucination detection, security, authentication, correctness

judges eval --preset security-only src/api.ts
judges eval --preset strict --format sarif src/app.ts > results.sarif

CI Output Formats

JUnit XML

Generate JUnit XML for Jenkins, Azure DevOps, GitHub Actions, or GitLab test result viewers:

judges eval --format junit src/api.ts > results.xml

Each judge maps to a <testsuite>, each finding becomes a <testcase> with <failure> for critical/high severity.

CodeClimate / GitLab Code Quality

Generate CodeClimate JSON for GitLab Code Quality or similar tools:

judges eval --format codeclimate src/api.ts > codequality.json

Score Badges

Generate SVG or text badges for your README:

import { generateBadgeSvg, generateBadgeText } from "@kevinrabun/judges/badge";

const svg = generateBadgeSvg(85);          // shields.io-style SVG
const text = generateBadgeText(85);        // "✓ judges 85/100"
const svg2 = generateBadgeSvg(75, "quality"); // custom label

The Judge Panel

Judge	Domain	Rule Prefix	What It Evaluates
Data Security	Data Security & Privacy	`DATA-`	Encryption, PII handling, secrets management, access controls
Cybersecurity	Cybersecurity & Threat Defense	`CYBER-`	Injection attacks, XSS, CSRF, auth flaws, OWASP Top 10
Cost Effectiveness	Cost Optimization & Resource Efficiency	`COST-`	Algorithm efficiency, N+1 queries, memory waste, caching strategy
Scalability	Scalability & Performance	`SCALE-`	Statelessness, horizontal scaling, concurrency, bottlenecks
Cloud Readiness	Cloud-Native Architecture & DevOps	`CLOUD-`	12-Factor compliance, containerization, graceful shutdown, IaC
Software Practices	Software Engineering Best Practices & Secure SDLC	`SWDEV-`	SOLID principles, type safety, error handling, input validation
Accessibility	Accessibility (a11y)	`A11Y-`	WCAG compliance, screen reader support, keyboard navigation, ARIA
API Design	API Design & Contracts	`API-`	REST conventions, versioning, pagination, error responses
Reliability	Reliability & Resilience	`REL-`	Error handling, timeouts, retries, circuit breakers
Observability	Monitoring & Diagnostics	`OBS-`	Structured logging, health checks, metrics, tracing
Performance	Runtime Performance	`PERF-`	N+1 queries, sync I/O, caching, memory leaks
Compliance	Regulatory & License Compliance	`COMP-`	GDPR/CCPA, PII protection, consent, data retention, audit trails
Data Sovereignty	Data, Technological & Operational Sovereignty	`SOV-`	Data residency, cross-border transfers, vendor key management, AI model portability, identity federation, circuit breakers, audit trails, data export
Testing	Test Quality & Coverage	`TEST-`	Test coverage, assertions, test isolation, naming
Documentation	Documentation & Developer Experience	`DOC-`	JSDoc/docstrings, magic numbers, TODOs, code comments
Internationalization	i18n & Localization	`I18N-`	Hardcoded strings, locale handling, currency formatting
Dependency Health	Supply Chain & Dependencies	`DEPS-`	Version pinning, deprecated packages, supply chain
Concurrency	Concurrency & Thread Safety	`CONC-`	Race conditions, unbounded parallelism, missing await
Ethics & Bias	AI/ML Fairness & Ethics	`ETHICS-`	Demographic logic, dark patterns, inclusive language
Maintainability	Code Maintainability & Technical Debt	`MAINT-`	Any types, magic numbers, deep nesting, dead code, file length
Error Handling	Error Handling & Fault Tolerance	`ERR-`	Empty catch blocks, missing error handlers, swallowed errors
Authentication	Authentication & Authorization	`AUTH-`	Hardcoded creds, missing auth middleware, token in query params
Database	Database Design & Query Efficiency	`DB-`	SQL injection, N+1 queries, connection pooling, transactions
Caching	Caching Strategy & Data Freshness	`CACHE-`	Unbounded caches, missing TTL, no HTTP cache headers
Configuration Management	Configuration & Secrets Management	`CFG-`	Hardcoded secrets, missing env vars, config validation
Backwards Compatibility	Backwards Compatibility & Versioning	`COMPAT-`	API versioning, breaking changes, response consistency
Portability	Platform Portability & Vendor Independence	`PORTA-`	OS-specific paths, vendor lock-in, hardcoded hosts
UX	User Experience & Interface Quality	`UX-`	Loading states, error messages, pagination, destructive actions
Logging Privacy	Logging Privacy & Data Redaction	`LOGPRIV-`	PII in logs, token logging, structured logging, redaction
Rate Limiting	Rate Limiting & Throttling	`RATE-`	Missing rate limits, unbounded queries, backoff strategy
CI/CD	CI/CD Pipeline & Deployment Safety	`CICD-`	Test infrastructure, lint config, Docker tags, build scripts
Code Structure	Structural Analysis	`STRUCT-`	Cyclomatic complexity, nesting depth, function length, dead code, type safety
Agent Instructions	Agent Instruction Markdown Quality & Safety	`AGENT-`	Instruction hierarchy, conflict detection, unsafe overrides, scope, validation, policy guidance
AI Code Safety	AI-Generated Code Quality & Security	`AICS-`	Prompt injection, insecure LLM output handling, debug defaults, missing validation, unsafe deserialization of AI responses
Framework Safety	Framework-Specific Security & Best Practices	`FW-`	React hooks ordering, Express middleware chains, Next.js SSR/SSG pitfalls, Angular/Vue lifecycle patterns, Django/Flask/FastAPI safety, Spring Boot security, ASP.NET Core auth & CORS, Go Gin/Echo/Fiber patterns
IaC Security	Infrastructure as Code	`IAC-`	Terraform, Bicep, ARM template misconfigurations, hardcoded secrets, missing encryption, overly permissive network/IAM rules
Security	General Security Posture	`SEC-`	Holistic security assessment — insecure data flows, weak cryptography, unsafe deserialization
Hallucination Detection	AI-Hallucinated API & Import Validation	`HALLU-`	Detects hallucinated APIs, fabricated imports, and non-existent modules from AI code generators
Intent Alignment	Code–Comment Alignment & Stub Detection	`INTENT-`	Detects mismatches between stated intent and implementation, placeholder stubs, TODO-only functions
API Contract Conformance	API Design & REST Best Practices	`API-`	API endpoint input validation, REST conformance, request/response contract consistency
Multi-Turn Coherence	Code Coherence & Consistency	`COH-`	Self-contradicting patterns, duplicate definitions, dead code, inconsistent naming
Model Fingerprint Detection	AI Code Provenance & Model Attribution	`MFPR-`	Detects stylistic fingerprints characteristic of specific AI code generators
Over-Engineering	Simplicity & Pragmatism	`OVER-`	Unnecessary abstractions, wrapper-mania, premature generalization, over-complex patterns
Logic Review	Semantic Correctness & Logic Integrity	`LOGIC-`	Inverted conditions, dead code, name-body mismatch, off-by-one, incomplete control flow
False-Positive Review	False Positive Detection & Finding Accuracy	`FPR-`	Meta-judge reviewing pattern-based findings for false positives: string literal context, comment/docstring matches, test scaffolding, IaC template gating

How It Works

The tribunal operates in three layers:

Pattern-Based Analysis — All tools (evaluate_code, evaluate_code_single_judge, evaluate_project, evaluate_diff) perform heuristic analysis using regex pattern matching to catch common anti-patterns. This layer is instant, deterministic, and runs entirely offline with zero external API calls.
AST-Based Structural Analysis — The Code Structure judge (STRUCT-* rules) uses real Abstract Syntax Tree parsing to measure cyclomatic complexity, nesting depth, function length, parameter count, dead code, and type safety with precision that regex cannot achieve. All supported languages — TypeScript, JavaScript, Python, Rust, Go, Java, C#, and C++ — are parsed via tree-sitter WASM grammars (real syntax trees compiled to WebAssembly, in-process, zero native dependencies). A scope-tracking structural parser is kept as a fallback when WASM grammars are unavailable. No external AST server required.
LLM-Powered Deep Analysis (Prompts) — The server exposes MCP prompts (e.g., judge-data-security, judge-cybersecurity) that provide each judge's expert persona as a system prompt. When used by an LLM-based client (Copilot, Claude, Cursor, etc.), the host LLM performs deeper, context-aware probabilistic analysis beyond what static patterns can detect. This is where the systemPrompt on each judge comes alive — Judges itself makes no LLM calls, but it provides the expert criteria so your AI assistant can act as 45 specialized reviewers.

Composable by Design

Judges Panel is a dual-layer review system: instant deterministic tools (offline, no API keys) for pattern and AST analysis, plus 45 expert-persona MCP prompts that unlock LLM-powered deep analysis when connected to an AI client. It does not try to be a CVE scanner or a linter. Those capabilities belong in dedicated MCP servers that an AI agent can orchestrate alongside Judges.

Built-in AST Analysis

Unlike earlier versions that recommended a separate AST MCP server, Judges Panel now includes real AST-based structural analysis out of the box:

TypeScript, JavaScript, Python, Rust, Go, Java, C#, C++ — All parsed with a unified tree-sitter WASM engine for full syntax-tree analysis (functions, complexity, nesting, dead code, type safety). Falls back to a scope-tracking structural parser when WASM grammars are unavailable

The Code Structure judge (STRUCT-*) uses these parsers to accurately measure:

Rule	Metric	Threshold
`STRUCT-001`	Cyclomatic complexity	> 10 per function (high)
`STRUCT-002`	Nesting depth	> 4 levels (medium)
`STRUCT-003`	Function length	> 50 lines (medium)
`STRUCT-004`	Parameter count	> 5 parameters (medium)
`STRUCT-005`	Dead code	Unreachable statements (low)
`STRUCT-006`	Weak types	`any`, `dynamic`, `Object`, `interface{}`, `unsafe` (medium)
`STRUCT-007`	File complexity	> 40 total cyclomatic complexity (high)
`STRUCT-008`	Extreme complexity	> 20 per function (critical)
`STRUCT-009`	Extreme parameters	> 8 parameters (high)
`STRUCT-010`	Extreme function length	> 150 lines (high)

Recommended MCP Stack

When your AI coding assistant connects to multiple MCP servers, each one contributes its specialty:

┌─────────────────────────────────────────────────────────┐
│                   AI Coding Assistant                   │
│              (Claude, Copilot, Cursor, etc.)            │
└──────┬──────────────────┬──────────┬───────────────────┘
       │                  │          │
       ▼                  ▼          ▼
  ┌──────────────┐  ┌────────┐  ┌────────┐
  │   Judges     │  │  CVE / │  │ Linter │
  │   Panel      │  │  SBOM  │  │ Server │
  │ ─────────────│  └────────┘  └────────┘
  │ 44 Heuristic │   Vuln DB     Style &
  │   judges     │   scanning    correctness
  │ + AST judge  │
  └──────────────┘
   Patterns +
   structural
   analysis

Layer	What It Does	Example Servers
Judges Panel	45-judge quality gate — security patterns, AST analysis, cost, scalability, a11y, compliance, sovereignty, ethics, dependency health, agent instruction governance, AI code safety, framework safety	This server
CVE / SBOM	Vulnerability scanning against live databases — known CVEs, license risks, supply chain	OSV, Snyk, Trivy, Grype MCP servers
Linting	Language-specific style and correctness rules	ESLint, Ruff, Clippy MCP servers
Runtime Profiling	Memory, CPU, latency measurement on running code	Custom profiling MCP servers

What This Means in Practice

When you ask your AI assistant "Is this code production-ready?", the agent can:

Judges Panel → Scan for hardcoded secrets, missing error handling, N+1 queries, accessibility gaps, compliance issues, plus analyze cyclomatic complexity, detect dead code, and flag deeply nested functions via AST
CVE Server → Check every dependency in package.json against known vulnerabilities
Linter Server → Enforce team style rules, catch language-specific gotchas

Each server returns structured findings. The AI synthesizes everything into a single, actionable review — no single server needs to do it all.

MCP Tools

`evaluate_v2`

Run a V2 context-aware tribunal evaluation designed to raise feedback quality toward lead engineer/architect-level review:

Policy profile calibration (default, startup, regulated, healthcare, fintech, public-sector)
Context ingestion (architecture notes, constraints, standards, known risks, data-boundary model)
Runtime evidence hooks (tests, coverage, latency, error rate, vulnerability counts)
Specialty feedback aggregation by judge/domain
Confidence scoring and explicit uncertainty reporting

Supports:

Code mode: code + language
Project mode: files[]

Parameter	Type	Required	Description
`code`	string	conditional	Source code for single-file mode
`language`	string	conditional	Programming language for single-file mode
`files`	array	conditional	`{ path, content, language }[]` for project mode
`context`	string	no	High-level review context
`includeAstFindings`	boolean	no	Include AST/code-structure findings (default: true)
`minConfidence`	number	no	Minimum finding confidence to include (0-1, default: 0)
`policyProfile`	enum	no	`default`, `startup`, `regulated`, `healthcare`, `fintech`, `public-sector`
`evaluationContext`	object	no	Structured architecture/constraint context
`evidence`	object	no	Runtime/operational evidence for confidence calibration

`evaluate_app_builder_flow`

Run a 3-step app-builder workflow for technical and non-technical stakeholders:

Tribunal review (code/project/diff)
Plain-language translation of top risks
Prioritized remediation tasks with AI-fixable P0/P1 extraction

Supports:

Code mode: code + language
Project mode: files[]
Diff mode: code + language + changedLines[]

Parameter	Type	Required	Description
`code`	string	conditional	Full source content (code/diff mode)
`language`	string	conditional	Programming language (code/diff mode)
`files`	array	conditional	`{ path, content, language }[]` for project mode
`changedLines`	number[]	no	1-based changed lines for diff mode
`context`	string	no	Optional business/technical context
`maxFindings`	number	no	Max translated top findings (default: 10)
`maxTasks`	number	no	Max generated tasks (default: 20)
`includeAstFindings`	boolean	no	Include AST/code-structure findings (default: true)
`minConfidence`	number	no	Minimum finding confidence to include (0-1, default: 0)

`evaluate_public_repo_report`

Clone a public repository URL, run the full judges panel across eligible source files, and generate a consolidated markdown report.

Parameter	Type	Required	Description
`repoUrl`	string	yes	Public repository URL (`https://...`)
`branch`	string	no	Optional branch name
`outputPath`	string	no	Optional path to write report markdown
`maxFiles`	number	no	Max files analyzed (default: 600)
`maxFileBytes`	number	no	Max file size in bytes (default: 300000)
`maxFindingsInReport`	number	no	Max detailed findings in output (default: 150)
`credentialMode`	string	no	Credential detection mode: `standard` (default) or `strict`
`includeAstFindings`	boolean	no	Include AST/code-structure findings (default: true)
`minConfidence`	number	no	Minimum finding confidence to include (0-1, default: 0)
`enableMustFixGate`	boolean	no	Enable must-fix gate summary for high-confidence dangerous findings (default: false)
`mustFixMinConfidence`	number	no	Confidence threshold for must-fix gate triggers (0-1, default: 0.85)
`mustFixDangerousRulePrefixes`	string[]	no	Optional dangerous rule prefixes for gate matching (e.g., `AUTH`, `CYBER`, `DATA`)
`keepClone`	boolean	no	Keep cloned repo on disk for inspection

Quick examples

Generate a report from CLI:

npm run report:public-repo -- --repoUrl https://github.com/microsoft/vscode --output reports/vscode-judges-report.md

# stricter credential-signal mode (optional)
npm run report:public-repo -- --repoUrl https://github.com/openclaw/openclaw --credentialMode strict --output reports/openclaw-judges-report-strict.md

# judge findings only (exclude AST/code-structure findings)
npm run report:public-repo -- --repoUrl https://github.com/openclaw/openclaw --includeAstFindings false --output reports/openclaw-judges-report-no-ast.md

# show only findings at 80%+ confidence
npm run report:public-repo -- --repoUrl https://github.com/openclaw/openclaw --minConfidence 0.8 --output reports/openclaw-judges-report-high-confidence.md

# include must-fix gate summary in the generated report
npm run report:public-repo -- --repoUrl https://github.com/openclaw/openclaw --enableMustFixGate true --mustFixMinConfidence 0.9 --mustFixDangerousPrefix AUTH --mustFixDangerousPrefix CYBER --output reports/openclaw-judges-report-mustfix.md

# opinionated quick-start mode (recommended first run)
npm run report:quickstart -- --repoUrl https://github.com/openclaw/openclaw --output reports/openclaw-quickstart.md

Call from MCP client:

{
  "tool": "evaluate_public_repo_report",
  "arguments": {
    "repoUrl": "https://github.com/microsoft/vscode",
    "branch": "main",
    "maxFiles": 400,
    "maxFindingsInReport": 120,
    "credentialMode": "strict",
    "includeAstFindings": false,
    "minConfidence": 0.8,
    "enableMustFixGate": true,
    "mustFixMinConfidence": 0.9,
    "mustFixDangerousRulePrefixes": ["AUTH", "CYBER", "DATA"],
    "outputPath": "reports/vscode-judges-report.md"
  }
}

Typical response summary includes:

overall verdict and average score
analyzed file count and total findings
per-judge score table
highest-risk findings and lowest-scoring files

Sample report snippet:

# Public Repository Full Judges Report

Generated from https://github.com/microsoft/vscode on 2026-02-21T12:00:00.000Z.

## Executive Summary
- Overall verdict: WARNING
- Average file score: 78/100
- Total findings: 412 (critical 3, high 29, medium 114, low 185, info 81)

`get_judges`

List all available judges with their domains and descriptions.

`evaluate_code`

Submit code to the full judges panel. all 45 judges evaluate independently and return a combined verdict.

Parameter	Type	Required	Description
`code`	string	yes	The source code to evaluate
`language`	string	yes	Programming language (e.g., `typescript`, `python`)
`context`	string	no	Additional context about the code
`includeAstFindings`	boolean	no	Include AST/code-structure findings (default: true)
`minConfidence`	number	no	Minimum finding confidence to include (0-1, default: 0)
`config`	object	no	Inline configuration (see Configuration)

`evaluate_code_single_judge`

Submit code to a specific judge for targeted review.

Parameter	Type	Required	Description
`code`	string	yes	The source code to evaluate
`language`	string	yes	Programming language
`judgeId`	string	yes	See judge IDs below
`context`	string	no	Additional context
`minConfidence`	number	no	Minimum finding confidence to include (0-1, default: 0)
`config`	object	no	Inline configuration (see Configuration)

`evaluate_project`

Submit multiple files for project-level analysis. all 45 judges evaluate each file, plus cross-file architectural analysis detects code duplication, inconsistent error handling, and dependency cycles.

Parameter	Type	Required	Description
`files`	array	yes	Array of `{ path, content, language }` objects
`context`	string	no	Optional project context
`includeAstFindings`	boolean	no	Include AST/code-structure findings (default: true)
`minConfidence`	number	no	Minimum finding confidence to include (0-1, default: 0)
`config`	object	no	Inline configuration (see Configuration)

`evaluate_diff`

Evaluate only the changed lines in a code diff. Runs all 45 judges on the full file but filters findings to lines you specify. Ideal for PR reviews and incremental analysis.

Parameter	Type	Required	Description
`code`	string	yes	The full file content (post-change)
`language`	string	yes	Programming language
`changedLines`	number[]	yes	1-based line numbers that were changed
`context`	string	no	Optional context about the change
`includeAstFindings`	boolean	no	Include AST/code-structure findings (default: true)
`minConfidence`	number	no	Minimum finding confidence to include (0-1, default: 0)
`config`	object	no	Inline configuration (see Configuration)

`analyze_dependencies`

Analyze a dependency manifest file for supply-chain risks, version pinning issues, typosquatting indicators, and dependency hygiene. Supports package.json, requirements.txt, Cargo.toml, go.mod, pom.xml, and .csproj files.

Parameter	Type	Required	Description
`manifest`	string	yes	Contents of the dependency manifest file
`manifestType`	string	yes	File type: `package.json`, `requirements.txt`, etc.
`context`	string	no	Optional context

`evaluate_git_diff`

Evaluate only changed lines from a git diff. Provide either repoPath for a live git diff or diffText for a pre-computed unified diff.

Parameter	Type	Required	Description
`repoPath`	string	conditional	Absolute path to the git repository
`base`	string	no	Git ref to diff against (default: `HEAD~1`)
`diffText`	string	conditional	Pre-computed unified diff text
`confidenceFilter`	number	no	Minimum confidence threshold for findings (0–1)
`autoTune`	boolean	no	Apply feedback-driven auto-tuning (default: false)
`maxPromptChars`	number	no	Max character budget for LLM prompts (default: 100000, 0 = unlimited)
`config`	object	no	Inline configuration

`re_evaluate_with_context`

Re-run the tribunal with prior findings as context for iterative refinement. Supports dispute resolution, developer context injection, and focus-area filtering.

Parameter	Type	Required	Description
`code`	string	yes	Source code to re-evaluate
`language`	string	yes	Programming language
`disputedRuleIds`	string[]	no	Rule IDs the developer disputes as false positives
`acceptedRuleIds`	string[]	no	Rule IDs the developer accepts
`developerContext`	string	no	Free-form explanation of developer intent
`focusAreas`	string[]	no	Specific areas to focus on (e.g., `["security"]`)
`confidenceFilter`	number	no	Minimum confidence threshold (default: 0.5)
`filePath`	string	no	File path for context-aware evaluation
`deepReview`	boolean	no	Include LLM deep-review prompt section
`relatedFiles`	array	no	Cross-file context `{ path, snippet, relationship? }[]`
`maxPromptChars`	number	no	Max character budget for LLM prompts (default: 100000, 0 = unlimited)

Additional MCP Tools

Tool	Description
`evaluate_file`	Read a file from disk and submit it to the full panel. Auto-detects language from extension.
`evaluate_code_streaming`	Streaming evaluation — returns per-judge results as each judge completes with running aggregates.
`evaluate_focused`	Run only specified judges. Use after an initial full evaluation to re-check specific areas.
`evaluate_batch`	Evaluate multiple code files in a single call. Returns per-file verdicts plus aggregate statistics.
`evaluate_then_fix`	Evaluate code and automatically generate fix patches for all findings with auto-fix support.
`evaluate_with_progress`	Evaluate with progress callbacks for long-running evaluations.
`evaluate_policy_aware`	Policy-aware evaluation with named profiles (startup, regulated, healthcare, fintech, public-sector).
`fix_code`	Evaluate code and apply all available auto-fix patches. Returns fixed code with applied/remaining summary.
`explain_finding`	Explain a finding in plain language with OWASP/CWE references, risk context, and remediation guidance.
`triage_finding`	Set triage status of a finding (accepted-risk, deferred, wont-fix, false-positive) with attribution.
`record_feedback`	Record user feedback (true-positive, false-positive, wont-fix) to calibrate confidence scores.
`get_finding_stats`	Finding lifecycle statistics: open, fixed, recurring, and triaged counts plus trends.
`get_suppression_analytics`	Analyze suppression patterns: FP rates by rule, suppression rates, auto-suppress candidates.
`list_triaged_findings`	List triaged findings, optionally filtered by triage status.
`benchmark_gate`	Run benchmarks against quality thresholds. Returns pass/fail with F1, precision, recall metrics.
`run_benchmark`	Run the full benchmark suite with per-judge, per-category, per-difficulty breakdowns.
`scaffold_judge`	Generate boilerplate files to add a new judge: definition, evaluator skeleton, and registration.
`scaffold_plugin`	Generate a starter plugin template with custom rules, judges, and lifecycle hooks.
`session_status`	Current evaluation session state: evaluation count, frameworks, verdict history, stability.
`list_files`	List files and directories in the workspace for project exploration.
`read_file`	Read file contents from the workspace.

Judge IDs

data-security · cybersecurity · security · cost-effectiveness · scalability · cloud-readiness · software-practices · accessibility · api-design · api-contract · reliability · observability · performance · compliance · data-sovereignty · testing · documentation · internationalization · dependency-health · concurrency · ethics-bias · maintainability · error-handling · authentication · database · caching · configuration-management · backwards-compatibility · portability · ux · logging-privacy · rate-limiting · ci-cd · code-structure · agent-instructions · ai-code-safety · framework-safety · iac-security · hallucination-detection · intent-alignment · multi-turn-coherence · model-fingerprint · over-engineering · logic-review · false-positive-review

MCP Prompts

Each judge has a corresponding prompt for LLM-powered deep analysis:

Prompt	Description
`judge-data-security`	Deep data security review
`judge-cybersecurity`	Deep cybersecurity review
`judge-cost-effectiveness`	Deep cost optimization review
`judge-scalability`	Deep scalability review
`judge-cloud-readiness`	Deep cloud readiness review
`judge-software-practices`	Deep software practices review
`judge-accessibility`	Deep accessibility/WCAG review
`judge-api-design`	Deep API design review
`judge-reliability`	Deep reliability & resilience review
`judge-observability`	Deep observability & monitoring review
`judge-performance`	Deep performance optimization review
`judge-compliance`	Deep regulatory compliance review
`judge-data-sovereignty`	Deep data, technological & operational sovereignty review
`judge-testing`	Deep testing quality review
`judge-documentation`	Deep documentation quality review
`judge-internationalization`	Deep i18n review
`judge-dependency-health`	Deep dependency health review
`judge-concurrency`	Deep concurrency & async safety review
`judge-ethics-bias`	Deep ethics & bias review
`judge-maintainability`	Deep maintainability & tech debt review
`judge-error-handling`	Deep error handling review
`judge-authentication`	Deep authentication & authorization review
`judge-database`	Deep database design & query review
`judge-caching`	Deep caching strategy review
`judge-configuration-management`	Deep configuration & secrets review
`judge-backwards-compatibility`	Deep backwards compatibility review
`judge-portability`	Deep platform portability review
`judge-ux`	Deep user experience review
`judge-logging-privacy`	Deep logging privacy review
`judge-rate-limiting`	Deep rate limiting review
`judge-ci-cd`	Deep CI/CD pipeline review
`judge-code-structure`	Deep AST-based structural analysis review
`judge-agent-instructions`	Deep review of agent instruction markdown quality and safety
`judge-ai-code-safety`	Deep review of AI-generated code risks: prompt injection, insecure LLM output handling, debug defaults, missing validation
`judge-framework-safety`	Deep review of framework-specific safety: React hooks, Express middleware, Next.js SSR/SSG, Angular/Vue, Django, Spring Boot, ASP.NET Core, Flask, FastAPI, Go frameworks
`judge-iac-security`	Deep review of infrastructure-as-code security: Terraform, Bicep, ARM template misconfigurations
`judge-security`	Deep holistic security posture review: insecure data flows, weak cryptography, unsafe deserialization
`judge-hallucination-detection`	Deep review of AI-hallucinated APIs, fabricated imports, non-existent modules
`judge-intent-alignment`	Deep review of code–comment alignment, stub detection, placeholder functions
`judge-api-contract`	Deep review of API contract conformance, input validation, REST best practices
`judge-multi-turn-coherence`	Deep review of code coherence: self-contradictions, duplicate definitions, dead code
`judge-model-fingerprint`	Deep review of AI code provenance and model attribution fingerprints
`judge-over-engineering`	Deep review of unnecessary abstractions, wrapper-mania, premature generalization
`judge-logic-review`	Deep review of logic correctness, semantic mismatches, and dead code in AI-generated code
`judge-false-positive-review`	Meta-judge review of pattern-based findings for false positive detection and accuracy

Configuration

Create a .judgesrc.json (or .judgesrc) file in your project root to customize evaluation behavior. See .judgesrc.example.json for a copy-paste-ready template, or reference the JSON Schema for full IDE autocompletion.

{
  "$schema": "https://github.com/KevinRabun/judges/blob/main/judgesrc.schema.json",
  "preset": "strict",
  "minSeverity": "medium",
  "disabledRules": ["COST-*", "I18N-001"],
  "disabledJudges": ["accessibility", "ethics-bias"],
  "ruleOverrides": {
    "SEC-003": { "severity": "critical" },
    "DOC-*": { "disabled": true }
  },
  "languages": ["typescript", "python"],
  "format": "text",
  "failOnFindings": false,
  "baseline": "",
  "regulatoryScope": ["GDPR", "PCI-DSS", "SOC2"],
  "consensusThreshold": 0.7
}

Field	Type	Default	Description
`$schema`	`string`	—	JSON Schema URL for IDE validation
`preset`	`string`	—	Named preset (see Named Presets for all 22 options)
`minSeverity`	`string`	`"info"`	Minimum severity to report: `critical` · `high` · `medium` · `low` · `info`
`disabledRules`	`string[]`	`[]`	Rule IDs or prefix wildcards to suppress (e.g. `"COST-*"`, `"SEC-003"`)
`disabledJudges`	`string[]`	`[]`	Judge IDs to skip entirely (e.g. `"cost-effectiveness"`)
`ruleOverrides`	`object`	`{}`	Per-rule overrides keyed by rule ID or wildcard — `{ disabled?: boolean, severity?: string }`
`languages`	`string[]`	`[]`	Restrict analysis to specific languages (empty = all)
`format`	`string`	`"text"`	Default output format: `text` · `json` · `sarif` · `markdown` · `html` · `pdf` · `junit` · `codeclimate` · `github-actions`
`failOnFindings`	`boolean`	`false`	Exit code 1 when verdict is `fail` — useful for CI gates
`baseline`	`string`	`""`	Path to a baseline JSON file — matching findings are suppressed
`plugins`	`string[]`	`[]`	Plugin module specifiers (npm packages or relative paths) that export custom judges
`judgeWeights`	`object`	`{}`	Weighted importance per judge for aggregated scoring (e.g. `{ "cybersecurity": 2.0 }`)
`failOnScoreBelow`	`number`	—	Minimum score (0–100) for the run to pass; complements `failOnFindings`
`regulatoryScope`	`string[]`	—	Regulatory frameworks in scope (e.g. `["GDPR", "PCI-DSS"]`). Findings citing ONLY out-of-scope frameworks are suppressed. Run `judges list --frameworks` for supported values.
`consensusThreshold`	`number`	—	Consensus suppression (0–1). If this fraction of judges report zero findings, minority findings are suppressed. Recommended: `0.7` for CI.
`escalationThreshold`	`number`	—	Confidence threshold (0–1) below which findings are flagged for human review
`overrides`	`array`	`[]`	Path-scoped config overrides (e.g. `[{ "files": "*/.test.ts", "disabledJudges": ["documentation"] }]`)
`customRules`	`array`	`[]`	User-defined regex-based rules for business logic validation

All evaluation tools (CLI and MCP) accept the same configuration fields via --config <path> or inline config parameter.

Advanced Features

Inline Suppressions

Suppress specific findings directly in source code using comment directives:

const x = eval(input); // judges-ignore SEC-001
// judges-ignore-next-line CYBER-002
const y = dangerousOperation();
// judges-file-ignore DOC-*    ← suppress globally for this file

Supported comment styles: //, #, /* */. Supports comma-separated rule IDs and wildcards (*, SEC-*).

Auto-Fix Patches

Certain findings include machine-applicable patches in the patch field:

Pattern	Auto-Fix
`new Buffer(x)`	→ `Buffer.from(x)`
`http://` URLs (non-localhost)	→ `https://`
`Math.random()`	→ `crypto.randomUUID()`

Patches include oldText, newText, startLine, and endLine for automated application.

Cross-Evaluator Deduplication

When multiple judges flag the same issue (e.g., both Data Security and Cybersecurity detect SQL injection on line 15), findings are automatically deduplicated. The highest-severity finding wins, and the description is annotated with cross-references (e.g., "Also identified by: CYBER-003").

Human Focus Guide

Every tribunal evaluation includes a humanFocusGuide that categorizes findings into three buckets for human reviewers:

Bucket	Description	When to use
✅ Trust	High-confidence (≥80%), evidence-backed findings with AST/taint confirmation	Act directly — these have strong automated evidence
🔍 Verify	Lower-confidence or absence-based findings	Use your judgment — the issue may exist elsewhere in the project
🔦 Blind Spots	Areas automated analysis cannot evaluate	Focus your manual review time here

Blind spots are detected from code characteristics: complex branching logic, external service calls, financial calculations, PII handling, state machines, and complex regex. The guide appears in CLI text/markdown output, JSON/SARIF output, and GitHub Action step summaries.

Regulatory Scope

Configure which regulatory frameworks apply to your project in .judgesrc:

{ "regulatoryScope": ["GDPR", "PCI-DSS", "SOC2"] }

Findings that cite ONLY out-of-scope frameworks are suppressed. Findings with no regulatory reference (general code quality) are always kept. Run judges list --frameworks to see all 17 supported frameworks (GDPR, CCPA, HIPAA, PCI-DSS, SOC2, SOX, COPPA, FedRAMP, NIST, ISO27001, ePrivacy, DORA, NIS2, EU-AI-Act, and more).

Self-Teaching Amendments

The LLM benchmark system auto-generates precision amendments for judges with high false-positive rates. Amendments are data-driven corrections injected into prompts that improve accuracy over successive benchmark runs.

The self-teaching loop:

Run benchmark → analyzer identifies judges below 70% precision
Generates targeted amendments (e.g., "Judge ERR: do not flag clean Express code with framework error middleware")
Next benchmark run loads amendments → precision improves
Run judges codify-amendments to bake amendments permanently into the distributed package

Taint Flow Analysis

The engine performs inter-procedural taint tracking to trace data from user-controlled sources (e.g., req.body, process.env) through transformations to security-sensitive sinks (e.g., eval(), exec(), SQL queries). Taint flows are used to boost confidence on true-positive findings and suppress false positives where sanitization is detected.

Positive Signal Detection

Code that demonstrates good practices receives score bonuses (capped at +15):

Signal	Bonus
Parameterized queries	+3
Security headers (helmet)	+3
Auth middleware (passport, etc.)	+3
Proper error handling	+2
Input validation libs (zod, joi, etc.)	+2
Rate limiting	+2
Structured logging (pino, winston)	+2
CORS configuration	+1
Strict mode / strictNullChecks	+1
Test patterns (describe/it/expect)	+1

Framework-Aware Rules

Judges include framework-specific detection for Express, Django, Flask, FastAPI, Spring, ASP.NET, Rails, and more. Framework middleware (e.g., helmet(), express-rate-limit, passport.authenticate()) is recognized as mitigation, reducing false positives.

Cross-File Import Resolution

In project-level analysis, imports are resolved across files. If one file imports a security middleware module from another file in the project, findings about missing security controls are automatically adjusted with reduced confidence.

Scoring

Each judge scores the code from 0 to 100:

Severity	Score Deduction
Critical	−30 points
High	−18 points
Medium	−10 points
Low	−5 points
Info	−2 points

Verdict logic:

FAIL — Any critical finding, or score < 60
WARNING — Any high finding, any medium finding, or score < 80
PASS — Score ≥ 80 with no critical, high, or medium findings

The overall tribunal score is the average of all 45 judges. The overall verdict fails if any judge fails.

Project Structure

judges/
├── src/
│   ├── index.ts              # MCP server entry point — tools, prompts, transport
│   ├── api.ts                # Programmatic API entry point
│   ├── cli.ts                # CLI argument parser and command router
│   ├── types.ts              # TypeScript interfaces (Finding, JudgeEvaluation, etc.)
│   ├── config.ts             # .judgesrc configuration parser and validation
│   ├── errors.ts             # Custom error types (ConfigError, EvaluationError, ParseError)
│   ├── language-patterns.ts  # Multi-language regex pattern constants and helpers
│   ├── judge-registry.ts     # Unified JudgeRegistry — single source of truth for all judges
│   ├── plugins.ts            # Plugin API façade (delegates to JudgeRegistry)
│   ├── scoring.ts            # Confidence scoring and calibration
│   ├── dedup.ts              # Finding deduplication engine
│   ├── fingerprint.ts        # Finding fingerprint generation
│   ├── comparison.ts         # Tool comparison benchmark data
│   ├── cache.ts              # Evaluation result caching
│   ├── calibration.ts        # Confidence calibration from feedback data
│   ├── fix-history.ts        # Auto-fix application history tracking
│   ├── ast/                  # AST analysis engine (built-in, no external deps)
│   │   ├── index.ts          # analyzeStructure() — routes to correct parser
│   │   ├── types.ts          # FunctionInfo, CodeStructure interfaces
│   │   ├── tree-sitter-ast.ts    # Tree-sitter WASM parser (all 8 languages)
│   │   ├── structural-parser.ts  # Fallback scope-tracking parser
│   │   ├── cross-file-taint.ts   # Cross-file taint propagation analysis
│   │   └── taint-tracker.ts      # Single-file taint flow tracking
│   ├── evaluators/           # Analysis engine for each judge
│   │   ├── index.ts          # evaluateWithJudge(), evaluateWithTribunal(), evaluateProject(), etc.
│   │   ├── shared.ts         # Scoring, verdict logic, markdown formatters
│   │   └── *.ts              # One analyzer per judge (45 files)
│   ├── formatters/           # Output formatters
│   │   ├── sarif.ts              # SARIF 2.1.0 output
│   │   ├── html.ts               # Self-contained HTML report (dark/light theme, filters)
│   │   ├── junit.ts              # JUnit XML output (Jenkins, Azure DevOps, GitHub Actions)
│   │   ├── codeclimate.ts        # CodeClimate/GitLab Code Quality JSON
│   │   ├── diagnostics.ts        # Diagnostics formatter
│   │   └── badge.ts              # SVG and text badge generator
│   ├── commands/             # CLI subcommands
│   │   ├── init.ts               # Interactive project setup wizard
│   │   ├── fix.ts                # Auto-fix patch preview and application
│   │   ├── watch.ts              # Watch mode — re-evaluate on save
│   │   ├── report.ts             # Project-level local report
│   │   ├── hook.ts               # Pre-commit hook install/uninstall
│   │   ├── ci-templates.ts       # GitLab, Azure, Bitbucket CI templates
│   │   ├── diff.ts               # Evaluate unified diff (git diff)
│   │   ├── deps.ts               # Dependency supply-chain analysis
│   │   ├── baseline.ts           # Create baseline for finding suppression
│   │   ├── completions.ts        # Shell completions (bash/zsh/fish/PowerShell)
│   │   ├── docs.ts               # Per-judge rule documentation generator
│   │   ├── feedback.ts           # False-positive tracking & finding feedback
│   │   ├── benchmark.ts          # Detection accuracy benchmark suite
│   │   ├── rule.ts               # Custom rule authoring wizard
│   │   ├── language-packs.ts     # Language-specific rule pack presets
│   │   └── config-share.ts       # Shareable team/org configuration
│   ├── presets.ts            # Named evaluation presets (strict, lenient, security-only, …)
│   ├── patches/
│   │   └── index.ts              # 201 deterministic auto-fix patch rules
│   ├── tools/                # MCP tool registrations
│   │   ├── register.ts           # Tool registration orchestrator
│   │   ├── register-evaluation.ts    # Evaluation tools (evaluate_code, etc.)
│   │   ├── register-workflow.ts      # Workflow tools (app builder, reports, etc.)
│   │   ├── prompts.ts            # MCP prompt registrations (per-judge prompts)
│   │   └── schemas.ts            # Zod schemas for tool parameters
│   ├── reports/
│   │   └── public-repo-report.ts   # Public repo clone + full tribunal report generation
│   └── judges/               # Judge definitions (id, name, domain, system prompt)
│       ├── index.ts          # Side-effect imports + re-exports (JUDGES, getJudge, getJudgeSummaries)
│       └── *.ts              # One self-registering definition per judge (45 files)
├── scripts/
│   ├── generate-public-repo-report.ts  # Run: npm run report:public-repo -- --repoUrl <url>
│   ├── daily-popular-repo-autofix.ts   # Run: npm run automation:daily-popular
│   └── debug-fp.ts                     # Debug false-positive findings
├── examples/
│   ├── sample-vulnerable-api.ts  # Intentionally flawed code (triggers all judges)
│   ├── demo.ts                   # Run: npm run demo
│   └── quickstart.ts             # Quick-start evaluation example
├── tests/
│   ├── judges.test.ts            # Core judge evaluation tests
│   ├── negative.test.ts          # Negative / FP-avoidance tests
│   ├── subsystems.test.ts        # Subsystem integration tests
│   ├── extension-logic.test.ts   # VS Code extension logic tests
│   └── tool-routing.test.ts      # MCP tool routing tests
├── grammars/                 # Tree-sitter WASM grammar files
│   ├── tree-sitter-typescript.wasm
│   ├── tree-sitter-cpp.wasm
│   ├── tree-sitter-python.wasm
│   ├── tree-sitter-go.wasm
│   ├── tree-sitter-rust.wasm
│   ├── tree-sitter-java.wasm
│   └── tree-sitter-c_sharp.wasm
├── judgesrc.schema.json      # JSON Schema for .judgesrc config files
├── server.json               # MCP Registry manifest
├── package.json
├── tsconfig.json
└── README.md

Scripts

Command	Description
`npm run build`	Compile TypeScript to `dist/`
`npm run dev`	Watch mode — recompile on save
`npm test`	Run the full test suite
`npm run demo`	Run the sample tribunal demo
`npm run report:public-repo -- --repoUrl <url>`	Generate a full tribunal report for a public repository URL
`npm run report:quickstart -- --repoUrl <url>`	Run opinionated high-signal report defaults for fast adoption
`npm run automation:daily-popular`	Analyze up to 10 rotating popular repos/day and open up to 5 remediation PRs per repo
`npm start`	Start the MCP server
`npm run clean`	Remove `dist/`
`judges init`	Interactive project setup wizard
`judges fix <file>`	Preview auto-fix patches (add `--apply` to write)
`judges watch <dir>`	Watch mode — re-evaluate on file save
`judges report <dir>`	Full tribunal report on a local directory
`judges hook install`	Install a Git pre-commit hook
`judges diff`	Evaluate changed lines from unified diff
`judges deps`	Analyze dependencies for supply-chain risks
`judges baseline create`	Create baseline for finding suppression
`judges ci-templates`	Generate CI pipeline templates
`judges docs`	Generate per-judge rule documentation
`judges completions <shell>`	Shell completion scripts
`judges feedback submit`	Mark findings as true positive, false positive, or won't fix
`judges feedback stats`	Show false-positive rate statistics
`judges benchmark run`	Run detection accuracy benchmark suite
`judges rule create`	Interactive custom rule creation wizard
`judges rule list`	List custom evaluation rules
`judges pack list`	List available language packs
`judges config export`	Export config as shareable package
`judges config import <src>`	Import a shared configuration
`judges compare`	Compare judges against other code review tools
`judges list`	List all 45 judges with domains and descriptions
`judges list --frameworks`	List supported regulatory frameworks and `.judgesrc` usage
`judges codify-amendments`	Bake self-teaching amendments into judge source files

Daily Popular Repo Automation

This repo includes a scheduled workflow at .github/workflows/daily-popular-repo-autofix.yml that:

selects up to 10 repositories per day from a default pool of 100+ popular repos (or a manually supplied target),
runs the full Judges evaluation across supported source languages,
applies only conservative, single-line remediations that reduce matching finding counts,
opens up to 5 PRs per repository with attribution to both Judges and the target repository,
skips repositories unless they are public and PR creation is possible with existing GitHub auth (no additional auth flow).
enforces hard runtime caps of 10 repositories/day and 5 PRs/repository.

Each run writes daily-autofix-summary.json (or SUMMARY_PATH) with per-repository telemetry, including:

runAggregate — compact run-level totals and cross-repo top prioritized rules,
runAggregate.totalCandidatesDiscovered and runAggregate.totalCandidatesAfterLocationDedupe — signal how much overlap was removed before attempting fixes,
runAggregate.totalCandidatesAfterPriorityThreshold — candidates that remain after applying minimum priority score,
runAggregate.dedupeReductionPercent — percent reduction from location dedupe for quick runtime-efficiency tracking,
runAggregate.priorityThresholdReductionPercent — percent reduction from minimum-priority filtering after dedupe,
priorityRulePrefixesUsed — dangerous rule prefixes used during prioritization,
minPriorityScoreUsed — minimum candidatePriorityScore applied for candidate inclusion,
candidatesDiscovered, candidatesAfterLocationDedupe, and candidatesAfterPriorityThreshold — per-repo candidate counts after each filter stage,
topPrioritizedRuleCounts — most common rule IDs among ranked candidates,
topPrioritizedCandidates — top ranked candidate samples (rule, severity, confidence, file, line, priority score).

Optional runtime control:

AUTOFIX_MIN_PRIORITY_SCORE — minimum candidate priority score required after dedupe (default: 0, disabled).

Required secret:

JUDGES_AUTOFIX_GH_TOKEN — GitHub token with permission to fork/push/create PRs for target repositories.

Manual run:

gh workflow run "Judges Daily Full-Run Autofix PRs" -f targetRepoUrl=https://github.com/owner/repo

Programmatic API

Judges can be consumed as a library (not just via MCP). Import from @kevinrabun/judges/api:

import {
  evaluateCode,
  evaluateProject,
  evaluateCodeSingleJudge,
  getJudge,
  JUDGES,
  findingsToSarif,
} from "@kevinrabun/judges/api";

// Full tribunal evaluation
const verdict = evaluateCode("const x = eval(input);", "typescript");
console.log(verdict.overallScore, verdict.overallVerdict);

// Single judge
const result = evaluateCodeSingleJudge("cybersecurity", code, "typescript");

// SARIF output for CI integration
const sarif = findingsToSarif(verdict.evaluations.flatMap(e => e.findings));

Package Exports

Entry Point	Description
`@kevinrabun/judges/api`	Programmatic API (default)
`@kevinrabun/judges/server`	MCP server entry point
`@kevinrabun/judges/sarif`	SARIF 2.1.0 formatter
`@kevinrabun/judges/junit`	JUnit XML formatter
`@kevinrabun/judges/codeclimate`	CodeClimate/GitLab Code Quality JSON
`@kevinrabun/judges/badge`	SVG and text badge generator
`@kevinrabun/judges/diagnostics`	Diagnostics formatter
`@kevinrabun/judges/plugins`	Plugin system API (see Plugin Guide)
`@kevinrabun/judges/fingerprint`	Finding fingerprint utilities
`@kevinrabun/judges/comparison`	Tool comparison benchmarks

SARIF Output

Convert findings to SARIF 2.1.0 for GitHub Code Scanning, Azure DevOps, and other CI/CD tools:

import { findingsToSarif, evaluationToSarif, verdictToSarif } from "@kevinrabun/judges/sarif";

const sarif = verdictToSarif(verdict, "src/app.ts");
fs.writeFileSync("results.sarif", JSON.stringify(sarif, null, 2));

Custom Error Types

All thrown errors extend JudgesError with a machine-readable code property:

Error Class	Code	When
`ConfigError`	`JUDGES_CONFIG_INVALID`	Malformed `.judgesrc` or invalid inline config
`EvaluationError`	`JUDGES_EVALUATION_FAILED`	Unknown judge, analyzer crash
`ParseError`	`JUDGES_PARSE_FAILED`	Unparseable source code or input data

import { ConfigError, EvaluationError } from "@kevinrabun/judges/api";
try {
  evaluateCode(code, "typescript");
} catch (e) {
  if (e instanceof ConfigError) console.error("Config issue:", e.code);
}

License

MIT

Available Tools

25 tools

analyze_dependenciesA

Analyze a PACKAGE MANAGER manifest file (NOT infrastructure code) for supply-chain risks, version pinning issues, typosquatting indicators, and dependency hygiene. ONLY accepts: package.json, requirements.txt, Cargo.toml, go.mod, pom.xml, .csproj. Do NOT use this for Bicep, Terraform, ARM templates, CloudFormation, Dockerfiles, or any other infrastructure/deployment configuration — use evaluate_code or evaluate_code_single_judge for those.

ParametersJSON Schema

Name	Required	Description	Default
`manifest`	Yes	The full content of the manifest file
`manifestType`	Yes	The type of manifest file

TDQS

A4.5/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden for behavioral disclosure. The description states the tool analyzes files for risks, which implies a non-destructive read operation. However, it does not disclose whether the tool has side effects (e.g., saving reports), authentication requirements, rate limits, or the return format. While adequate, it lacks some behavioral context that would be helpful.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise, consisting of two sentences. The first sentence clearly states the purpose and scope, and the second sentence provides exclusion criteria and alternatives. No extraneous information, well front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity (2 parameters, no output schema, no nested objects) and the lack of annotations, the description is complete enough for correct selection and invocation. It specifies input format, allowed manifest types, and what not to use the tool for. The absence of output schema details is not a flaw as the tool likely produces a standard report.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 100% schema coverage, the schema already documents both parameters (manifest and manifestType). The description adds value by clarifying that 'manifest' should be the full content of the file and lists the exact enum values for manifestType, reinforcing the schema's meaning. This adds practical context beyond the schema alone.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool analyzes package manager manifest files for supply-chain risks, version pinning issues, typosquatting indicators, and dependency hygiene. It explicitly lists accepted file types (package.json, requirements.txt, Cargo.toml, go.mod, pom.xml, .csproj) and distinguishes from sibling tools by specifying what not to use it for (infrastructure code like Bicep, Terraform, etc.).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit usage guidance: it is specifically for manifest files and not for infrastructure/deployment configuration. It names alternative tools (evaluate_code or evaluate_code_single_judge) that should be used for excluded scenarios, giving clear when-to-use and when-not-to-use instructions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

benchmark_gateA

Run the benchmark suite and check results against quality thresholds. Returns pass/fail with metric details including F1, precision, recall, and detection rate. Use in CI pipelines to prevent quality regressions.

ParametersJSON Schema

Name	Required	Description
`minF1`	No	Minimum F1 score (0-1, default: 0.6)
`minRecall`	No	Minimum recall (0-1, default: 0.5)
`minPrecision`	No	Minimum precision (0-1, default: 0.5)
`minDetectionRate`	No	Minimum detection rate (0-1, default: 0.5)

TDQS

A3.8/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries full burden. It explains the tool runs checks and returns pass/fail with metrics, but does not disclose whether any state changes or side effects occur (e.g., writing logs or locking resources). The behavior appears read-only but is not confirmed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences covering purpose, return value, and use case. No fluff; every sentence is informative and front-loaded. Ideal for quick understanding.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description explains the return format (pass/fail with metrics) and use case (CI pipelines), which compensates for the lack of an output schema. It could mention how thresholds are used or default actions, but is largely sufficient.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, with each parameter having a description. The tool description adds little beyond listing return metrics; it does not explain how parameters interact or provide usage examples. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool runs a benchmark suite and checks results against thresholds, returning pass/fail with metrics. It distinguishes from 'run_benchmark' by emphasizing the gate functionality, though it does not explicitly differentiate from siblings.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly recommends use in CI pipelines to prevent quality regressions, providing clear context. However, it does not mention when not to use or compare to other tools like 'run_benchmark', leaving some ambiguity.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

evaluate_app_builder_flowA

Run a 3-step app-builder workflow: tribunal review, plain-language risk translation, and prioritized remediation tasks with AI-fixable P0/P1 items.

ParametersJSON Schema

Name	Required	Description
`code`	No	Source code to evaluate (use with language for single-file mode)
`files`	No	Project files for multi-file mode
`context`	No	Optional context about business purpose or constraints
`language`	No	Programming language for single-file or diff mode
`maxTasks`	No	Maximum number of remediation tasks to return (default: 20)
`maxFindings`	No	Maximum number of translated top findings to return (default: 10)
`changedLines`	No	1-based changed line numbers for diff mode
`minConfidence`	No	Minimum finding confidence to include (0-1, default: 0)
`includeAstFindings`	No	Include AST/code-structure findings (default: true)

TDQS

A3.7/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Discloses the three steps and mentions AI-fixable items, but with no annotations it fails to detail potential side effects (e.g., does it modify code?), auth needs, or rate limits; only partial transparency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Single sentence, front-loaded with key steps, no filler. Every word earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given 9 parameters, no output schema, and no annotations, the 1-sentence description is insufficient for full understanding. Missing return format, prerequisites, and error handling; adequate but not complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so baseline is 3. Description adds minimal meaning beyond the schema, only hinting at workflow steps but not explaining how parameters like code/files/context map to those steps.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly specifies the tool runs a 3-step app-builder workflow (tribunal review, risk translation, remediation) with AI-fixable P0/P1 items, distinguishing it from single-step evaluation siblings like evaluate_code or evaluate_project.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Implies usage for app-builder flows but provides no explicit guidance on when to choose this over alternatives like evaluate_code/evaluate_project, nor any exclusions or prerequisites.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

evaluate_batchB

Evaluate multiple code files in a single call. Returns per-file verdicts with scores and findings, plus aggregate statistics.

ParametersJSON Schema

Name	Required	Description	Default
`files`	Yes	Array of files to evaluate
`config`	No	Optional inline configuration (same format as .judgesrc)

TDQS

B3.4/5.0

Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must disclose all behavioral traits. It only mentions return structure (per-file verdicts, aggregate statistics) but omits whether the operation is destructive, required authentication, error handling, performance implications of batching, or limits on input size.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with purpose and output summary. No redundant or extraneous information. Every word adds value.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the lack of output schema and moderate complexity (nested objects, optional config), the description should explain more: behavior when config is omitted, error handling, maximum batch size, and how config overrides affect verdicts. The description is too brief for a batch tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and all parameters are described in the schema. The description adds no additional meaning beyond what the schema provides. With high coverage, baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ('Evaluate') and resource ('multiple code files'), clearly distinguishing it from siblings that evaluate single files or diffs. It states exactly what the tool does and what it returns.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies use for multiple files but provides no guidance on when not to use it or how it differs from alternatives like evaluate_code, evaluate_file, or evaluate_diff. No explicit when-to-use or alternative recommendations.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

evaluate_codeA

Submit code to the full Judges Panel for evaluation. Handles ALL code types including application code, infrastructure-as-code (Bicep, Terraform, ARM, CloudFormation), and configuration files. All 45 judges will independently review the code using both automated pattern detection and deep contextual analysis criteria. Returns a combined verdict with scores, findings, and expert review guidance for thorough evaluation.

ParametersJSON Schema

Name	Required	Description
`code`	Yes	The source code to evaluate. Include the full file content for best results.
`config`	No	Optional inline configuration (same format as .judgesrc)
`context`	No	Optional additional context about the code — e.g., what the code does, which framework it uses, or the deployment target.
`language`	Yes	The programming language of the code (e.g., 'typescript', 'python', 'javascript', 'csharp', 'java').
`relatedFiles`	No	Related files that provide cross-file context for deeper analysis (imports, shared types, call sites)
`minConfidence`	No	Minimum finding confidence to include (0-1, default: 0)
`includeAstFindings`	No	Include AST/code-structure findings (default: true)

TDQS

A3.8/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description bears the full burden. It discloses that 45 judges independently review with automated and deep contextual analysis, and returns a combined verdict. However, it does not disclose potential costs, execution time, or side effects, which would be valuable for such a resource-heavy operation.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is three concise sentences, front-loaded with the main action. Each sentence adds value without redundancy, making it efficient and easy to digest.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given 7 parameters, 2 required, nested config, and no output schema, the description covers the tool's input, process, and output adequately. It mentions the verdict includes scores, findings, and guidance, but could be more explicit about the output structure. Overall, it provides enough context for the agent.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3. The description adds minimal parameter-level insight beyond what the schema provides, such as listing code types but not explaining the config object or relatedFiles in more depth.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb ('Submit code to the full Judges Panel for evaluation') and specifies the resource (code). It distinguishes from siblings by emphasizing 'full Judges Panel' and 'ALL code types', contrasting with tools like evaluate_code_single_judge or evaluate_diff.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for full panel evaluation but does not explicitly state when to use this tool over alternatives such as evaluate_code_single_judge or evaluate_code_streaming. No exclusions or conditions are provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

evaluate_code_single_judgeA

Submit code to a specific judge for targeted domain analysis. Handles ALL code types including application code, infrastructure-as-code (Bicep, Terraform, ARM, CloudFormation), and configuration files. Key domains: cybersecurity, data-sovereignty, iac-security, compliance, cost-effectiveness, authentication, cloud-readiness, and 38 more. Available judge IDs: accessibility, agent-instructions, ai-code-safety, api-contract, api-design, authentication, backwards-compatibility, caching, ci-cd, cloud-readiness, code-structure, compliance, concurrency, configuration-management, cost-effectiveness, cybersecurity, data-security, data-sovereignty, database, dependency-health, documentation, error-handling, ethics-bias, framework-safety, hallucination-detection, iac-security, intent-alignment, internationalization, logging-privacy, logic-review, maintainability, model-fingerprint, multi-turn-coherence, observability, over-engineering, performance, portability, rate-limiting, reliability, scalability, security, software-practices, testing, ux, false-positive-review

ParametersJSON Schema

Name	Required	Description
`code`	Yes	The source code to evaluate. Include the full file content for best results.
`config`	No	Optional inline configuration (same format as .judgesrc)
`context`	No	Optional additional context about the code — e.g., what the code does, which framework it uses, or the deployment target.
`judgeId`	Yes	The ID of the judge to use. One of: accessibility, agent-instructions, ai-code-safety, api-contract, api-design, authentication, backwards-compatibility, caching, ci-cd, cloud-readiness, code-structure, compliance, concurrency, configuration-management, cost-effectiveness, cybersecurity, data-security, data-sovereignty, database, dependency-health, documentation, error-handling, ethics-bias, framework-safety, hallucination-detection, iac-security, intent-alignment, internationalization, logging-privacy, logic-review, maintainability, model-fingerprint, multi-turn-coherence, observability, over-engineering, performance, portability, rate-limiting, reliability, scalability, security, software-practices, testing, ux, false-positive-review
`language`	Yes	The programming language of the code (e.g., 'typescript', 'python', 'javascript', 'csharp', 'java').
`relatedFiles`	No	Related files that provide cross-file context for deeper analysis
`minConfidence`	No	Minimum finding confidence to include (0-1, default: 0)

TDQS

A3.5/5.0

Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries full burden. It discloses no behavioral traits such as permissions, side effects, rate limits, or return format. For a tool that submits code for analysis, details about whether the analysis is synchronous, what happens on failure, or any idempotency guarantees are missing, leaving agents uninformed about critical operational aspects.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences: the first clearly states purpose, the second lists domains and judge IDs. It is front-loaded and relatively concise, though the list of judge IDs is lengthy and partially redundant with the schema. Every sentence adds value, but the list could be trimmed or referenced instead of enumerated.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (7 parameters, nested config object, no output schema), the description lacks completeness. It does not explain what the tool returns (e.g., findings, severity, confidence), how errors are handled, or how the analysis relates to sibling tools like evaluate_code_streaming. The user needs more context to set expectations for the result.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3. The description adds no additional meaning beyond the schema; it repeats the judge IDs list already in the schema's description. It does not explain the interaction of parameters (e.g., how config overrides work with judgeId) or provide behavioral nuance. Thus, no extra value beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states 'Submit code to a specific judge for targeted domain analysis,' identifying the verb (submit), resource (code), and target (single judge). It distinguishes from sibling tools like evaluate_code and evaluate_batch, and lists all 43 judge IDs, making the tool's scope unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explains that the tool handles all code types and lists domains and judge IDs, providing clear context for when to use a specific judge. It does not explicitly state when not to use or name alternatives, but the sibling tool list (e.g., evaluate_code, evaluate_project) implies different use cases, making the guidance strong but not fully explicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

evaluate_code_streamingA

Submit code for streaming evaluation — returns per-judge results as each judge completes, with running aggregate scores. Ideal for long evaluations where you want progressive feedback. All 45 judges run sequentially with per-judge results accumulated into a single structured response.

ParametersJSON Schema

Name	Required	Description
`code`	Yes	The source code to evaluate.
`config`	No	Optional inline configuration (same format as .judgesrc)
`context`	No	Optional context about the code.
`language`	Yes	The programming language (e.g., 'typescript', 'python', 'javascript').
`minConfidence`	No	Minimum finding confidence to include (0-1, default: 0)
`includeAstFindings`	No	Include AST/code-structure findings (default: true)

TDQS

A4.2/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full responsibility. It discloses key behaviors: all 45 judges run sequentially, per-judge results stream as each completes, and running aggregates are provided. Missing details on error handling or auth, but the disclosed streaming behavior is adequate.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two concise sentences: the first defines the core functionality and output, the second reinforces the use case and sequential execution. No wasted words, front-loaded with essential information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no output schema and a moderately complex input, the description covers the streaming behavior well. It lacks explicit details about the expected response structure (beyond 'single structured response') or error scenarios, but the core functionality is clear and complete for the tool's purpose.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with solid field descriptions. The tool description does not add new parameter-level insights beyond restating the streaming nature. Baseline of 3 is appropriate since the schema already documents the parameters sufficiently.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly states the tool submits code for streaming evaluation, specifies it returns per-judge results with running aggregates, and contrasts with siblings like evaluate_code and evaluate_code_single_judge by emphasizing sequential execution and progressive feedback.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Description explicitly recommends the tool for long evaluations requiring progressive feedback, which provides a clear use case. While it does not mention exclusions or alternative tools by name, the sibling list implies differentiation and the context is sufficient for an agent to decide.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

evaluate_diffA

Evaluate only the changed lines in a code diff. Runs all 45 judges on the full file but filters findings to only those affecting the specified changed lines. Ideal for PR reviews and incremental analysis.

ParametersJSON Schema

Name	Required	Description
`code`	Yes	The full file content (post-change)
`config`	No	Optional inline configuration (same format as .judgesrc)
`context`	No	Optional context about the change
`language`	Yes	The programming language
`changedLines`	Yes	Array of 1-based line numbers that were changed (added or modified)
`minConfidence`	No	Minimum finding confidence to include (0-1, default: 0)
`includeAstFindings`	No	Include AST/code-structure findings (default: true)

TDQS

A4.2/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description discloses that all 45 judges run on the full file but findings are filtered to changed lines, explaining the underlying behavior. No annotations are provided, so this is valuable context. It does not mention performance implications, but the core pipeline is transparent.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two concise sentences with zero waste. It is front-loaded with the key purpose and immediately provides the unique filtering behavior.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite 7 parameters and a nested config object, the description explains the core mechanism (full analysis + filter) and mentions 45 judges. No output schema exists, but the description is still informative. A complete outline of the return value is missing but not critical.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the schema already describes all parameters. The description adds no additional meaning beyond briefly referencing changed lines. Baseline 3 is appropriate as the description does not enhance parameter understanding.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb 'evaluate' and the resource 'changed lines in a code diff', distinguishing it from sibling tools like evaluate_code (which evaluates entire file) and evaluate_git_diff. It specifies the unique filtering behavior.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly recommends the tool for 'PR reviews and incremental analysis,' providing clear context for use. It does not mention when not to use it or list alternative tools, but the context is sufficient.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

evaluate_fileA

Read a file from disk and submit it to the full Judges Panel for evaluation. Automatically detects the programming language from the file extension. All 45 judges review the code with pattern detection and deep contextual analysis.

ParametersJSON Schema

Name	Required	Description
`config`	No	Optional inline configuration (same format as .judgesrc)
`context`	No	Optional context about the code — framework, use-case, deployment target.
`filePath`	Yes	Absolute or relative path to the file to evaluate.
`language`	No	Override the detected language (e.g., 'typescript', 'python').
`minConfidence`	No	Minimum finding confidence to include (0-1, default: 0)
`includeAstFindings`	No	Include AST/code-structure findings (default: true)

TDQS

A3.6/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so description carries burden. Discloses reading file, auto-detection, and full judge panel, but lacks details on side effects, authorization, rate limits, or return format. Partially transparent.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, no redundancy, front-loaded with action. Every sentence adds value.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema or annotations; description omits result format and post-evaluation steps. With 6 parameters and many sibling tools, more context on what the tool returns and how to use results is needed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema has 100% description coverage, so description adds little beyond stating config format similar to .judgesrc. Baseline score appropriate as schema already documents parameters.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly states verb (read and submit), resource (file from disk), and behavior (auto-detect language, all 45 judges). Distinguishes from siblings like evaluate_code or evaluate_batch through focus on file reading.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Implies use for evaluating a file from disk, but no explicit when-to-use or when-not-to-use guidance. No mention of alternatives like evaluate_code (for code strings) or evaluate_project.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

evaluate_git_diffA

Evaluate code changes from a git diff. Parses the unified diff from a git repository, identifies changed files and lines, and runs the full tribunal on each changed file — filtering findings to only those on changed lines. Supports both live git repos (provide repoPath + base ref) and pre-computed diffs (provide diffText).

ParametersJSON Schema

Name	Required	Description
`base`	No	Git ref to diff against (e.g., 'main', 'HEAD~1', 'origin/main'). Default: 'HEAD~1'
`config`	No	Optional inline configuration (same format as .judgesrc)
`autoTune`	No	Apply feedback-driven auto-tuning to reduce false positives (default: false)
`diffText`	No	Pre-computed unified diff text. When provided, repoPath is used only for reading file contents.
`repoPath`	No	Absolute path to the git repository. Required when not providing diffText.
`maxPromptChars`	No	Maximum character budget for LLM prompts. Controls truncation of deep-review prompts. Set to 0 to disable all truncation. Default: 100000.
`confidenceFilter`	No	Minimum confidence threshold for findings (default: no filter)

TDQS

A3.5/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations exist, so the description must fully disclose behavior. It states the tool filters findings to changed lines and supports two input modes. However, it does not mention side effects (e.g., read-only nature), permissions, or limitations (e.g., repo must be accessible). The description covers core behavior but omits important operational details.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise: three sentences that front-load the purpose and then detail modes. Every sentence adds value with no redundancy or fluff. Ideal length for quick comprehension.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has 7 parameters, a nested config object, and no output schema. The description explains the overall workflow and two input modes but does not cover configuration options, auto-tuning, or confidence filtering. While the schema fills the gaps, the agent might benefit from a brief summary of optional features.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3. The description adds context about the two key modes (repoPath+base vs diffText) but does not elaborate on other parameters like config, autoTune, maxPromptChars, or confidenceFilter. It provides minimal added value beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool evaluates code changes from a git diff, parses unified diff, runs tribunal on changed lines, and supports both live repos and pre-computed diffs. The verb 'evaluate' is specific and the resource 'git diff' is well-defined. However, it does not distinguish from the sibling tool 'evaluate_diff', which may have overlapping functionality.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explains two usage modes (live repo vs pre-computed diff) and provides default base ref, but does not give explicit guidance on when not to use this tool or mention alternative tools (e.g., evaluate_code for whole-file analysis). The context is clear but lacks exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

evaluate_policy_awareA

Run policy-aware tribunal evaluation with named policy profiles (startup, regulated, healthcare, fintech, public-sector), evidence calibration from runtime metrics, specialty-per-judge feedback, confidence scoring, and uncertainty reporting. Use this when code must meet specific compliance or vertical requirements.

ParametersJSON Schema

Name	Required	Description
`code`	No	Source code for single-file mode
`files`	No	Project files for multi-file mode
`context`	No	Optional high-level context
`evidence`	No	Runtime/operational evidence used for confidence calibration
`language`	No	Language for single-file mode
`minConfidence`	No	Minimum finding confidence to include (0-1, default: 0)
`policyProfile`	No	Policy profile for domain-specific severity calibration
`evaluationContext`	No	Structured context to improve semantic relevance
`includeAstFindings`	No	Include AST/code-structure findings (default: true)

TDQS

A3.9/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It describes the tool's features like policy profiles, evidence calibration, and confidence scoring but does not disclose any side effects, permissions, or whether it is read-only. Given the lack of annotations, the description is adequate but not fully transparent.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise with two sentences: the first lists key features, the second provides usage guidance. It is front-loaded with essential information and contains no extraneous text.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has 9 parameters, nested objects, and no output schema. The description covers the high-level purpose and usage but omits details about the output format, confidence scoring specifics, or uncertainty reporting structure. Given the complexity, the description is adequate but could be more comprehensive.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so baseline is 3. The description adds context by explaining how parameters like evidence and policyProfile fit into the overall evaluation, but it does not significantly augment the meaning beyond what the schema already provides.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool performs policy-aware evaluation with specific named profiles, evidence calibration, and confidence scoring. It differentiates from siblings like evaluate_code by focusing on compliance and vertical-specific requirements.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly says 'Use this when code must meet specific compliance or vertical requirements,' providing a clear directive. It implies alternatives for general evaluation but does not explicitly state when not to use, which would push it to 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

evaluate_projectA

Submit multiple files for project-level analysis. All 45 judges evaluate each file, plus cross-file architectural analysis detects issues like code duplication, inconsistent error handling, and dependency cycles.

ParametersJSON Schema

Name	Required	Description
`files`	Yes	Array of project files to analyze
`config`	No	Optional inline configuration (same format as .judgesrc)
`context`	No	Optional context about the project
`minConfidence`	No	Minimum finding confidence to include (0-1, default: 0)
`includeAstFindings`	No	Include AST/code-structure findings (default: true)

TDQS

A3.7/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden of behavioral disclosure. It reveals that all 45 judges run on each file and cross-file analysis occurs, but omits details on whether the tool is read-only, data retention, rate limits, or side effects. The provided information is helpful but incomplete for a tool handling multiple files.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise: two sentences that front-load the core action and outcome. Every word adds value; no redundant or verbose phrasing.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (multi-file input, 45 judges, cross-file analysis) and lack of output schema, the description adequately outlines the scope. However, it omits what the output looks like (e.g., a report or findings list) and does not mention how results are returned. This leaves gaps for an agent attempting to invoke the tool and process results.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3. The description does not add extra meaning beyond the schema; it mentions 'multiple files' but does not elaborate on config, context, or other parameters. No additional value beyond what the schema already provides.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Submit multiple files for project-level analysis.' It specifies that all 45 judges evaluate each file and that cross-file architectural analysis detects issues like code duplication, inconsistent error handling, and dependency cycles. This contrasts with sibling tools that focus on single files or specific judges, making it easy to distinguish.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for comprehensive project analysis but lacks explicit guidance on when to use this tool versus alternatives like evaluate_batch, evaluate_code, or evaluate_file. No 'when not to use' or prerequisite conditions are provided, leaving the agent to infer context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

evaluate_public_repo_reportB

Clone a public repository URL, run the full judges panel across source files, and generate a consolidated markdown report.

ParametersJSON Schema

Name	Required	Description
`branch`	No	Optional branch name (defaults to repository default branch)
`repoUrl`	Yes	Public repository URL (HTTP/HTTPS)
`maxFiles`	No	Maximum number of source files to analyze (default: 600)
`keepClone`	No	Keep cloned repository on disk for inspection
`outputPath`	No	Optional path to write the markdown report
`maxFileBytes`	No	Maximum single file size in bytes (default: 300000)
`minConfidence`	No	Minimum finding confidence to include (0-1, default: 0)
`credentialMode`	No	Credential detection mode: standard (default) or strict
`enableMustFixGate`	No	Enable must-fix gate for high-confidence dangerous findings (default: false)
`includeAstFindings`	No	Include AST/code-structure findings (default: true)
`maxFindingsInReport`	No	Maximum number of detailed findings in report (default: 150)
`mustFixMinConfidence`	No	Minimum confidence threshold for must-fix gate triggers (0-1, default: 0.85)
`mustFixDangerousRulePrefixes`	No	Optional rule prefixes considered dangerous for must-fix gate

TDQS

B3.2/5.0

Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries full burden. It mentions cloning and report generation but lacks details on data persistence (e.g., temporary clone cleanup), security implications, resource usage (e.g., disk space), or potential side effects. Behavioral traits like 'keepClone' parameter exist but are not explained upfront.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Single sentence efficiently conveys the essential action and output. No fluff; every word adds value. Properly front-loaded with the key steps.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given 13 parameters, no output schema, and no annotations, the description is insufficiently complete. It does not explain 'full judges panel,' report format, or default behavior for critical parameters like maxFiles or maxFileBytes. An agent would need to infer or test many details.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% (all 13 parameters described in input schema). The description adds no additional meaning beyond the high-level purpose; it does not explain parameter interactions or defaults. Baseline 3 is appropriate as schema handles the documentation.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action (clone, run, generate), the resource (public repository), and the output (consolidated markdown report). It effectively distinguishes from sibling tools like evaluate_code (single file) or evaluate_project (likely local) by specifying it works on a public repo via URL.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No explicit guidance on when to use this tool vs. alternatives (e.g., evaluate_code, evaluate_project). The description implies it is for public repos but does not state prerequisites, limitations, or cases where other tools are preferable.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

evaluate_then_fixA

Evaluate code and automatically generate fix patches for all findings that have auto-fix support. Returns the evaluation verdict alongside ready-to-apply patches. Use this for a single-step 'review + fix' workflow.

ParametersJSON Schema

Name	Required	Description
`code`	Yes	The source code to evaluate and fix.
`config`	No	Optional inline configuration (same format as .judgesrc)
`context`	No	Optional context about the code.
`language`	Yes	The programming language (e.g., 'typescript', 'python').
`minConfidence`	No	Minimum finding confidence to include (0-1, default: 0)
`includeAstFindings`	No	Include AST/code-structure findings (default: true)

TDQS

A3.9/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the transparency burden. It mentions auto-fix support, returning patches, and the combined workflow, but lacks details on side effects (e.g., no file modification), error states, or prerequisites. It is acceptable but not thorough.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise (two sentences), front-loaded with the primary functionality, and contains no unnecessary words. It earns its space efficiently.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the complexity (6 parameters, nested config, no output schema), the description is minimal. It mentions output as 'verdict alongside patches' but does not detail the response format or how patches are applied. It is adequate for the tool's simplicity but could provide more guidance.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so all parameters are described in the input schema. The tool description adds no additional explanation beyond the schema. Baseline score is 3, and no extra value is provided.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool evaluates code and generates fix patches for auto-fixable findings. It returns both a verdict and patches. This distinguishes it from sibling tools like `evaluate_code` (evaluation only) and `fix_code` (apply fixes), as it combines both steps.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly advises to use this for a 'single-step review + fix workflow', providing clear context. However, it does not explicitly state when to use alternative tools (e.g., evaluate_code for pure evaluation), though this is implied by sibling tool names and the description.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

fix_codeA

Evaluate code with the Judges Panel and automatically apply all available auto-fix patches. Returns the fixed code along with a summary of applied and remaining findings. Use this to fix security, performance, and quality issues in a single step.

ParametersJSON Schema

Name	Required	Description
`code`	Yes	The source code to evaluate and fix. Include the full file content.
`config`	No	Optional inline configuration (same format as .judgesrc)
`context`	No	Optional additional context about the code — e.g., what the code does, which framework it uses.
`judgeId`	No	Optional: restrict fixes to a single judge. One of: accessibility, agent-instructions, ai-code-safety, api-contract, api-design, authentication, backwards-compatibility, caching, ci-cd, cloud-readiness, code-structure, compliance, concurrency, configuration-management, cost-effectiveness, cybersecurity, data-security, data-sovereignty, database, dependency-health, documentation, error-handling, ethics-bias, framework-safety, hallucination-detection, iac-security, intent-alignment, internationalization, logging-privacy, logic-review, maintainability, model-fingerprint, multi-turn-coherence, observability, over-engineering, performance, portability, rate-limiting, reliability, scalability, security, software-practices, testing, ux, false-positive-review
`language`	Yes	The programming language of the code (e.g., 'typescript', 'python', 'javascript', 'csharp', 'java').
`minConfidence`	No	Minimum finding confidence to include fixes for (0-1, default: 0.5)

TDQS

A3.5/5.0

Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description bears full responsibility. While it states the tool evaluates and applies fixes, it fails to disclose important behavioral traits: whether it is destructive (modifies input), idempotent, what the 'Judges Panel' entails, error handling, or any side effects. For a mutation tool, this is insufficient transparency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is exceptionally concise (two sentences) and front-loads the core action ('evaluate ... and automatically apply ... patches'). Every sentence is functional with zero redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 6 parameters, full schema coverage, and no output schema, the description provides a high-level overview but lacks details on output format, behavior when no fixes exist, or error conditions. It is adequate but not comprehensive, especially given the absence of annotations.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so baseline is 3. The description adds minimal context beyond schema descriptions (e.g., 'same format as .judgesrc' for config), but does not significantly enhance understanding of parameters. It meets the minimal bar without adding extra value.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states it evaluates code and applies auto-fix patches, returning fixed code and a summary. It uses strong verbs ('evaluate', 'apply', 'returns') and specifies the resource ('code'). It distinguishes itself from sibling evaluation-only tools like 'evaluate_code' by explicitly mentioning automatic fix application and returning fixed code.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description suggests using this tool to fix issues in a single step, but does not explicitly guide when to use alternatives (e.g., 'evaluate_code' for evaluation only, 'evaluate_then_fix' for a two-step process). The usage is implied rather than explicitly stated with when/when-not conditions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_judgesA

List all available judges on the Agent Tribunal panel, including their areas of expertise and what they evaluate.

ParametersJSON Schema

Name	Required	Description	Default
No parameters

TDQS

A4.2/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, but the description accurately conveys a read-only listing operation with no side effects. It does not mention pagination or ordering, but for a 0-parameter tool, the transparency is adequate.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single concise sentence that efficiently conveys the tool's purpose and output. Every word adds value.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool has no parameters and no output schema, the description adequately specifies what is returned (list of judges with expertise and evaluation criteria). However, it lacks details on the response structure or any limits, but for a simple list tool, this is sufficient.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has 0 parameters, so the description does not need to add parameter details. The schema coverage is 100%, and the description reinforces that no parameters are required.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('List all available judges'), the specific resource ('Agent Tribunal panel'), and what information is included ('areas of expertise and what they evaluate'). It distinguishes from sibling tools which are all about evaluation actions, not listing judges.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies when to use this tool (when you need to see available judges), but no explicit guidance on when not to use it or alternatives. Siblings like 'evaluate_code' and 'evaluate_batch' serve different purposes, but no direct comparison is provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_filesA

List files and directories in the workspace. Useful for exploring project structure before evaluating code.

ParametersJSON Schema

Name	Required	Description	Default
`path`	No	Relative directory path to list (default: workspace root)
`depth`	No	Max recursion depth (default: 2, max: 6)

TDQS

A3.8/5.0

Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided. Description only says 'List files and directories.' No mention of hidden files, sorting, permissions, or whether recursion is included beyond schema defaults. Minimal behavioral disclosure.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences: first states action, second provides usage context. No filler words, front-loaded with key information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Sufficient for a simple listing tool with full schema descriptions and a clear use case. No output schema needed. Missing details like output format, but overall adequate.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with descriptions for both parameters (path and depth). Description adds no additional meaning beyond the schema. Baseline score of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clear verb+resource: 'List files and directories'. States a specific use case: 'before evaluating code'. Distinguishes from siblings like read_file (reads content) and evaluate_file (evaluates).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides clear context: 'Useful for exploring project structure before evaluating code.' Does not explicitly state when not to use or alternatives, but the sibling set implies when more specific tools are appropriate.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_triaged_findingsA

List findings that have been triaged (accepted-risk, deferred, wont-fix, false-positive). Optionally filter by triage status. Shows the triage decision, reason, and who made it.

ParametersJSON Schema

Name	Required	Description
`dir`	No	Project directory containing .judges-findings.json (default: current directory)
`format`	No	Output format (default: text)
`status`	No	Filter to a specific triage status

TDQS

A3.6/5.0

Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must cover behavioral traits. It only states what the tool does, not side effects, permissions, or safety profile. Assumed read-only but not explicitly stated.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two concise sentences with no extraneous words. The purpose is front-loaded, and the structure is effective for quick understanding.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no annotations or output schema, the description adequately covers what the tool returns and its filtering capability. However, missing details like pagination, ordering, or output format descriptions.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so parameters are already well-documented. The description adds minimal value by mentioning filtering by triage status, but does not enrich parameter meaning beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool lists triaged findings with optional filtering by status, and specifies it shows the triage decision, reason, and author. It distinguishes itself from sibling tools which are evaluation or analysis tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for retrieving triaged findings with optional filtering, but does not provide guidance on when to use vs alternatives or when not to use. No exclusions or context on prerequisites.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

read_fileA

Read the contents of a file in the workspace. Returns the file text, or an error if the file is too large or missing.

ParametersJSON Schema

Name	Required	Description
`path`	Yes	Relative file path to read
`endLine`	No	Last line to read (1-based, default: end of file)
`startLine`	No	First line to read (1-based, default: 1)

TDQS

A3.8/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must cover behavioral traits. It mentions read operation and errors for large/missing files but does not explicitly state non-destructive nature or other side effects.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences with no fluff. The first sentence states the core action, the second specifies return value and error conditions. Highly concise and front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description covers core functionality and error conditions. While it does not describe return format or line numbering, the schema handles those details. Minor gaps in behavior disclosure.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with clear parameter descriptions. The description adds no additional meaning beyond 'Read the contents of a file'. Baseline score of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses the specific verb 'Read' and resource 'file', clearly stating the tool's function. It distinguishes from sibling tools like 'list_files' which list directory contents.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage when file contents are needed but does not explicitly state when to use or alternatives. No exclusion criteria or context-sensitive guidance is provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

record_feedbackA

Record user feedback on a finding — mark it as a true positive (tp), false positive (fp), or won't fix (wontfix). This feedback calibrates confidence scores in subsequent evaluations during the current session, reducing noise from rules the user considers inaccurate.

ParametersJSON Schema

Name	Required	Description	Default
`ruleId`	Yes	The rule ID of the finding (e.g., 'SEC-001', 'AUTH-003').
`verdict`	Yes	The feedback verdict: tp (true positive), fp (false positive), wontfix (acknowledged but won't fix).

TDQS

A4.2/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Despite no annotations, the description reveals that feedback calibrates confidence scores during the current session. This is sufficient behavioral context for a simple feedback tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two concise sentences cover purpose, use, and effect with no wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity (2 params, no output schema), the description provides adequate context for an agent to use it correctly. Minor omission of return behavior is acceptable.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with descriptions for both parameters. The description adds no additional parameter-specific meaning beyond restating verdict values, so baseline of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool records user feedback on a finding with specific verdicts (tp, fp, wontfix). It distinguishes from sibling tools like evaluate_* and fix_code, which serve different purposes.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explains when to use (to mark findings as tp, fp, or wontfix) and the effect (calibrates confidence scores). It lacks explicit when-not-to-use instructions, but the context is clear.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

re_evaluate_with_contextA

Re-evaluate code with developer-provided context from a multi-turn conversation. Accepts disputed findings, accepted findings, and additional context to adjust the evaluation. This is the agentic feedback loop — the developer explains their intent and the tribunal re-evaluates with that context, applying auto-tune and confidence filtering.

ParametersJSON Schema

Name	Required	Description
`code`	Yes	The source code to re-evaluate
`filePath`	No	File path for context-aware evaluation
`language`	Yes	Programming language (e.g., typescript, python, go)
`deepReview`	No	Whether to include the LLM deep-review prompt section in the result
`focusAreas`	No	Specific areas to focus the re-evaluation on (e.g., ['security', 'performance'])
`relatedFiles`	No	Cross-file context for more accurate evaluation
`maxPromptChars`	No	Maximum character budget for LLM prompts. Controls truncation of source code, related files, and context strings in deep-review prompts. Set to 0 to disable all truncation. Default: 100000.
`acceptedRuleIds`	No	Rule IDs the developer accepts (these will not be filtered)
`disputedRuleIds`	No	Rule IDs the developer disputes as false positives (e.g., ['SEC-001', 'PERF-003'])
`confidenceFilter`	No	Minimum confidence threshold — findings below this are dropped (default: 0.5)
`developerContext`	No	Free-form explanation from the developer about their intent, design decisions, or why certain findings are incorrect

TDQS

A4/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full behavioral burden. It discloses that the tool applies 'auto-tune and confidence filtering' and explains parameters like maxPromptChars (truncation control) and confidenceFilter (threshold for dropping findings). It does not mention side effects or destructive actions, but the behaviors described are accurate and useful.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences, with the first clearly stating the purpose and the second elaborating on the context. It is reasonably concise, though the second sentence is somewhat lengthy. It front-loads the key purpose, which aids agent comprehension.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description does not explain the return value or output format, which would help the agent understand what to expect after invocation. Given the complexity (11 parameters, no output schema), additional detail on result structure would improve completeness. The description covers high-level behavior but leaves output unspecified.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so baseline is 3. The description adds context about the feedback loop but does not provide significant additional semantic detail beyond the parameter descriptions in the schema. Each parameter is adequately described in the schema, so the description's added value is marginal.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: re-evaluating code with developer-provided context from a multi-turn conversation. It specifies the inputs (disputed findings, accepted findings, additional context) and positions it as an 'agentic feedback loop' for adjusting evaluations, distinguishing it from simpler evaluation tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage when the developer wants to provide contextual feedback from a conversation, mentioning disputedRuleIds, acceptedRuleIds, and developerContext. However, it does not explicitly state when to use this tool instead of alternatives like evaluate_code, leaving some ambiguity about the precise trigger for re-evaluation.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

run_benchmarkA

Run the full benchmark suite and return a detailed dashboard with per-judge, per-category, and per-difficulty breakdowns. Includes precision, recall, F1, false positive rates, and individual case results. Use this to understand overall system quality and identify weak spots.

ParametersJSON Schema

Name	Required	Description	Default
`format`	No	Output format: markdown (full report), json (raw data), summary (key metrics only). Default: markdown
`judgeId`	No	Optional: restrict benchmark to a single judge. One of: accessibility, agent-instructions, ai-code-safety, api-contract, api-design, authentication, backwards-compatibility, caching, ci-cd, cloud-readiness, code-structure, compliance, concurrency, configuration-management, cost-effectiveness, cybersecurity, data-security, data-sovereignty, database, dependency-health, documentation, error-handling, ethics-bias, framework-safety, hallucination-detection, iac-security, intent-alignment, internationalization, logging-privacy, logic-review, maintainability, model-fingerprint, multi-turn-coherence, observability, over-engineering, performance, portability, rate-limiting, reliability, scalability, security, software-practices, testing, ux, false-positive-review

TDQS

A4/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must carry full behavioral disclosure. It details the output metrics (precision, recall, F1, etc.) but does not mention potential side effects (e.g., runtime, destructive actions) or authorization needs. Moderate transparency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences with no redundancy. First sentence states action and output details; second sentence states purpose. Extremely concise and efficient.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (full benchmark suite) and no output schema, the description adequately covers output metrics and breakdowns. Missing details like expected runtime or prerequisites, but sufficient for understanding the tool's value.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so baseline is 3. The description adds no additional parameter meaning, but the schema already clearly documents 'format' (enum) and optional 'judgeId' with allowed values.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states verb 'Run' and resource 'full benchmark suite' with specific deliverables (dashboard with per-judge, per-category, per-difficulty breakdowns). Distinguishes from sibling tools like evaluate_code or evaluate_batch by emphasizing comprehensive scope.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly advises using the tool to 'understand overall system quality and identify weak spots', providing clear use context. However, does not mention when not to use it or point to alternatives for more targeted evaluations.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

scaffold_judgeA

Generate the boilerplate files to add a new judge to the Judges Panel. Creates the judge definition (with self-registration), evaluator skeleton, and tells you the one line to add to index.ts. Validates that the judge ID and rule prefix are unique.

ParametersJSON Schema

Name	Required	Description
`id`	Yes	Unique kebab-case judge identifier (e.g., 'supply-chain', 'code-review')
`name`	Yes	Human-readable judge name. Must start with "Judge " (e.g., "Judge Supply Chain")
`domain`	Yes	Expertise area (e.g., "Supply Chain Security")
`dryRun`	No	If true, preview the generated files without writing to disk (default: false)
`rulePrefix`	Yes	Uppercase prefix for rule IDs (e.g., "SCS"). Must be unique across all judges.
`description`	Yes	One-sentence summary of what this judge evaluates
`samplePatterns`	No	Example code patterns the evaluator should detect (strings or regex snippets). Used to seed the evaluator with starter detection logic.
`tableDescription`	Yes	Comma-separated keywords for the README table (e.g., "Dependency provenance, SBOM, build integrity")
`promptDescription`	Yes	Short action phrase for the prompts table (e.g., "Deep supply chain security review")
`evaluationCriteria`	No	List of evaluation criteria / categories to include in the system prompt (e.g., ['Dependency provenance', 'Build integrity'])

TDQS

A3.7/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It mentions validation of ID and rule prefix uniqueness but does not disclose whether file creation is destructive, what happens on validation failure, or if it modifies index.ts directly. Some behavioral traits are missing.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences. First sentence immediately states the tool's main function (generate boilerplate), second adds validation. No wasted words, front-loaded with key action.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Description lacks details on return value, error handling, and the effect of the dryRun parameter. Given no output schema and no annotations, more context about the generation process and its outcomes would be beneficial.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all 10 parameters. The description adds minimal value beyond summarizing the purpose and restating the uniqueness validation. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly states it generates boilerplate files for a new judge, creates definition, evaluator skeleton, and provides the line to add to index.ts. The verb 'scaffold' + resource 'judge' is specific and distinguishes from siblings like scaffold_plugin.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for adding a new judge but does not explicitly state when to use this tool versus alternatives like scaffold_plugin or when not to use it. No comparison with siblings is provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

scaffold_pluginC

Generate a starter plugin template for the Judges Panel. Creates a self-contained plugin file with custom rules, optional custom judges, and lifecycle hooks.

ParametersJSON Schema

Name	Required	Description
`name`	Yes	Unique plugin name (e.g., "my-org-rules", "acme-standards")
`dryRun`	No	Preview output without writing to disk
`version`	No	Semantic version (default: 1.0.0)
`outputPath`	No	File path to write the plugin (default: "judges-plugin.ts")
`rulePrefix`	Yes	Rule ID prefix for this plugin's rules (e.g., "ACME")
`description`	No	Plugin description (e.g., "ACME Corp internal coding standards")
`includeHooks`	No	Include beforeEvaluate/afterEvaluate hook stubs (default: true)
`includeCustomJudge`	No	Include a custom judge definition stub (default: false)

TDQS

C2.9/5.0

Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full burden for behavioral disclosure. It only states that it 'creates a self-contained plugin file', but does not mention whether it overwrites existing files, required permissions, rate limits, or what happens if the file already exists.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise with two sentences. The first sentence immediately states the action and target, followed by a clear elaboration of what the generated file includes. No wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite having 8 parameters and no output schema, the description does not explain the return value (e.g., file path written), error conditions, or side effects. For a tool that interacts with the filesystem, this is insufficient.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

All 8 parameters have descriptions in the input schema (100% coverage). The description adds no additional meaning about parameters, remaining at a general overview level. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool generates a starter plugin template with specific features (custom rules, optional custom judges, lifecycle hooks). It uses a specific verb 'Generate' and identifies the resource. However, it does not explicitly distinguish from the sibling tool 'scaffold_judge', which could cause minor confusion.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives like scaffold_judge. It lacks any context about prerequisites, use cases, or when not to use it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

session_statusA

Get the current evaluation session status — how many evaluations have been run, detected frameworks, verdict history per file, and stability indicators. Useful for understanding what the tribunal has already reviewed.

ParametersJSON Schema

Name	Required	Description	Default
No parameters

TDQS

A4.5/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Although no annotations are provided, the description implies a read-only operation with 'Get' and explains the return content. It does not mention side effects, authorization, or rate limits, but for a status query, the behavioral traits are sufficiently clear.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise: two sentences, no unnecessary words. The first sentence defines what the tool does and returns, the second provides usage context. Every part earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool has no parameters and no output schema, the description fully covers what an agent needs to know: what the tool returns and when to use it. It is complete for a simple status-checking tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has no parameters, so the description does not need to elaborate on them. According to the guidelines, 0 parameters leads to a baseline score of 4.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Get the current evaluation session status' and lists specific information returned (evaluations run, frameworks, verdict history, stability indicators). This verb+resource combination is specific and distinguishes from sibling tools that focus on running evaluations or analyzing code.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description includes a usage hint: 'Useful for understanding what the tribunal has already reviewed.' This provides context for when to use the tool. However, it does not explicitly state when not to use it or mention any alternative tools for similar purposes.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

25 tool updatesv3.129.9Jul 17, 2026
- First observedanalyze_dependencies
- First observedbenchmark_gate
- First observedevaluate_app_builder_flow
- First observedevaluate_batch
- First observedevaluate_code
- First observedevaluate_code_single_judge
- First observedevaluate_code_streaming
- First observedevaluate_diff
- First observedevaluate_file
- First observedevaluate_git_diff
- First observedevaluate_policy_aware
- First observedevaluate_project
- First observedevaluate_public_repo_report
- First observedevaluate_then_fix
- First observedfix_code
- First observedget_judges
- First observedlist_files
- First observedlist_triaged_findings
- First observedre_evaluate_with_context
- First observedread_file
- First observedrecord_feedback
- First observedrun_benchmark
- First observedscaffold_judge
- First observedscaffold_plugin
- First observedsession_status

Install Server

RayrunBuild, deploy, and operate MCP servers without becoming an infrastructure team.ray.runAd

TDQS

A3.5/5.0

Disambiguation4/5

The tools cover distinct functionalities (evaluation, dependency analysis, benchmarking, scaffolding, file ops, feedback), but the many evaluate_* variants (evaluate_code, evaluate_code_streaming, evaluate_diff, etc.) could be confused despite descriptive differences.

Naming Consistency4/5

Most tools follow a verb_noun pattern (e.g., evaluate_code, fix_code, list_files), with 'evaluate_' heavily used. Minor deviations like benchmark_gate and session_status are exceptions.

Tool Count3/5

At 25 tools, the count is on the higher end but still reasonable given the server's broad scope (code evaluation, dependency analysis, benchmarking, scaffolding, etc.). Could be streamlined.

Completeness4/5

The tool surface covers evaluation, fixing, dependency analysis, benchmarking, feedback, and scaffolding comprehensively. Minor gaps: no tool to update triage decisions or evaluate arbitrary specific lines outside diffs.

Maintenance

ActivityInactive

ResponsivenessSyncing

Resources

Need Help?

Related Servers

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

validate A
io.github.machinegrade
Deterministic validation for AI-generated artifacts: JSON Schema, OpenAPI response, SQL syntax.
API Testing Testing & QA Tools
121
Korext: AI Code Governance A
io.github.Korext
Governance copilot for AI-assisted coding. 72 packs, 532 rules, proof bundles.
Code Analysis Developer Tools Autonomous Agents
524
Reality Graph Verification Tools A
dev.realitygraph
Read-only AI coding tools for change verification, release readiness, capacity, and guidance.
Code Analysis Developer Tools Search
102
Frogeye Security Scanner A
io.github.frogeye-ai
Zero-config MCP security scanner for AI-generated apps. 25K+ vulnerability patterns.
Security Penetration Testing
2

Related MCP Servers

MCP Code Crosscheck
Code Analysis Developer Tools Testing & QA Tools
olaservo
A
license
Not graded
quality
Not graded
maintenance
Enables AI-assisted code review with bias mitigation strategies through cross-model evaluation and bias-aware prompting. Detects AI-generated code from commit authors and provides structured reviews with security, performance, and maintainability analysis.
Updated a year ago (2025-06-23 21:49 UTC)
-
CodeBase Optimizer
Code Analysis Developer Tools Autonomous Agents
liadgez
A
license
Not graded
quality
D
maintenance
Provides comprehensive codebase analysis including project structure evaluation, cross-language duplicate detection, microservices validation, and configuration optimization with AI-powered pattern learning that generates actionable improvement reports.
Updated a year ago (2025-07-01 12:51 UTC)
MIT
Argus MCP
Code Analysis Testing & QA Tools Developer Tools
lokafinnsw
A
license
Not graded
quality
C
maintenance
Enables AI-powered, zero-trust code review with multiple models, supporting single files, git diffs, and multiple files, with security, performance, and architecture checks across 10+ languages.
Updated 8 months ago (2025-12-25 10:20 UTC)
13
MIT
CodeGuardian MCP
Code Analysis Developer Tools Testing & QA Tools
codegoddy
A
license
Not graded
quality
D
maintenance
Validates AI-generated code against actual codebases to catch hallucinations, dead code, and API mismatches before runtime.
Updated 6 months ago (2026-02-27 14:29 UTC)
24
1
MIT

Latest Blog Posts

Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)
By Om-Shree-0709 on July 25, 2026.
mcp
Agent Identity
OAuth 2.1
Your AI Chatbot Just Exposed Your CEO's Salary to an Intern
By Om-Shree-0709 on July 2, 2026.
Agent Identity
MCP Security
OAuth Delegation
Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)
By Om-Shree-0709 on June 30, 2026.
Agentic Ai
Prompt Injection
WebAssembly

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/KevinRabun/judges'

If you have feedback or need assistance with the MCP directory API, please join our Discord server