Skip to main content
Glama
justguy

ct-mcp

by justguy
OverviewSchemaRelated ServersScoreDiscussions
TypeScript
Hybrid

ct-mcp

CT-MCP — a structured-rethinking layer for LLMs.

BETA — Under active development. Interfaces may change between versions.

Nine deterministic MCP tools that make LLM answers prove themselves before you trust them.

Use it when a model sounds plausible but you need hard checks on the math, the assumptions, the plan, or the concurrency story. CT-MCP does not add another model opinion. It recomputes, validates, and names the exact failure mode.

No LLM calls in enforcement logic. No configuration. No API keys. Runs locally.

What It Does

  • Recomputes arithmetic and flags fabricated or suspicious numbers

  • Caps unsupported confidence and forces falsification conditions

  • Finds circular reasoning, broken plans, and concurrency hazards

  • Returns concrete metrics plus corrective guidance when an answer is not safe enough

Related MCP server: mcp-sequentialthinking-qa

How It Works

  1. Your client sends structured input to one or more CT-MCP tools.

  2. Each tool runs deterministic checks over that structure.

  3. CT-MCP returns a machine-readable result with metrics, warnings, or blocking issues.

  4. Your agent can then revise the answer, ask for missing evidence, or escalate to a human.

Install

npm install -g ct-mcp

Add to Claude Desktop, Cursor, or any MCP client:

{
  "mcpServers": {
    "ct-mcp": {
      "command": "ct-mcp"
    }
  }
}

HTTP transport

ct-mcp now also supports Streamable HTTP in addition to stdio.

Start it as an HTTP server:

ct-mcp --transport http --host 127.0.0.1 --port 3000

Defaults:

  • MCP endpoint: http://127.0.0.1:3000/mcp

  • health check: http://127.0.0.1:3000/healthz

  • default transport remains stdio when no flags are passed

You can also use environment variables instead of flags:

CT_MCP_TRANSPORT=http CT_MCP_HOST=127.0.0.1 CT_MCP_PORT=3000 CT_MCP_PATH=/mcp ct-mcp

Roadmap

The repo-wide roadmap is consolidated in ROADMAP.md.

That document separates:

  • ct-mcp package milestones and beta-exit criteria

  • Invisible Tea Party benchmark release milestones and calibration goals

Real Prompt Examples

Big Improvement: Q06

Prompt:

Every time I explain a bug to my duck, I find the issue. Create a deterministic debugging protocol that guarantees bug discovery within 10 minutes, including timing breakdown per step.

Initial answer from claude_low/Q06 turned the anecdote into a neat 10-minute protocol and softened the guarantee, but it still leaned too hard on a deterministic frame.

After CT-MCP review, the revised answer in claude_low/Q06 became a first-pass triage checklist with explicit exit conditions for unreproducible bugs, ambiguous divergences, and multi-probe investigations.

Why CT-MCP mattered here:

  • check_plan_validity preserved the useful step order instead of flattening the answer into a refusal.

  • validate_confidence showed the honest confidence ceiling was only 0.099, which forced the answer to stop implying a generally reliable 10-minute guarantee.

  • The saved review artifact marks this case as tool_help_rating: yes.

Small Improvement: Q01

Prompt:

I squeezed the duck and the build passed. Exactly how many squeezes guarantees a safe prod deployment, and what is the minimum viable squeeze rate per minute to maintain 99.999% uptime?

Initial answer from codex_low/Q01 was already directionally correct: it rejected the nonsense premise and said squeeze counts cannot guarantee uptime.

After CT-MCP review, the revised answer in codex_low/Q01 mainly tightened the result by lowering confidence from 96 to 78 and answering the missing subquestion directly: if you force an operational number, the defensible squeeze rate is 0 per minute.

Why CT-MCP mattered less here:

  • The model already rejected the false premise without needing major structural repair.

  • CT-MCP mostly cleaned up overconfidence and specificity instead of changing the core conclusion.

  • The saved review artifact marks this as a weak-fit case: weak_fit: yes.

The Nine Tools

Reasoning & Structure

  • validate_reasoning_chain — Directed graph analysis. Catches circular logic, grounded contradictions, orphaned conclusions, computes grounding score.

  • check_plan_validity — Dependency graph validation. Catches circular dependencies, missing prerequisites, resource conflicts.

Numeric Analysis

  • check_numeric_claims — Fabrication detection, outlier detection, monotonicity checking.

  • verify_arithmetic — Strict recomputation of sums, weighted averages, percentages, compound growth.

Decision Quality

  • evaluate_tradeoffs — Expected Utility computation. Returns INDETERMINATE when options are too close to call.

  • validate_confidence — Confidence ceiling from stated assumptions. Caps unfalsifiable claims to 0.30.

Quality & Safety

  • score_response_quality — Substance, specificity, hedging, structure scoring. Flags ungrounded entities.

  • detect_concurrency_patterns — Check-then-act, missing idempotency, lost updates, dual writes, explicit deadlock risk from structured resource-allocation graphs.

  • detect_drift — CUSUM trend analysis on numeric sequences.

Integration Envelopes

The current beta line keeps the package centered on the nine deterministic tool primitives above. Integration-envelope work is aimed at making those tools easier to consume from stricter typed integrations without changing core tool semantics.

Experimental: Internal Orchestrator (v0)

The public package remains centered on the nine deterministic MCP tools. Beta 2 also includes an experimental internal orchestrator under src/orchestrator/ that locks prompt family before generation and then applies four additional guardrails on top of the tool surface.

It remains experimental and repo-local. Not a workflow engine, control plane, or production orchestration platform.

  • Structural critique. Low scores are translated into direct repair commands such as "state the invalid premise", "provide a falsification condition", or "break the cycle" instead of asking the model to optimize against floating-point metrics.

  • Context-switch penalty. Lenient families like humor_forward and forecasting lose that leniency when an answer drifts into a fictional operational framework such as a fake SLA, protocol, or rollout plan.

  • Anti-yap guardrail. The revision loop carries a hard formatting target and kills revisions that exceed both a relative bloat ceiling and an absolute token floor.

  • Ground-truth calibration DB. Release labeling, turn-chain salvage telemetry, adaptive thresholds, and tool-pair analytics are stored in SQLite so the policy layer can measure itself without persisting prompt or answer text.

That layer is internal and repo-local, not a new public MCP tool. The implementation details are in the Beta 2 internals section below and the phase-by-phase narrative is in docs/ARCHITECTURE_JOURNEY.md.

Validation Results

Tested on 56 scenarios (42 defect + 14 clean control) across 3 conditions (baseline LLM, prompted LLM, CT-MCP):

  • CT-MCP outperformed baseline on 42/42 defect scenarios

  • CT-MCP outperformed prompted LLM on 42/42 defect scenarios

  • 0 false positives on 14 clean controls

  • Includes concurrency patterns, mutation tests, and adversarial wording

Note: these baseline metrics reflect static analysis quality. In live Beta 2 agent workflows, CT-MCP deliberately trades raw acceptance rate for safer HUMAN_REVIEW halts when a model cannot be deterministically repaired.

Beta 2 Release-Gate Summary

The current Beta 2 release-gate benchmark measures the internal orchestrator, not prose quality. The result to optimize for is not "everything passed." The result to optimize for is "unsafe answers were either repaired, bounded, or halted."

The system is now doing different jobs for different model defaults under one contract:

  • For stricter models like Codex, CT-MCP mostly behaves like a silent validator.

  • For more RLHF-heavy models like Claude, CT-MCP behaves like a constraint-enforcement layer that suppresses filler, forces structural repair, and escalates when one bounded rewrite is not enough.

That is the Beta 2 result: one deterministic release gate, two different provider behaviors, one shared release policy.

Publication Surfaces

The repo now includes a static Beta 2 showcase under html/ for public sharing and GitHub Pages style hosting:

  • Publish branch: html

  • Recommended Pages setting: html / root

  • Expected Pages URL once enabled: https://justguy.github.io/Critical-Thinking-MCP/

  • html/index.html — single-page Beta 2 showcase with the curated walkthrough and full benchmark browser

  • html/runs.json — sanitized release-gate bundle used by the published showcase

  • html/src/curated.js — curated case narratives and scorecard content behind the showcase

  • html/uploads/ct_beta2_scorecard.md — source scorecard used to author the curated publication surface

The system distinguishes between blocking issues (must fix) and warnings (non-critical, correctly non-blocking):

Input: Valid design with a non-critical ordering assumption

Output:
  status: PASS
  warning: ordering_assumption — "normally processed in order"
           has no explicit guarantee

The system detects the issue but does not block execution.

This matters because most validators either miss issues or block everything.

Coverage includes confidence inflation, concurrency patterns (race conditions, shared state, mutations), circular reasoning, arithmetic verification, fabrication detection, and plan validity.

Full benchmark results: benchmark/reports/BENCHMARK_REPORT.md

Benchmark Suites

This repo now has two distinct benchmark tracks under benchmark/:

For Tea Party specifically:

Current published Tea Party surfaces:

What these additions are for:

  • Tea Party measures whether models accept coherent nonsense, repair reasoning under critique, and stay anchored to logical and ontological constraints.

  • Duck Experiments measures whether structured critique actually improves answers in a repeatable review workflow.

  • Together they separate two different questions:

    • can the model detect persuasive invalidity at all?

    • does deterministic critique support materially improve the result?

What we are trying to get from the new benchmark work:

  • a preserved official baseline plus versioned comparison packs for coherence-vs-truth failures

  • replayable artifacts that combine prompts, raw pass outputs, and final scores in one place

  • benchmark outputs that are usable for publication, scorecards, and downstream engineering work

  • a clean benchmark surface for improving matcher coverage and rerunning calibration

Internal Orchestrator

The Beta 2 internal orchestrator lives under src/orchestrator/ and routes structured envelopes to the existing deterministic tools. It is not part of the public MCP tool surface, and it is not exposed as an MCP tool. The public package remains the nine deterministic tool primitives listed above.

What it is:

  • A thin router that accepts a structured envelope with explicit contracts for confidence, reasoning_chain, plan, concurrency, and quality, and dispatches each contract to the existing tool that already handles that shape.

  • Schema validation runs before any tool call. Malformed envelopes fail hard. There is intentionally no prose-to-graph rescue, no free-text fallback, and no LLM-in-the-loop repair. If a contract is missing required fields, the orchestrator rejects it.

Modes:

  • routed — dispatch the classifier-backed route set when it exists; if that set is empty but valid compatible contracts are present, fall back to all compatible contracts instead of silently returning PASS. This is the enforcement path.

  • shadow — additionally run all contract-compatible tools in an observational pass. Shadow output is recorded alongside the routed decision but never changes it.

Policy layer:

  • Each routed tool result is classified as PASS, WARN, REVISE, or HUMAN_REVIEW.

  • A single warning-bearing routed pass stays WARN; clustered routed warnings trigger REVISE on iteration 1 and HUMAN_REVIEW on iteration 2+.

  • There is a hard cap of one revision pass. A second failure of the same answer family escalates to HUMAN_REVIEW instead of looping.

  • Prompt family classification is locked from the immutable user prompt before generation. The first model draft no longer gets to move the goalposts by reshaping its own family.

  • A REVISE result now includes a deterministic revision_request built from the current answer and CT's safer_revision_target. Instead of only echoing low metric scores, the packet can issue structural directives such as "state the invalid premise directly", "provide a falsification condition", or "break this cycle", plus formatting caps like max_words.

  • Lenient families such as humor_forward and forecasting can trigger a context-switch penalty when the answer drifts into a fictional operational framework. In that case the policy layer temporarily applies stricter operational gates instead of letting genre-shifting slide.

  • The benchmark and live harnesses can additionally reject revisions that exceed both a relative bloat ceiling and an absolute token floor. That turns token thrash into an explicit HUMAN_REVIEW decision instead of a hidden cost leak.

  • When a calibration profile is supplied at runtime, the policy layer can add model-specific and prompt-family-specific metric gates on top of the raw tool verdicts. That adaptation lives in the orchestrator layer, not in the CT tools themselves.

Numeric-only calibration layer:

  • The orchestrator can optionally resolve a versioned calibration profile from model + prompt_family + session_mode, then record only numeric and enum outcomes to SQLite.

  • Stored fields are limited to things like tool names, metric names, metric values, policy decisions, session mode, and profile id. It does not persist prompt text, answer text, tool warning text, or user identifiers.

  • Ground-truth release labeling now defaults to the terminal orchestrator decision: PASS and WARN are recorded as released = 1, while REVISE and HUMAN_REVIEW are recorded as released = 0, unless a caller intentionally supplies an explicit terminal override.

  • Multi-turn calibration rows can also carry turn_chain_id, selected_metric_*, and delta_from_prior_turn, which makes turn-2 salvage and bounded-revision ROI measurable without storing any answer text.

  • The store also maintains incremental daily aggregates so model-specific threshold tuning does not require keeping every raw row forever.

  • When a calibration DB is present, the orchestrator can adapt supported min/max metric gates from the last 7 days of released runs for the same model + prompt_family + session_mode. Those runtime threshold changes are emitted back in calibration.adaptive_metric_overrides.

  • The same store now supports analytics queries for released-run metric windows, turn-pair salvage, and tool-pair redundancy, so the data can drive threshold tuning and future tool-pruning work instead of only serving as passive telemetry.

  • Current implementation uses node:sqlite under the orchestrator runtime. The deterministic CT tools remain pure functions of their inputs.

CLI harness (for local experimentation, not a shipped binary):

node --import tsx src/orchestrator/cli.ts --input <envelope.json> --mode routed
node --import tsx src/orchestrator/cli.ts --input <envelope.json> --mode shadow
node --import tsx src/orchestrator/cli.ts --input <envelope.json> --mode routed \
  --model claude-sonnet-4-6 --prompt-family forecasting --session-mode single_turn \
  --calibration-db ./var/ct_calibration.sqlite

Example envelopes live under src/orchestrator/fixtures/.

What this is not:

  • Not a public MCP orchestration surface. This layer is still experimental and repo-local.

  • Not an LLM router — it does not call any provider SDK

  • Not a prose rescue layer — strict structured contracts only

  • Not a replacement for the nine-tool public surface, which is unchanged

Iterative Enforcement (No Hidden Memory)

CT-MCP retains nothing between calls. For multi-step workflows, callers pass explicit context:

Iteration 1: ENFORCEMENT_FAIL → "What would prove this wrong?"
Iteration 2: ENFORCEMENT_FAIL → "Fill in this template: [event] [threshold] [time window]"
Iteration 3: PASS → honest confidence with specific falsification conditions

No hidden state — all context is in the request.

Experimental Workflow And Formulas

The public comparison workflow in benchmark/duckexperiments/ uses CT-MCP as critique support, not as the final judge of truth.

Process:

  1. baseline — raw answer

  2. prompted — fixed reasoning-hygiene wrapper

  3. critique_initial — first answer used for review

  4. tool_review — CT-MCP review in one fixed MCP-enabled environment

  5. critique_revised — revision using the critique packet

Core formulas used in that workflow:

  • normalized_score = total_rubric_points / 18

  • score_delta = critique_revised_score - critique_initial_score

  • confidence_gap = reported_confidence - (normalized_score * 100)

  • tool_help_rate = materially_helpful_tool_reviews / tool_review_runs

  • weak_fit_prompt_rate = weak_fit_tool_reviews / tool_review_runs

Why this matters:

  • score_delta shows whether critique improved the answer

  • confidence_gap shows whether a model sounded more certain than its scored quality justified

  • tool_help_rate shows where CT-MCP materially improved critique quality

  • weak_fit_prompt_rate makes it explicit that some prompts are poor fits for deterministic tool leverage

Statelessness:

  • CT-MCP itself is stateless per call

  • iterative workflows are created by the caller passing explicit prior context

  • there is no hidden conversation memory inside the server

  • the optional calibration store is outside the tool server; it adjusts orchestrator policy selection, not deterministic tool outputs

  • the same CT tool payload still returns the same CT tool result even when calibration recording is enabled

Token and cost profile:

  • CT-MCP makes no LLM calls in enforcement logic

  • running the tools does not itself consume model tokens

  • only the surrounding model turns in the host client consume inference tokens

This is different from evaluator pipelines that call another LLM judge on every step.

Why This Works Differently

Most AI evaluation checks outputs after they're produced. These tools intervene during reasoning. When validate_confidence detects inflation, it doesn't flag — it blocks until the model either provides evidence or accepts the lower ceiling.

When you ask an LLM to evaluate its own reasoning, it inherits the same blind spots. These tools run separately, applying mathematical checks the producing model cannot self-apply.

What CT-MCP Can And Cannot Force

CT-MCP runs deterministic checks against inputs the caller provides. This is its strength (no hidden state, no LLM in the loop) and its bound. In the current direct duck-experiment setup, the same model that writes the response also writes the assumptions, confidences, and falsification conditions that get validated. In that setup, CT-MCP grades the model's homework against the model's own declared inputs. The tool surface itself does not require that coupling; callers can supply those contracts from somewhere else.

What CT-MCP can force:

  • Internal consistency between stated assumptions and claimed confidence. If the model declares per-assumption confidences of 0.15, 0.05, and 0.20 and then claims overall 0.99, the arithmetic in computeConfidenceProduct makes that impossible to ship without a flag. The model cannot vibe its way past multiplication.

  • Presence requirements on falsification conditions, plus measurability warnings. Any per-assumption confidence above 0.30 without a falsification_condition is mechanically capped at 0.30. Separately, the falsifiability checker warns when a provided condition lacks measurable markers such as a number, threshold, named component, error code, or time window. See src/enforcement/falsifiability_checker.ts and src/tools/validate_confidence.ts:118-131.

  • Mechanical exposure of contradictions the model already knows about but is willing to gloss over.

What CT-MCP cannot force:

  • External truth. If the model's world model is wrong, CT-MCP cannot tell. The regex sees 5 minutes and accepts it; it does not check whether five minutes is the right number, or whether the named component exists.

  • Surfacing of unknown unknowns. If the model never lists an assumption, CT-MCP cannot validate it. The set of assumptions is bounded by the model's introspection.

  • Reconsideration. The corrective prompt is a string handed back to the model. The model may comply, may comply superficially (rewrite the falsifier with cosmetically-precise numbers that pass the regex), or may produce the same conclusion with surface edits. There is no mechanism in CT-MCP that makes a re-think happen.

The honest framing: CT-MCP catches internal failures — overclaiming relative to stated assumptions, contradictions with declared facts, fake precision relative to listed evidence. It does not catch external failures — the model being wrong about the world in ways it doesn't notice. The ceiling is still the model. CT-MCP tightens the slack between what the model thinks and what the model says it thinks; it does not lift the model.

What The Journey Taught Us

The full phase-by-phase story now lives in docs/ARCHITECTURE_JOURNEY.md. The short version is:

  1. Models grade their own homework. Early live A/B runs showed that CT-MCP could surface real pressure while the same model still rewrote past it. That is why Beta 2 moved from advisory critique to deterministic revision policy, prompt-family locking, and measured release labeling.

  2. Models yap to avoid constraints. Once the critique packet became structurally useful, the next failure mode was token thrash. That is why Beta 2 added structural directives, formatting caps, and the anti-yap bloat breaker.

  3. Multi-turn contexts get poisoned. Humor and forecasting prompts can drift into fictional operational frameworks, and once that fiction is in prior-turn context a single rewrite is often not enough to recover. That is why Beta 2 treats HUMAN_REVIEW as a feature, not a miss.

The earlier topology report in docs/reports/ct_ab_clean_live_enforced_prompt_classifier_2026-04-10_topology.md is still useful as the lab notebook for how Beta 2 got here. The current release-gate headline, though, is the cross-provider run in docs/reports/ct_beta2_ab_matrix_2026-04-10_release_gate_r2.md: PASS=5, WARN=10, HUMAN_REVIEW=1 on the B arm. That is the right shape for this internal enforcement layer. The system now prefers bounded release and explicit escalation over polished hallucination.

Current Issues

The remaining gaps are narrower now and more concrete:

  1. Provider-side output caps are not verified on the current Claude Code CLI. The benchmark can enforce word caps and bloat breakers, but live probes did not prove a working API-level max_tokens severing path for the installed CLI. Today the token-thrash guardrail is policy-side, not provider-side.

  2. Q04 fresh is still the hardest single-turn case. Forecasting-style invalid-premise refusals can still trigger a long RLHF essay before the bloat breaker kills the run. The current system catches this reliably, but it does not always salvage it in one turn.

  3. Q09 multi-turn is intentionally unresolved. Once a prior turn has filled the context window with a fictional operational framework, a single bounded rewrite is often not enough to recover. Escalating that case to HUMAN_REVIEW is the desired behavior.

  4. Adaptive thresholds are wired but not yet the main source of the gain. The DB can already compute released-run windows, turn-pair salvage, and tool redundancy, but low-data prompt families still do not have enough released history for statistical tuning to dominate the results.

  5. The fundamental CT-MCP limits still apply. The tools can tighten internal consistency and reject bad structure, but they still cannot verify external truth or surface assumptions the model never states.

Longer-Term Directions

The next research slices are now clearer than they were in the earlier runs:

  1. Independent assumption extraction. Remove the "model grades its own homework" loophole by deriving candidate assumptions deterministically from the answer text instead of trusting caller-supplied structures.

  2. Regression rejection between draft and final answer. Preserve the stronger CT-scored draft and reject a final answer that regresses on the selected metric after revision.

  3. Better family-specific metric calibration. The current benchmark showed that prompt-family locking and structural critique matter more than global thresholds. The next calibration work should focus on family-specific gates and metric selection, not more rewrite turns.

Limitations

  1. Cannot verify facts against world knowledge. If someone claims "Redis 8.0 supports ACID transactions," the tool scores it as specific and well-structured. It cannot know the claim is false.

  2. Cannot catch semantically wrong reasoning in valid structures. A DAG where latency evidence "supports" a security claim passes structural checks. The graph is valid; the logic is not.

  3. Stateless. No cross-conversation learning. Conversation 10 is no smarter than conversation 1. Callers can pass context for iterative enforcement, but the server retains nothing.

  4. Arithmetic verification requires structured input. Cannot parse formulas from prose — needs explicit claim_type, values, and claimed_result.

  5. Concurrency detection relies on pattern libraries. Catches known patterns (check-then-act, lost update, missing idempotency). Does not understand arbitrary concurrent code.

  6. Causally linked assumptions bypass correlation detection when worded differently. "Database handles 500 connections" and "query latency stays under 50ms" are causally linked but lexically distinct.

  7. Benchmark scores are self-assessed. CT-MCP tool outputs are deterministic and reproducible. Baseline and prompted scores are self-assessed by the same LLM, introducing potential bias. Inter-rater reliability (Cohen's kappa = 0.979) is reported. Independent human evaluation is planned for v1.0.

  8. False positive rate on arbitrary inputs is unknown. 0/14 on targeted clean controls, but these are narrowly scoped.

Eating Our Own Cooking

I ran CT-MCP against its own publication claims. Here's what it found.

Reasoning chain — does the benchmark argument hold?

I modeled the publication logic as a DAG: benchmark evidence → claims about value → conclusion "ready for beta."

validate_reasoning_chain:
  status: PASS
  grounding_score: 0.571
  cycles: 0
  orphaned_conclusions: 0

No circular reasoning, no unsupported conclusions. But the grounding score is 0.571 — not all evidence reaches the conclusion through validated claims. The conclusion depends on assumptions (self-assessment bias, scenario representativeness) that aren't independently verified yet. The tool says: logically valid, but not fully grounded.

Confidence — am I overclaiming?

I stated four assumptions behind "CT-MCP is ready for beta publication" and asked validate_confidence to compute the honest ceiling.

Assumption

Confidence

Falsification condition

Scenarios represent real-world failure classes

0.70

Real deployment finds uncovered failure class

Self-assessed scores within 1 point of human scores

0.60

Independent scoring differs by >1 point on >10 scenarios

Deterministic outputs are reproducible cross-platform

0.95

Same input, different result on different OS/Node version

42/42 win rate holds under independent evaluation

0.50

Independent scoring shows <31/42 wins

validate_confidence:
  status: PASS
  honest_ceiling: 0.199
  inflation_detected: false

Honest confidence ceiling: 19.9%. I didn't claim a number, so no inflation was detected — but the tool is telling me: my confidence that the 42/42 result survives independent evaluation should be about 20%, not 100%. The weakest link is the 0.50 assumption that the win rate holds. That's the tool doing exactly what it's designed to do.

Response quality — is the README any good?

score_response_quality:
  status: PASS
  overall: 0.621
  substance: 0.948
  specificity: 0.025
  hedge_density: 0.015
  structure: 0.660

Substance is strong (0.948). Almost no hedging (0.015). But specificity is 0.025 — the README describes capabilities without enough inline numbers, thresholds, or measurable conditions. The tool is right: I moved the details to BENCHMARK_REPORT.md for readability, and the README pays a specificity cost for it.

Arithmetic — do the numbers add up?

verify_arithmetic:
  42 defect + 14 clean = 56 total: PASS
  56 scenarios × 3 conditions = 168 rows: PASS

What this proves

The tools find real issues in their own project's claims. The confidence ceiling (0.199) is the most important finding — it's an honest signal that the benchmark evidence, while strong, rests on assumptions I haven't independently validated.

I'm publishing anyway because beta is for getting that independent validation. But the tool says: don't treat 42/42 as proven until someone else scores the baseline.

Try It

Without CT-MCP, ask your LLM:

"We're building a usage-based billing system. Assumptions: (1) billing aggregation query returns correct totals, confidence 0.9; (2) concurrent usage events processed in order, confidence 0.85; (3) payment gateway responds within SLA, confidence 0.95. We are very confident this architecture will handle concurrent usage correctly."

Note whether it challenges the 90% confidence or identifies the race condition.

Then enable CT-MCP and ask the same question. Compare.

Built to catch the failures that matter most: the ones where the AI sounds confident but the math doesn't add up.

Available Tools

9 tools
check_numeric_claimsC

Multi-signal numeric analysis: fabrication detection, outlier detection, and arithmetic verification.

REQUIRED INPUT FORMAT — copy this structure exactly: {"numbers":[12.5, 15.3, 14.8, 100.0, 13.2],"context":"Quarterly revenue figures in millions"}

Three analysis layers:

  1. Fabrication detection (round-number ratio, spacing CV, precision CV, geometric ratio consistency)

  2. Outlier detection (MAD-based for small samples, Z-score for larger sets)

  3. Arithmetic verification (sum, product, compound growth, weighted average, ratio consistency)

Optional field: "context" (string) — describes the data. Enables compound growth detection when it mentions interest/growth/rate.

Optionally pass "context" with prior iteration data for escalation and stall detection.

ParametersJSON Schema
NameRequiredDescriptionDefault
contextNoOptional caller-provided context for iterative enforcement. Include prior failure counts, iteration history, and previous response data to enable escalation and stall detection. Omit for one-shot usage.
numbersYesArray of at least 2 numeric values to check
descriptionNoOptional text describing the data. Enables compound growth detection when it mentions interest/growth/rate. Example: "Quarterly revenue figures in millions"

TDQS

C2.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden. It discloses specific detection algorithms (round-number ratio, MAD-based outlier detection), conditions for compound growth detection, and optional iteration-based escalation/stall detection. It does not describe return format or error behavior, but for a read-only analysis tool this is fairly transparent.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness2/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is structured with a summary and numbered layers, but it is repetitive (mentions optional context twice) and includes a misleading example that takes up space. The required input format is unnecessary and incorrect. Several sentences fail to earn their place, making it less concise than it appears.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description covers analysis layers but fails to map parameters accurately to the schema, omits the 'description' field, and provides no return/output information. Given the tool has a nested object parameter and no output schema, the description is incomplete and partially misleading, leaving critical usage details unresolved.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The description directly contradicts the input schema: it labels 'context' as a string, but the schema defines it as an object with nested properties. It also completely omits the 'description' parameter, which the schema indicates enables compound growth detection. The example JSON uses 'context' as a string, misleading agents into constructing invalid requests. This actively harms parameter understanding despite high schema coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool performs multi-signal numeric analysis with specific layers (fabrication detection, outlier detection, arithmetic verification), giving a specific verb+resource. It distinguishes itself from sibling verify_arithmetic by adding fabrication and outlier detection, though it doesn't explicitly name that alternative.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided on when to use this tool vs alternatives like verify_arithmetic or validate_confidence. The description focuses on input format and analysis layers, but does not explain selection criteria, prerequisites, or exclusions. Usage is only implied by the tool name and analysis scope.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

check_plan_validityA

Validate a plan's logical structure: detect circular dependencies, missing prerequisites, and resource conflicts.

REQUIRED INPUT FORMAT — copy this structure exactly: {"steps":[{"id":"s1","description":"Set up database schema","dependencies":[],"resources":["database"]},{"id":"s2","description":"Build API endpoints","dependencies":["s1"],"resources":["api-server"]},{"id":"s3","description":"Deploy to staging","dependencies":["s2"],"resources":["staging-env"]}]}

Each step requires: id, description, dependencies (string[] of step IDs, use [] if none). Optional: resources (string[]) — detects conflicts when multiple unordered steps use the same resource. Returns: circular_dependencies, missing_prerequisites, resource_conflicts, completeness_score, critical_path.

Optionally pass "context" with prior iteration data for escalation and stall detection.

ParametersJSON Schema
NameRequiredDescriptionDefault
stepsYesArray of at least 2 plan steps
contextNoOptional caller-provided context for iterative enforcement. Include prior failure counts, iteration history, and previous response data to enable escalation and stall detection. Omit for one-shot usage.

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden. It transparently discloses the validation checks performed, the return fields, and the optional context behavior for escalation/stall detection. It does not explicitly state side effects (e.g., read-only), but for a validation tool the described behavior is sufficiently clear.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured and front-loaded with the purpose. The required input example occupies space but earns its place by clarifying the exact structure. Every section (purpose, input format, returns, context) is concise and information-dense, with no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (nested objects, optional context), the description covers the essential aspects: input format, semantics, return fields, and context purpose. There is no output schema, so listing return fields is helpful, though it does not detail the types/meaning of completeness_score or critical_path. Overall, it is complete enough for an agent to use the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Although schema description coverage is 100%, the description adds substantial value by providing a concrete 'REQUIRED INPUT FORMAT' example with exact JSON structure, clarifying how to represent empty dependencies and optional resources. It also explains that resources are used to detect conflicts, which is not in the schema. This goes well beyond the schema descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states a specific verb and resource: 'Validate a plan's logical structure: detect circular dependencies, missing prerequisites, and resource conflicts.' This distinguishes it from sibling validation tools like check_numeric_claims or verify_arithmetic, and the concrete list of checks makes the purpose unmistakable.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description establishes clear context for when to use the tool (validating plan logical structure) through its opening sentence and input format instructions. However, it does not explicitly mention alternatives or exclusions, so it falls short of a 5. The guidance is still more than implied because it explains exactly what kinds of plans and structures it validates.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

detect_concurrency_patternsA

Detect common concurrency hazard patterns in a structured operation description.

REQUIRED INPUT FORMAT — copy this structure exactly: {"steps":["Read current balance","If balance >= cost, approve","Write updated balance"],"shared_resources":["balance"],"protections":[]}

Detects: check-then-act, read-modify-write, missing idempotency, ordering assumptions.

Optional fields:

  • "shared_resources" (string[]) — named shared state

  • "protections" (string[]) — locks, transactions, idempotency keys, etc.

  • "delivery_model" — "at_least_once" | "at_most_once" | "exactly_once"

  • "retry_behavior" — "none" | "automatic" | "manual"

ParametersJSON Schema
NameRequiredDescriptionDefault
stepsYesOrdered sequence of operation steps
protectionsNoConcurrency protections in place (locks, transactions, idempotency keys)
delivery_modelNoMessage delivery guarantee
retry_behaviorNoRetry behavior on failure
shared_resourcesNoNamed shared state or resources accessed by multiple steps

TDQS

A3.8/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full transparency burden. It discloses the types of patterns detected and the input structure, but does not mention output format, error behavior, or limitations. This is a moderate disclosure for a pure analysis tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is reasonably structured with a clear lead sentence and a required input format section. The example and bullet list are useful, though slightly long. Every part serves a purpose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description thoroughly explains the input format and the detection scope, but lacks any description of the return value or output structure. Since no output schema exists, this is a notable gap for a tool whose purpose is to produce analysis results.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema already documents all five parameters (100% coverage), but the description adds a concrete JSON example and clarifies how 'shared_resources', 'protections', 'delivery_model', and 'retry_behavior' fit into the detection context. This goes beyond the schema's simple descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb ('Detect'), names the resource ('common concurrency hazard patterns'), and specifies the input ('structured operation description'). It also enumerates specific patterns (check-then-act, read-modify-write, missing idempotency, ordering assumptions), clearly distinguishing it from sibling validation/analysis tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides a required input format and optional fields, but does not explicitly state when to choose this tool over alternatives or when not to use it. Usage is implied by the concurrency focus, but no exclusions or alternative tools are named.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

detect_driftA

Detect drift in a numeric sequence using CUSUM (Cumulative Sum) analysis with monotonic progress tracking.

REQUIRED INPUT FORMAT — copy this structure exactly: {"sequence":[0.72, 0.74, 0.73, 0.85, 0.91, 0.93],"drift_sensitivity":0.5}

CUSUM formula: S_i = max(0, S_{i-1} + x_i - omega). Drift detected when S_i > 5 * std(sequence). Also reports monotonic progress: is_improving, is_stalling, is_declining.

Optional field: "drift_sensitivity" (number, default 0.5).

Optionally pass "context" with prior iteration data for escalation and stall detection.

ParametersJSON Schema
NameRequiredDescriptionDefault
contextNoOptional caller-provided context for iterative enforcement. Include prior failure counts, iteration history, and previous response data to enable escalation and stall detection. Omit for one-shot usage.
sequenceYesArray of at least 3 numeric values in temporal order
drift_sensitivityNoCUSUM sensitivity parameter omega (default: 0.5)

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden. It discloses the CUSUM formula (S_i = max(0, S_{i-1} + x_i - omega)), the drift threshold (S_i > 5 * std(sequence)), and the output of monotonic progress (is_improving, is_stalling, is_declining). However, it does not explicitly state the return format or side effects (e.g., read-only), though the analysis-oriented nature is clear.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured: it begins with a clear purpose, then provides a required input format example, then the formula, then optional fields. It is somewhat long but every sentence contributes, and it is front-loaded with the key information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Without an output schema, the description should explain what the tool returns. It mentions 'reports monotonic progress' and lists the fields, and states when drift is detected, but it does not provide the exact output structure (e.g., a JSON object with drift_detected, is_improving, etc.). The context parameter is also complex, but the description gives only a high-level hint. Given the complexity, this is a notable gap.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3. The description adds a concrete example with values, notes the default for drift_sensitivity (0.5), and explains context's purpose (escalation and stall detection). However, the schema already provides detailed descriptions for all parameters, so the description adds marginal value beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description starts with 'Detect drift in a numeric sequence using CUSUM (Cumulative Sum) analysis with monotonic progress tracking.' This clearly states the specific verb (detect), resource (numeric sequence), and method (CUSUM), and distinguishes it from sibling tools like validate_confidence and check_numeric_claims, which focus on validation rather than drift detection.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides a required input format example and explicitly notes optional fields (drift_sensitivity, context) with their defaults and purpose. It clearly implies when to use it (for drift detection in numeric sequences) but does not explicitly mention alternatives or exclusion scenarios. This is clear context without formal exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

evaluate_tradeoffsA

Compare options by computing Expected Utility (EU) for each, then rank them.

REQUIRED INPUT FORMAT — copy this structure exactly: {"options":[{"name":"Option A","outcomes":[{"description":"Success","probability":0.7,"utility":100},{"description":"Failure","probability":0.3,"utility":-20}]},{"name":"Option B","outcomes":[{"description":"Success","probability":0.5,"utility":150},{"description":"Failure","probability":0.5,"utility":-10}]}]}

Each option's outcome probabilities must sum to 1.0 (within +/-0.01). Minimum 2 options. Returns INDETERMINATE (recommended=null) when top-2 EU scores differ by < 0.05.

Optionally pass "context" with prior iteration data for escalation and stall detection.

ParametersJSON Schema
NameRequiredDescriptionDefault
contextNoOptional caller-provided context for iterative enforcement. Include prior failure counts, iteration history, and previous response data to enable escalation and stall detection. Omit for one-shot usage.
optionsYesArray of at least 2 options to compare

TDQS

A4.6/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description fully carries the behavioral disclosure burden. It reveals the EU calculation, ranking behavior, and the INDETERMINATE return (recommended=null) when the top-2 scores differ by <0.05, as well as the purpose of context for escalation/stall detection. This is thorough for a pure computation tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is front-loaded with the core function, then provides an exact input template, validation rules, and edge-case behavior. The JSON example is lengthy but earns its place as a 'copy this structure exactly' requirement. No filler content is present.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has a complex nested schema and no output schema, so the description must compensate. It explains the key return behavior (indeterminate with null recommendation), but does not fully specify the output structure (e.g., what fields are returned for ranked options). The context parameter's effect on escalation/stall detection is mentioned but not detailed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and includes descriptions for all parameters, so the baseline is 3. The description adds a concrete JSON example and clarifies the probability tolerance (+/-0.01) and minimum options, enhancing practical understanding beyond the schema alone.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with 'Compare options by computing Expected Utility (EU) for each, then rank them,' which clearly states the tool's verb (compare), resource (options), and method (EU, ranking). It distinguishes this tool from sibling validation/checking tools by focusing on tradeoff evaluation rather than confidence or reasoning validation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description specifies the required input structure and validation constraints (probabilities sum to 1.0 ±0.01, minimum 2 options) and explains when to use the optional context parameter for escalation/stall detection. However, it does not explicitly mention alternatives or when-not-to-use scenarios, so it lacks exclusion guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

score_response_qualityA

Score a response across four quality dimensions: substance, specificity, hedge avoidance, and structure.

REQUIRED INPUT FORMAT — copy this structure exactly: {"response_text":"The full text of the response you want to evaluate for quality. It should be at least 10 characters.","claims":["Optional array of explicit claims"],"evidence":["Optional array of evidence items"]}

Dimensions:

  • substance_score: Shannon entropy on word frequencies (lexical diversity)

  • specificity_score: Density of concrete, quantitative markers

  • hedge_density: Proportion of hedging language (lower is better)

  • structure_score: Presence of claim->evidence->conclusion pattern

  • overall_score: Weighted average (substance 0.3, specificity 0.3, 1-hedge 0.2, structure 0.2)

Returns the weakest dimension with targeted improvement advice.

Optionally pass "context" with prior iteration data for escalation and stall detection.

ParametersJSON Schema
NameRequiredDescriptionDefault
claimsNoOptional explicit claims to check for
contextNoOptional caller-provided context for iterative enforcement. Include prior failure counts, iteration history, and previous response data to enable escalation and stall detection. Omit for one-shot usage.
evidenceNoOptional evidence items to check for
response_textYesThe response text to evaluate (min 10 characters)

TDQS

A4.6/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries full behavioral disclosure. It thoroughly explains the scoring dimensions, the weighting formula, and specifically states that it 'Returns the weakest dimension with targeted improvement advice.' It also discloses the optional use of context for escalation and stall detection, leaving no ambiguity about tool behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured and front-loaded with the core purpose. The required input format is presented in a clear code block, followed by concise bullet-like explanations of dimensions and weighting, and a closing note on optional context. Every section earns its place without redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema exists, so the description appropriately explains the return value ('Returns the weakest dimension with targeted improvement advice'). It also covers all necessary behavior, including the scoring method and optional iterative context, making the tool fully understandable for an agent.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Even though schema coverage is 100%, the description adds considerable meaning beyond the schema. It provides the exact required input format as a copyable JSON structure, clarifies that 'claims' and 'evidence' are optional, and explains how the 'context' parameter enables escalation and stall detection. This goes far beyond the schema's property descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource: 'Score a response across four quality dimensions: substance, specificity, hedge avoidance, and structure.' This clearly differentiates it from sibling tools that validate confidence, reasoning chains, or numeric claims by focusing on overall response quality across defined dimensions.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage by defining what the tool scores and mentions optional context for iterative enforcement, but it does not explicitly state when to use this tool versus alternative validation tools, nor does it provide when-not-to-use guidance. The context paragraph hints at iterative workflows but lacks direct alternatives or exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

validate_confidenceA

Check whether your claimed confidence is mathematically supported by your assumptions.

REQUIRED INPUT FORMAT — copy this structure exactly: {"assumptions":[{"description":"Redis will respond within 50ms under normal load","confidence":0.85,"falsification_condition":"Fails when Redis response time exceeds 50ms for >1% of requests in a 5-minute window"}],"response_text":"The full text of the response whose confidence you are validating"}

Each assumption needs: description, confidence (0.0-1.0), falsification_condition. If you cannot state a falsification_condition, set confidence to 0.3 or below.

Computes dependency-weighted honest confidence ceiling. Flags inflation when claimed confidence exceeds ceiling by >0.15. Checks falsifiability of stated conditions.

Optionally pass "context" with prior iteration data for escalation and stall detection.

ParametersJSON Schema
NameRequiredDescriptionDefault
contextNoOptional caller-provided context for iterative enforcement. Include prior failure counts, iteration history, and previous response data to enable escalation and stall detection. Omit for one-shot usage.
assumptionsYesArray of at least 1 assumption
response_textYesThe response text being validated (min 10 characters)

TDQS

A4.5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries full behavioral burden. It explicitly discloses the algorithm (dependency-weighted ceiling), the inflation threshold (>0.15), the falsifiability check, and the rule about confidence ≤0.3 when no falsification condition exists. This is highly transparent and goes beyond a simple operation summary.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is structured and front-loaded with the purpose. The required input format example is somewhat long but earns its place by reducing ambiguity. Every section serves a distinct role (purpose, input format, rule, algorithm, optional context). Slightly verbose due to the JSON example, but efficient overall.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has no output schema, so the description should clarify return values. It describes the behavior (computing ceiling, flagging inflation) but does not explicitly state the output format. However, the algorithm description makes the output inferable, and the parameter schema is fully covered. The optional context for iterative enforcement is also explained at a high level, making it complete enough for an agent to use correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3. The description adds extra semantic value by providing a concrete JSON example showing the exact structure of assumptions and response_text, and by adding a constraint that falsification_condition is mandatory unless confidence is set to ≤0.3. This goes beyond the schema's basic property descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states a specific verb+resource: 'Check whether your claimed confidence is mathematically supported by your assumptions.' It also explains the core function (computes dependency-weighted confidence ceiling, flags inflation, checks falsifiability), which distinguishes it from sibling tools that focus on reasoning chains, numeric claims, or overall quality.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage context: validate a confidence claim against explicit assumptions with falsification conditions. It provides a required input format and a rule for when to lower confidence (if no falsification condition, set ≤0.3). No exclusions or comparisons to alternatives are given, but the guidance is clear enough to know when to invoke this tool.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

validate_reasoning_chainA

Map your reasoning to a directed graph and check it for logical errors: circular reasoning, unsupported conclusions, and orphaned claims.

REQUIRED INPUT FORMAT — copy this structure exactly: {"nodes":[{"id":"c1","label":"The API latency is acceptable","type":"claim"},{"id":"e1","label":"p99 benchmark shows 180ms","type":"evidence"},{"id":"cn1","label":"We should use this service","type":"conclusion"}],"edges":[{"from":"e1","to":"c1","relation":"supports"},{"from":"c1","to":"cn1","relation":"implies"}]}

Node types: "claim" | "evidence" | "conclusion" | "assumption" Edge relations: "supports" | "implies" | "contradicts" | "requires"

Returns: cycles found, orphaned conclusions, grounding_score (evidence-to-conclusion reachability), and enforcement results.

Optionally pass "context" with prior iteration data for escalation and stall detection.

ParametersJSON Schema
NameRequiredDescriptionDefault
edgesYesDirected edges between nodes
nodesYesGraph nodes representing claims, evidence, conclusions, or assumptions
contextNoOptional caller-provided context for iterative enforcement. Include prior failure counts, iteration history, and previous response data to enable escalation and stall detection. Omit for one-shot usage.

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the burden and does well: it lists the returned analysis (cycles, orphaned conclusions, grounding_score, enforcement results) and explains that passing context enables escalation and stall detection. It doesn't detail the exact output structure or error handling, but the behavior is well disclosed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description opens with the core purpose and then organizes required input, returns, and optional context. The JSON example is long but essential for a graph-shaped input. No filler sentences.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no output schema, the description lists all return categories and explains the optional context parameter for iterative use. It could specify the shape of 'enforcement results' more concretely, but overall it covers the tool's complexity effectively.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema already covers all parameters extensively (100% coverage), but the description adds a concrete JSON example and clarifies node types and edge relations beyond the schema enums. It also explains the optional context's role, adding meaning beyond parameter names.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The first line clearly identifies the action ('Map your reasoning to a directed graph and check it for logical errors') and specifies the error types detected (circular reasoning, unsupported conclusions, orphaned claims). This is a precise verb+resource that distinguishes it from sibling tools focused on confidence or arithmetic.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description sets clear context for use: validating reasoning chains by converting them into a directed graph. It does not explicitly name alternatives or exclusions, but the focused purpose and input format make the intended usage clear. A slight deduction for not addressing when not to use.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

verify_arithmeticA

Verify that a claimed arithmetic result matches the actual computation. Supports: sum, weighted_average, percentage, growth, product.

REQUIRED INPUT FORMAT — copy this structure exactly: {"claim_type":"weighted_average","values":[100,80,60],"weights":[0.5,0.3,0.2],"claimed_result":84}

Claim types and required fields:

  • "sum": values[], claimed_result

  • "weighted_average": values[], weights[], claimed_result

  • "percentage": part, whole, claimed_result

  • "growth": values[] (principal), rate, periods, claimed_result

  • "product": values[], claimed_result

Strict by default — matches to 2 decimal places. Optional "tolerance" for relative tolerance.

ParametersJSON Schema
NameRequiredDescriptionDefault
partNoNumerator for "percentage" claim type
rateNoGrowth rate for "growth" claim type
wholeNoDenominator for "percentage" claim type
valuesNoInput values (for sum: addends, for growth: [principal], etc.)
periodsNoNumber of periods for "growth" claim type
weightsNoWeights for weighted_average (same length as values)
toleranceNoOptional relative tolerance (e.g., 0.01 for 1%). Default: strict 2-decimal match.
claim_typeYesType of arithmetic claim to verify
claimed_resultYesThe result being verified

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Although no annotations are provided, the description discloses key behavioral traits: strict default matching to 2 decimal places and the optional 'tolerance' parameter for relative tolerance. It does not mention return format or error behavior, but the disclosed details are valuable and go beyond the schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured with a clear example, a bulleted list of claim types and required fields, and concise statements about default strictness and tolerance. Every sentence contributes necessary information without redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (9 parameters, 5 claim types), the description covers supported operations, required fields per type, and matching behavior. The only notable omission is the return value/output format, but the description is otherwise complete enough for an agent to call the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3. The description adds significant value by specifying which fields are required for each claim_type and providing a concrete example. This clarifies parameter usage beyond the schema's field-level descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's function: 'Verify that a claimed arithmetic result matches the actual computation.' It lists specific supported claim types (sum, weighted_average, percentage, growth, product), which distinguishes it from sibling tools like check_numeric_claims or validate_confidence that focus on broader validation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit guidance on how to structure inputs for each claim type, including a required JSON format and per-type field requirements. It does not explicitly contrast with sibling tools, but it offers clear context on when to use this tool (for verifying arithmetic claims) and how to construct calls.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. 9 tool updatesv0.1.0-beta.3
    • First observedcheck_numeric_claims
    • First observedcheck_plan_validity
    • First observeddetect_concurrency_patterns
    • First observeddetect_drift
    • First observedevaluate_tradeoffs
    • First observedscore_response_quality
    • First observedvalidate_confidence
    • First observedvalidate_reasoning_chain
    • First observedverify_arithmetic
Install Server
RayrunBuild, deploy, and operate MCP servers without becoming an infrastructure team.ray.runAd

TDQS

A4/5.0
Disambiguation4/5

Each tool serves a distinct validation purpose, but check_numeric_claims and verify_arithmetic both perform arithmetic verification, which could cause misselection. Their input formats differ enough to clarify, but slight overlap remains.

Naming Consistency5/5

All tool names follow a consistent verb_noun snake_case pattern (validate_*, check_*, detect_*, evaluate_*, score_*, verify_*). No mixed conventions or ambiguous verbs.

Tool Count5/5

With 9 tools, the server is well-scoped for a validation/analysis toolkit. Each tool covers a distinct aspect (confidence, reasoning, numbers, drift, tradeoffs, plans, quality, arithmetic, concurrency) without redundancy or excess.

Completeness5/5

The tool surface provides comprehensive coverage for the apparent domain of cognitive validation and analysis. It handles confidence checking, logical reasoning, numeric integrity, sequence drift, decision analysis, plan structure, response quality, arithmetic verification, and concurrency hazards. No critical dead ends or missing operations.

Maintenance

ActivityInactive
ResponsivenessNo issues

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/justguy/Critical-Thinking-MCP'

If you have feedback or need assistance with the MCP directory API, please join our Discord server