Skip to main content
Glama
kruslim

Canopy

by kruslim
OverviewSchemaRelated ServersScoreDiscussions
Python
Local

Canopy

An MCP server that exposes vehicle-diagnostic and CAN-bus domain logic as agent tools, with a LangGraph orchestration layer and a human-in-the-loop eval harness.

▶ Live showcase — replay real recorded traces: a cited answer and a grounded refusal, with the honest eval numbers. No install, no API key.

Status: Phase 4 (evals & human-in-the-loop) complete. A LangGraph agent answers natural-language diagnostics questions with validated, cited structured output — or refuses, grounded, when the connected source can't answer. Every consequential output can pass through a human-review interrupt whose structured corrections become permanent regression cases, and an LLM-judge scores traces against the same taxonomy the human uses. The design lives in docs/; the build ships one phase at a time.

Evaluation — the headline number

The LLM-judge agrees with human review on 85% of traces (n=20). Inter-rater reliability could not be measured with a panel — this is a solo project — so the same subset was scored twice, one week apart, reaching 90% self-agreement. The judge should therefore be read as approaching, not exceeding, the reliability ceiling of its ground truth. Agreement is perfect on the mechanically checkable failures — hallucinated_value and absence_as_negation, which a judge can verify against the trace — and every disagreement was an overconfident call (85%), a judgment a rubric only partially disciplines.

That paragraph is the point of Phase 4: a number, measured honestly, with its ceiling and its weak spot stated rather than hidden. That figure is the illustration — hand-seeded labels (a solo project has no panel), reproducible with no API key: uv run python scripts/calibrate.py. The machinery is real; the number is a rehearsal for the one below.

First real pass — the machinery on collected labels (n=8)

The first calibration on real labels — one human review pass over 8 captured traces — reads differently, and the gap is the finding:

50% judge–human agreement (n=8). Agreement is 100% on every failure mode except false_refusal (50%), which alone drags the headline down: the judge waved through four refusals that a human marked as answerable questions wrongly declined. No self-agreement ceiling yet — that needs a second review pass a week apart. Reproduce from recorded labels with uv run python scripts/calibrate.py --real; the report is calibration_report_realpass_a.json.

This is a deliberately pre-fix snapshot: the same over-refusal shows up independently in the regression suite (scripts/eval.py → 3/6, all three failures answerable questions refused). The next moves are an agent fix — each false refusal minted as a from_review regression case and tracked run-over-run (evals/tracking.py) — then review pass B for the ceiling. Publishing the low number before the fix is the honest version of the story.

Why a structured taxonomy, not thumbs-up/down? A thumbs-down says the answer was bad. A structured ErrorType says which of my defenses failed — whether a tool description needs a sentence, a refusal path didn't trigger, or a validator has a gap. The taxonomy is derived from the architecture's known weak points, so every label points at a fix.

Related MCP server: Deep Agent Harness Automation System

Architecture

The central bet: the GenAI layers are independent of the data source. A normalizer sits between the data and the intelligence, so swapping OBD for raw CAN later does not require rewriting the tools, the agent, or the evals.

┌──────────────────────────────────────────────────┐
│ L6  Evals & human-in-the-loop                    │  GenAI   ✅ Phase 4
│ L5  Structured outputs & validation              │  GenAI   ✅ Phase 3
│ L4  Agent orchestration  (LangGraph)             │  GenAI  ✅←core Phase 3
│ L3  MCP server                                   │  GenAI   ✅ Phase 2
│ L2  Tool design & schemas                        │  GenAI   ✅ Phase 1
╞══════════════════════════════════════════════════╡  ← THE SEAM
│ L1b Domain logic (diagnostic rules)              │  expertise  ✅ Phase 0
│ L1a Normalizer  (SignalSample / SignalSeries)    │  contract   ✅ Phase 0
│ L0  Data access:  synthetic | OBD | CAN+DBC      │  plumbing   ✅ synthetic
└──────────────────────────────────────────────────┘

Everything above the seam must be ignorant of whether a number came from an OBD PID or a decoded CAN frame. A seam-enforcement test fails CI if that leaks.

What's built (Phases 0–4)

  • The normalizer contract (model/signals.py) — SignalSample, SignalSeries, SignalSource. A time-ranged read is the general case; an OBD "value now" read is just a series of length one (is_point_read). Units always travel with values.

  • Structured findings (model/findings.py) — Finding with mandatory evidence: a rule that asserts without citing samples is one the agent would launder into a hallucination.

  • The data-access protocol (readers/base.py) — SignalReader with a first-class available_signals(), the mechanism by which the agent will later know what it cannot answer.

  • A deterministic synthetic reader (readers/synthetic.py) — seeded waveforms for a canonical OBD signal set, with injectable known anomalies. The fixture Layers 2–6 are developed and eval'd against; no hardware required.

  • The first diagnostic rule (domain/rules/correlation.py) — coolant rising while engine load is only moderate. Assumes a timeseries; degrades to a low-confidence finding on a point read.

  • Four schema'd tools (tools/) — list_available_signals, summarize_session, get_signal, run_diagnostic_rules. Each is a Pydantic input schema + a description written as a prompt fragment + a handler that returns structured errors (with available_signals and a hint) instead of raising into the agent loop.

  • The MCP server (mcp/server.py) — a thin stdio adapter over the tool layer. Schemas go over the wire as model_json_schema() verbatim; tool errors return isError payloads the model can recover from, while protocol errors stay JSON-RPC errors the model never sees. Reader selection is env-driven (CANOPY_SOURCE, resolved below the seam by readers/factory.py) — the server never learns which source it is serving.

  • The LangGraph agent (agent/graph.py) — agent → tools → validate → refuse as a state graph. Structured output arrives through a submit_answer tool schema; validation failure is a turn (the Pydantic error is fed back with the failing field and a legal escape), capped at two retries, then degrading to a code-built honest answer rather than crashing. The iteration cap converts to a forced-answer degraded turn.

  • The grounded refusal path (agent/contracts.py) — the headline behavior: a question the source cannot answer produces a Refusal naming the missing signal and what is available, filled by code from a tool result, never from model self-knowledge. A cross-validator rejects any answer citing a signal the trace never retrieved — confabulation caught mechanically.

  • The eval harness & HITL (evals/) — a reviewable Trace (full tool-call record, skipped rules, outcome); a review gate built as a LangGraph interrupt whose correct verdicts mint from_review regression cases; a structured feedback taxonomy derived from the architecture's weak points; a regression runner with deterministic fixtures and hard assertions that run in CI; and a calibrated LLM-judge scoring the trace, not just the answer.

Trade-offs (honest)

  • Synthetic-first, not hardware-first. If the architecture required a dongle to test, it would be wrong. Synthetic data is deterministic, so it doubles as the eval fixture. The cost: synthetic waveforms are plausible, not real — real captures arrive in Phase 5.

  • A normalizer up front costs an afternoon. The payoff is that adding raw CAN later touches nothing above the seam. If it turns out to leak, that's recorded honestly in the build log — a diagnosed leak is a better story than a lucky clean run.

  • OBD's coverage ceiling is a feature. OBD cannot see ADAS/camera signals. The right behavior when asked is a grounded refusal, not a guess — that refusal path is a core deliverable, not an edge case.

IP hygiene

No proprietary CAN databases, captures, or signal definitions — ever. Only public OBD-II PIDs, open DBCs, and synthetic/self-captured logs. See data/README.md for the provenance ledger.

Develop

uv sync --extra dev      # create env + install deps
uv run pytest            # run the suite, including the seam test
uv run ruff check .      # lint

Phase 0 done-signal:

from datetime import datetime, timedelta
from canopy.readers.synthetic import SyntheticReader

series = SyntheticReader(seed=42).read(
    "EngineRPM", datetime(2026, 1, 1), datetime(2026, 1, 1) + timedelta(seconds=10)
)
print(len(series.samples), series.unit, series.sample_rate_hz, series.is_point_read)

Phase 2 done-signal — a bare MCP client drives the real server as a subprocess (discovery, invocation, structured errors, clean shutdown; no LLM anywhere):

uv run python scripts/smoke_mcp.py

To explore the same server interactively, register it with any MCP client — e.g. in Claude Desktop's claude_desktop_config.json:

{
  "mcpServers": {
    "canopy": {
      "command": "/absolute/path/to/repo/.venv/bin/python",
      "args": ["-m", "canopy.mcp"],
      "env": { "CANOPY_SOURCE": "synthetic" }
    }
  }
}

The same server, zero code changes, backs the Phase 3 LangGraph agent — that is the decoupling MCP buys.

Phase 3 done-signal — ask the agent a question (needs a provider key in .env; copy .env.example). A grounded refusal on an unanswerable question is a success:

uv run python scripts/ask.py "Is the engine overheating?"
uv run python scripts/ask.py "Did the rear camera activate within 2 seconds?"   # → refusal

Web UI — chat with the agent

A two-pane workspace: a chatbot on the left, and on the right the full tool-call trace, the cited answer (or grounded refusal), and an evidence chart annotated with the exact samples the agent cited. Each question runs the real agent; the answer shape is identical to the recorded traces, so the same chart pipeline draws both.

uv run python scripts/serve.py            # → http://127.0.0.1:8000

  • Live runs need a provider key in .env (copy .env.example); the default provider is Gemini's free tier. Override with --provider anthropic or the CANOPY_PROVIDER / CANOPY_MODEL env vars.

  • No key? It still works. POST /api/ask degrades to replaying the closest recorded trace, so the whole UI is usable offline — the badge just reads replay instead of live.

  • The Simulated scenario selector (Normal / Overheat) chooses the ground-truth condition a live run reads, so you can reproduce the overheat chart on demand. It maps to build_reader(scenario=…) below the seam — the UI never names a data source.

The page is served straight from site/; its data.js is regenerated from the recorded traces with uv run python site/build_data.py.

Phase 4 done-signal — the eval harness. Hard assertions run hermetically in the test suite (no key); the live replay and the judge run against a real model; the calibration number is reproducible from recorded labels with no key:

uv run pytest tests/test_eval_runner.py     # regression suite, scripted model, no key
uv run python scripts/calibrate.py          # the 85% / 90% agreement report
uv run python scripts/eval.py --judge       # live replay + LLM-judge (needs a key)

Available Tools

4 tools
get_signalA

Retrieves one signal over a time range, returned as a timeseries with explicit units and timestamps.

The name must exactly match a name from list_available_signals. Calling this with an unknown name returns a structured error, not an estimate.

Different data sources have very different sample rates. A request-response source may return a SINGLE sample (a 'point read'), with actual_sample_rate_hz set to null. Do NOT perform timing analysis on a point read — check actual_sample_rate_hz before reasoning about how a signal changed over time.

Results are downsampled to max_samples. If truncated is true, the series is a decimation of the full data and fine timing detail may be lost.

Keep the time range bounded to the interval you actually need. An excessively wide window returns a window_too_large error, not data — and because results are downsampled to max_samples anyway, a wider window buys no extra resolution.

ParametersJSON Schema
NameRequiredDescriptionDefault
endYesEnd of the time range, inclusive, ISO 8601.
nameYesCanonical signal name, exactly as returned by list_available_signals. Case-sensitive. Do not guess or abbreviate.
startYesStart of the time range, inclusive, ISO 8601.
max_samplesNoDownsampling cap. The full series is decimated to at most this many evenly-spaced samples. Raise it only when fine timing detail matters; large values consume context without adding insight.

TDQS

A5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Given no annotations, description fully discloses behavior: error conditions (unknown name, window_too_large), return format with units/timestamps, sample rate variability, decimation behavior (downsampled to max_samples, truncated flag), and warning about point reads. No contradictions.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Approximately 12 sentences in short paragraphs, front-loaded with purpose. No unnecessary repetition; every sentence adds value (error handling, sampling details, usage constraints). Efficient despite comprehensive information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite no output schema and 4 parameters, description covers return format, errors, sampling behavior, and constraints. No gaps remain for agent to infer dangerously; complete enough for correct invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% (all parameters have descriptions), but description adds significant context: name must be exact and case-sensitive from list_available_signals, max_samples explanation (downsampling cap, trade-offs), start/end are inclusive ISO 8601. Adds rationale and usage tips beyond schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states verb 'retrieves', resource 'one signal over a time range', and return format 'timeseries with explicit units and timestamps'. Distinguishes from sibling tools (list_available_signals, run_diagnostic_rules, summarize_session) by focusing on signal data retrieval.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides explicit guidelines: name must exactly match list_available_signals, unknown name returns error, sample rate differences, point reads (actual_sample_rate_hz null) should not be used for timing, downsampling behavior, time range bounding to avoid errors. Clearly tells when and how to use the tool.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_available_signalsA

Returns the complete list of signals available from the currently connected data source, with units and typical ranges.

Call this FIRST whenever you are unsure whether a signal exists. Signal availability depends entirely on the data source, so the only reliable way to know what you can answer is to ask.

If the signal a user asks about does not appear in this list, it is NOT available: do not attempt to retrieve it, do not estimate it, and do not substitute a related signal. Tell the user the signal is unavailable and say which source is connected.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

A4.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries full burden. It transparently states that signal availability depends on the data source, and that absence from the list means unavailability. It also mentions the return includes units and typical ranges. It doesn't mention side effects or auth, but for a read-only list operation, this is sufficient and clear.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is three sentences: first states purpose, second gives usage guidance, third sets a rule. It is front-loaded with purpose and contains no extraneous information. Every sentence earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no output schema, the description adequately covers return format (list with units and ranges) and provides behavioral context for missing signals. The tool has no parameters, and the description is complete for its simple scope.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has no parameters and schema coverage is 100%. The description adds value by confirming 'No parameters. Returns everything the current data source exposes,' and it specifies the return content (units and ranges) beyond the schema. Baseline 3 is exceeded due to extra context.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool returns 'the complete list of signals available from the currently connected data source, with units and typical ranges.' It uses a specific verb ('list') and resource ('signals'), and the context distinguishes it from siblings like 'get_signal' which fetches specific signals.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly advises: 'Call this FIRST whenever you are unsure whether a signal exists.' It provides clear rules: if a signal is not in the list, do not attempt to retrieve, estimate, or substitute, and inform the user of unavailability and the connected source. This offers excellent when-to-use and what-to-do guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

run_diagnostic_rulesA

Runs the domain diagnostic rule set over a time range and returns structured findings, each citing the specific data samples that support it.

Every finding includes evidence — the actual samples the rule examined — and a confidence level. A finding with confidence 'low' usually means the rule ran against insufficient data (for example, a timing rule given a single-sample point read). Report low-confidence findings as tentative; never present them as established fact.

Rules requiring signals the current source cannot provide are SKIPPED, not failed. Check skipped before concluding that no problems exist: an empty findings list with a non-empty skipped list means 'we didn't look,' not 'nothing is wrong.'

ParametersJSON Schema
NameRequiredDescriptionDefault
endYes
startYes
rule_idsNoSpecific rules to run. Omit to run all rules applicable to the available signals. Rules whose required signals are unavailable are skipped and reported in `skipped`.

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the burden. It discloses that rules requiring unavailable signals are skipped (not failed), explains confidence levels, and notes that findings include evidence. No side effects are mentioned but the tool likely is read-only.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise with two focused paragraphs. First paragraph states purpose and key features, second provides usage guidance. Every sentence adds value with no redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given 3 parameters and no output schema, the description explains the output structure (findings with evidence and confidence) and behavior (skipped rules). It covers interpretation needs well, though more detail on output fields would be beneficial.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 33% (only rule_ids has a description). The description adds context for rule_ids (skipped behavior) but does not elaborate on start and end parameters beyond the time range concept. Baseline is lowered due to low coverage, but the added value is marginal for the required params.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb ('Runs'), the resource ('domain diagnostic rule set'), and the output ('structured findings, each citing the specific data samples'). This distinguishes it from sibling tools like get_signal or list_available_signals.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit guidance on interpreting low-confidence findings and checking the skipped list before concluding no problems exist. It does not explicitly compare to siblings but offers strong usage context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

summarize_sessionA

Returns a structural overview of a data session: which signals are present, how many samples each has, where there are gaps in coverage, and a count of findings by severity.

Use this BEFORE detailed analysis to understand what data actually exists. This tool returns no interpretation — only structure. It will not tell you what a finding means; call run_diagnostic_rules for that.

coverage_gaps matters: a signal can be 'present' while missing the exact interval a user is asking about.

ParametersJSON Schema
NameRequiredDescriptionDefault
endYes
startYes

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Discloses that it returns only structure, not interpretation, and that coverage gaps matter. No annotations are present, so the description carries the full burden. It could mention that the tool is read-only and safe to call repeatedly, but overall it's transparent.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three concise, front-loaded sentences. Each sentence provides distinct value: output description, usage guidance, and a critical warning. No wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no output schema and no annotations, the description covers purpose, usage, and a behavioral caveat. However, it lacks parameter details and does not hint at the output format or examples. Adequate but leaves some gaps.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has two required parameters (start, end) with no description for how to format them or what range they cover. Schema description coverage is 0%, and the description does not add any semantic meaning for the parameters. This is a significant gap.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states it returns a structural overview of a data session, listing specific elements (signals, samples, gaps, findings by severity). It distinguishes itself from sibling tools like run_diagnostic_rules and list_available_signals.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly advises to use 'BEFORE detailed analysis' and directs to call run_diagnostic_rules for interpretation. Also warns about coverage_gaps being meaningful. Provides clear when-to-use and alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. 4 tool updatesv0.1.0
    • First observedget_signal
    • First observedlist_available_signals
    • First observedrun_diagnostic_rules
    • First observedsummarize_session
Install Server
RayrunBuild, deploy, and operate MCP servers without becoming an infrastructure team.ray.runAd

TDQS

A4.5/5.0
Disambiguation5/5

Each tool has a clearly distinct purpose: listing available signals, retrieving specific signal data, running diagnostic rules, and summarizing session structure. There is no overlap or ambiguity.

Naming Consistency5/5

All tool names follow a consistent verb_noun pattern (list_available_signals, get_signal, run_diagnostic_rules, summarize_session) making them predictable and easy to differentiate.

Tool Count5/5

With 4 tools, the set is well-scoped for the domain of signal analysis and diagnostics. Each tool serves a necessary function without unnecessary bloat or missing essential operations.

Completeness4/5

The tools cover the core workflow: listing, retrieving, analyzing, and summarizing. Minor gaps exist, such as no tool for changing data sources or exporting results, but the core analysis loop is complete.

Maintenance

ActivitySlowing
ResponsivenessSyncing

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/kruslim/canopy'

If you have feedback or need assistance with the MCP directory API, please join our Discord server