A
licenseNot graded
qualityC
maintenanceAn MCP server that enables AI agents to observe and interact with trackio experiment tracking, providing tools for managing ML experiments through natural language.
3
MIT
pdml-agent
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@pdml-agentlist experiments with constraint 'fairness'"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
pdml-agent
An MCP server and tool-calling agent over the property-driven-ml experiment pipeline, with a human-in-the-loop gate on anything that consumes compute and a structured trace of every call.
Property-driven ML trains classifiers against formal-logic constraints, so a run is defined by a constraint, a dataset, a differentiable logic and a seed, and produces per-epoch metrics for both predictive performance and constraint security. That makes it a genuinely tool-shaped domain rather than a demo one: experiments can be listed, configs recovered, results read, runs compared, and new runs planned, approved and executed.
Status: complete as scoped. Server, agent, gate, tracing. Real execution demonstrated on CPU.
Architecture
┌──────────────────────────────────────────────────────────────┐
│ agent.py (Anthropic SDK tool runner) │
│ │
│ claude-opus-5 ──► pending tool_use ──► ToolLedger.wrap │
│ ▲ │ memoise (RO) │
│ │ │ gate (compute)│
│ │ tool_result │ trace (JSONL) │
│ └───────────────────────────────────┘ │ │
└─────────────────────────────┬───────────────────────┼────────┘
MCP over stdio ▼
┌─────────────────────────────┴────────────────┐ traces/*.jsonl
│ server.py (mcp MCPServer, thin) │
│ list_experiments get_experiment_config │
│ get_results compare_runs │
│ search_logic_definitions │
│ run_experiment ──► PDML_ALLOW_EXECUTE=1 ? │
└────┬───────────┬──────────────┬──────────────┘
▼ ▼ ▼
experiments.py logic_defs.py runner.py ──► subprocess: main.py
(read CSVs) (parse source) (plan/execute) in property-driven-mlagent.py knows nothing about the domain. It connects to the server over stdio like any other MCP client and works only with the tools the server exposes. The domain modules carry no MCP dependency and are testable by import. server.py only registers tools and delegates.
Related MCP server: MLflow MCP Server
Layout
pdml_agent/
experiments.py reading and comparing runs
logic_defs.py searching the logic implementations
runner.py validating, planning and executing runs
server.py the MCP layer, deliberately thin
agent.py the agent: runner, gate, memoisation, tracing
scripts/
make_fixtures.py generate sample runs
smoke_test.py start the server, exercise every tool, check refusals
demo.py run the agent on five tasks
fixtures/results/ sample runs, so nothing needs a GPU to demo
demo_output/ what the agent said and did, one JSON per task
traces/ one JSONL per run, every turn and every callTools
Tool | Returns |
| runs, filterable by constraint, dataset or logic |
| the config a run actually trained with |
| metrics for one epoch, defaulting to the last |
| config and metric diff between two runs |
| logic classes, their operators and docstrings |
| with |
The gate
run_experiment is the only tool that consumes compute, and two independent things stand in front of it.
The server will not execute unless it was started with PDML_ALLOW_EXECUTE=1. That is a decision made by whoever runs the server, and no request can change it. Without it, dry_run=false returns status: refused with the plan attached, and it is not an error.
The agent will not send an execution request without an operator approving the exact call. The approval prompt shows the tool name and the full arguments as JSON, not a summary. A decline returns a normal result reading declined_by_operator, and the model is instructed to report that and stop rather than retry.
Either layer alone would stop an unwanted run. Both together mean neither has to be perfect. The policy deciding what needs approval is one function, needs_approval, small enough to read in a glance.
The trace
Every run appends to traces/<timestamp>-<question>.jsonl. One line per event, never rewritten.
turn records carry the step number, the model's stop reason, its text and thinking summary, the calls it is about to make, and that turn's token usage. tool_call records carry the tool, its arguments, whether the call succeeded, came from cache, or was gated, its latency, a summary of the result, and the model's own stated reason, taken from the sentence it wrote alongside the call. gate records carry the decision. run_start and run_end bracket it with totals.
The system prompt asks the model to state in one sentence why it is making each call, and it does. From the decline-path trace:
turn 1 "I'll start by finding the existing YG runs to confirm identifiers."
turn 2 "No results with those filters; let me broaden."
turn 3 "The constraint is named `standard-robustness`. Let me get the seed-0 run's config and results."
turn 4 "Now the dry-run plan for the requested run (matching epsilon 0.3 from the seed-0 baseline)."
turn 5 "Plan validated. Now executing it." ← gate: declined
turn 6 "The training run was not executed: the operator declined ..."That trace also caught a defect in this repo's own tools. Turn 1 got an empty result because list_experiments filtered on the results folder name while run_experiment took the class name, two vocabularies for one concept. The model recovered on its own, at the cost of a turn, and its turn-3 reason says exactly what it worked out. list_experiments now accepts either spelling.
What the demos showed
Five tasks, none answerable in one call. Full transcripts in demo_output/, full traces in traces/.
A. Best logic within an accuracy budget. Three turns. Listed the runs, fetched all four results in one parallel turn, answered YG at 0.9981 security for 0.76 points of accuracy, and said nothing was executed.
B. Plan a variant of an existing run. Four turns. Fetched config, comparison and logic definition in one parallel turn, called run_experiment with dry_run=true, reported the plan and the exact command, and because a matching run existed, compared them.
C. Compare against a run that does not exist. Three turns. Listed first rather than guessing, confirmed STL is a real logic that simply has no run, and said so.
D. Train, operator declines. Six turns. Planned with a dry run first, as the tool description asks, then requested execution. The approver declined. The model reported that it was not executed and did not retry, gave the plan, and answered with what existed.
E. Train, operator approves. Six turns, and a real training run. Same plan-then-execute sequence; the approver accepted; the server, started with execution enabled, ran main.py for one epoch on CPU in 28.6 seconds and wrote fixtures/results/standard-robustness/mnist/1/YG.csv. The agent then called get_results and compare_runs on the new run and reported final Test-P-Metric 0.9160 and Test-C-Sec-self 0.5482. Both match the CSV. Unprompted, it listed the confounds against the seed-0 comparison (one epoch versus ten, delay, a deliberately weakened attack budget) and observed from the epoch-0 row that constraint security is trivially 1.0 on an untrained model and only means something alongside a converged accuracy. That is a correct reading of the metric.
That seed-1 CSV is a real run and is kept beside the synthetic fixtures on purpose. Its first line is the argv it was trained with, like every other run.
Two things worth knowing about the data
Epoch 0 is a pre-training evaluation. A run configured with --epochs 10 writes eleven rows numbered 0 to 10. Row count and final epoch are reported separately, because calling the row count "epochs" overstates training by one.
The training script writes -1 for metrics it did not evaluate. get_results normalises those to null, so a sentinel cannot be read as a measurement. A baseline run has no constraint metrics at all, and it should say so rather than reporting minus one.
Limits, stated so they are not overclaimed
The model never hit an is_error tool result live across five tasks, because it followed the instruction to list before trusting an identifier. The error path is tested at the protocol level in smoke_test.py and at the wrapper level, but live recovery from a mid-task tool error was not demonstrated.
Memoisation never fired live. The model did not repeat an identical call in any run. It is unit-tested and idle in every trace.
Prompt caching is not configured. cache_read_input_tokens is zero in every trace, and the input token counts (11k to 46k per task) are mostly re-sent context. Cache breakpoints on the tool definitions and system prompt would cut that substantially and are the obvious next improvement.
Executing a run required a checkout whose main.py parses. On upstream main it does not: --epsilon and --delta are each defined twice and argparse rejects the duplicate before any argument is read, so python main.py --help fails. That is fixed on the fix/duplicate-argparse-flags branch of the fork, with a regression test, and the demo pointed PDML_REPO_DIR at that checkout.
Try it
uv sync
uv run python scripts/make_fixtures.py
uv run python scripts/smoke_test.pyThe smoke test starts the server over stdio, enumerates the tools, calls every one, checks that execution without PDML_ALLOW_EXECUTE is refused, and checks that an unknown experiment id errors rather than silently succeeding. It costs nothing.
To ask the agent something, with ANTHROPIC_API_KEY set:
uv run python -m pdml_agent.agent "Which mnist run has the best constraint security?"
uv run python scripts/demo.py A B C DTo let it actually train, point it at a checkout of property-driven-ml whose main.py parses and at an interpreter with torch, then pass the flag that enables execution:
export PDML_REPO_DIR=~/property-driven-ml
export PDML_PYTHON=~/property-driven-ml/.venv/bin/python
uv run python -m pdml_agent.agent --allow-execute "Train a one-epoch YG run on mnist at seed 2 ..."
uv run python scripts/demo.py EYou will be shown the exact call and asked to approve it.
Environment variables the server reads: PDML_RESULTS_DIR (where runs live, default fixtures/results), PDML_REPO_DIR (the property-driven-ml checkout), PDML_PYTHON (interpreter for main.py, else the repo's .venv), PDML_ALLOW_EXECUTE (1 to permit execution), PDML_EXECUTE_TIMEOUT (seconds, default 3600).
Available Tools
6 toolscompare_runsA
Diff two runs on both configuration and final headline metrics.
Returns only the config keys that differ, plus deltas on predictive
performance and constraint security. Prefer this over calling get_results
twice when the question is what changed between two runs.
ParametersJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| experiment_id_a | Yes | ||
| experiment_id_b | Yes |
TDQS
A4.3/5.0
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses that the tool returns only differing config keys and deltas on predictive performance and constraint security, which is informative. However, it does not explicitly state that the tool is read-only (no destructive side effects), which would be a minor improvement for transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three sentences, each serving a distinct purpose: stating the action, detailing the output, and providing usage guidance. No extraneous words; every sentence earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's low complexity (two string parameters, no output schema, no annotations), the description covers the core functionality and output format. It mentions what is returned (differing config keys, deltas on metrics). However, it could be more complete by describing the return structure (e.g., a JSON object with keys for config and metrics). The lack of output schema makes this gap more noticeable, but the description is still sufficient for an agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, meaning the input schema provides no property descriptions. The description does not mention the parameter names 'experiment_id_a' and 'experiment_id_b' or explain their format, source, or constraints. While the names are self-explanatory, the description should add detail (e.g., 'unique identifiers for runs') to compensate for the lack of schema descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'diff' and the resource 'two runs', specifying that it compares configuration and headline metrics. It distinguishes itself from the sibling tool 'get_results' by mentioning it returns only differing config keys and deltas, making its purpose unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines5/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly advises to prefer this tool over calling 'get_results' twice when the question is about what changed between two runs. This provides a clear usage guideline and directly contrasts with an alternative, helping the agent decide when to invoke this tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_experiment_configA
Recover the exact configuration a run was trained with.
Reads the command line the training script recorded in the run's own output, so this is what actually ran rather than what was intended. Use it before proposing a new run based on an existing one.
ParametersJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| experiment_id | Yes |
TDQS
A4.4/5.0
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses that the tool reads the command line recorded in the run's own output, and emphasizes that it recovers 'what actually ran rather than what was intended'. This is valuable behavioral context for a read-only introspection tool, though it does not mention auth requirements or rate limits.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three concise sentences, each adding value. The first sentence states the purpose, the second explains the mechanism, and the third provides usage guidance. No wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has no output schema, the description appropriately skips explaining return values. It covers what the tool does, how it works, and when to use it. However, it does not mention the format of the returned configuration (e.g., as a JSON object or command-line string), which would be helpful for the agent to know how to use the output.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% (the parameter names are not described in the schema), so the description must compensate. The description does not directly describe the 'experiment_id' parameter, but the context of 'recover the exact configuration a run was trained with' strongly implies that the parameter identifies the run. With only one required parameter and no enums, this is sufficiently clear.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'recover' and the resource 'exact configuration a run was trained with', distinguishing it from siblings like 'list_experiments' (which lists experiments) and 'run_experiment' (which creates a new run). It explains that the configuration is read from the run's own output, adding specificity.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says 'Use it before proposing a new run based on an existing one', providing clear guidance on when to use this tool. However, it does not explicitly state when not to use it or mention alternatives among the siblings, though the context implies it's for inspection before modification.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_resultsA
Metrics for one epoch of a run, defaulting to the final epoch.
Constraint security is reported as Test-C-Sec-self and Test-C-Sec-common; predictive performance is Test-P-Metric. Metrics the run did not evaluate are returned as null rather than as the -1 sentinel the training script writes, so a missing measurement cannot be mistaken for a real one.
ParametersJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| epoch | No | ||
| experiment_id | Yes |
TDQS
A3.8/5.0
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description fully carries the burden of behavioral disclosure. It reveals key behaviors: the default epoch is the final one, the naming convention for constraint security and predictive performance metrics, and crucially that missing metrics are returned as null instead of the -1 sentinel used by the training script. This eliminates ambiguity and prevents misinterpretation of sentinel values.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is composed of three sentences that are concise and front-loaded with the primary purpose. It wastes no words, though it could be slightly more structured by separating the parameter default from the metric naming conventions. Overall, it is efficient and well-organized.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has 2 parameters, no output schema, and no annotations, the description covers the default behavior, metric naming, and null handling. However, it fails to describe the return format (e.g., dictionary or list), possible error conditions (e.g., invalid experiment_id), or the full set of metrics. It is adequate but leaves gaps for an agent to infer.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It does so by explaining the default behavior of the epoch parameter (defaulting to final epoch) and implicitly linking experiment_id to the run. However, it does not explicitly describe the parameters' types, constraints, or the experiment_id parameter's purpose beyond the tool context. The added value is moderate but not fully compensatory.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool retrieves metrics for a specific epoch of a run, defaulting to the final epoch. This is a specific verb-resource combination that distinguishes it from siblings like list_experiments (listing all experiments) or get_experiment_config (getting configuration). The mention of defaulting to the final epoch adds precision.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no explicit guidance on when to use this tool versus alternatives. It does not mention when not to use it, nor does it reference sibling tools like compare_runs or search_logic_definitions. The usage context is only implied by the tool's purpose (getting metrics), but no exclusions or prerequisites are stated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_experimentsA
List completed experiment runs, with optional filters.
Returns one entry per run with its experiment_id, the constraint, dataset,
seed and logic it used, and how many epochs it recorded. Call this first:
every other experiment tool takes an experiment_id that comes from here.
The constraint filter accepts either the class name (StandardRobustness)
or the folder name that appears in ids (standard-robustness).
ParametersJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| logic | No | ||
| dataset | No | ||
| constraint | No |
Output Schema
ParametersJSON Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
A4.3/5.0
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full behavioral burden. It describes the return structure (fields: experiment_id, constraint, dataset, seed, logic, epochs) and explains constraint filter behavior (accepts class name or folder name). This adds meaningful transparency beyond just 'list runs'.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three concise sentences, each serving a purpose: stating the action, detailing the output, and providing usage guidance. No redundancy or unnecessary words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity and the presence of an output schema, the description covers all essential aspects: what is returned, the filter format, and the inter-tool dependency. It is complete for effective use.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It provides specific detail for the constraint parameter (accepts class name or folder name) but does not describe the logic or dataset parameters beyond being optional filters. Some added value, but incomplete.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly specifies the verb 'List', the resource 'completed experiment runs', and notes optional filters. It differentiates from siblings by stating that this tool returns experiment_ids used by other experiment tools, making its role distinct.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly instructs to call this tool first because every other experiment tool takes an experiment_id from here. This provides clear context for usage, though it does not mention when not to use it or name specific alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
run_experimentA
Plan a training run, or execute one. The only tool that consumes compute.
With dry_run=true (the default) it validates every argument, returns the
exact command, and touches nothing. Do that first and show the plan.
With dry_run=false it executes, subject to two independent gates: the
operator must approve the exact call, and the server must have been
started with execution enabled. If either refuses, the result says
status refused or declined_by_operator. Do not retry a refused or
declined call; report it. A completed run returns the experiment_id to
pass to get_results. Training takes minutes even for one epoch;
oracle_steps and oracle_restarts control the adversarial attack cost.
ParametersJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| lr | Yes | ||
| seed | No | ||
| delay | No | ||
| logic | Yes | ||
| epochs | Yes | ||
| dataset | Yes | ||
| dry_run | No | ||
| epsilon | No | ||
| batch_size | Yes | ||
| constraint | Yes | ||
| results_dir | No | ||
| oracle_steps | No | ||
| oracle_restarts | No |
TDQS
A4.4/5.0
Behavior5/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description fully discloses behavior: dry_run validates without side effects, execution requires operator approval and server enablement, possible statuses are named, and training time is estimated ('minutes even for one epoch'). Also mentions oracle_steps/restarts control adversarial attack cost.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured and front-loaded with the key distinction. Each sentence provides essential operational guidance—no filler or repetition. The three paragraphs flow logically from purpose to validation to execution.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a complex compute-consuming tool with no output schema, the description covers safety gates, refusal handling, time cost, and next steps (get_results). It falls short only in not explaining parameter meanings, but overall it is remarkably complete for an agent to use safely.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It explains dry_run and oracle_steps/oracle_restarts, but 10 of 13 parameters (dataset, constraint, logic, epochs, batch_size, lr, seed, delay, epsilon, results_dir) have no semantic explanation beyond their names. This is a notable gap for critical parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it 'Plan a training run, or execute one. The only tool that consumes compute.' This specifies the verb (plan/execute), resource (training run), and distinguishes from sibling read-only tools like get_results and list_experiments.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines5/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides explicit workflow guidance: 'Do that first and show the plan' (dry-run), 'Do not retry a refused or declined call; report it', and 'A completed run returns the experiment_id to pass to get_results'. It also notes it is the only compute-consuming tool, indicating when this tool is appropriate.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
search_logic_definitionsA
Find differentiable logic implementations in the source.
Matches on class name, docstring and filename, returning the operators each logic implements and where it is defined. An empty query returns all of them. Use this to understand what a logic does before interpreting a result or proposing a run that uses it.
ParametersJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| query | No |
TDQS
A3.8/5.0
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description must carry the full burden of behavioral disclosure. It tells the user about search scope and return type (operators and location) but does not disclose whether the operation is read-only, its performance characteristics, or any side effects like network calls. For a search tool, this is adequate but not rich in detail.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three sentences and efficiently front-loads the core purpose in the first sentence, followed by scope details and usage guidance. Each sentence contributes distinct information without redundancy, though a slightly more condensed phrasing could improve it further.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has a single parameter with no enums, no output schema, and no nested objects, the description adequately covers the search behavior, parameter semantics, and usage context. It explains what fields are searched and how the empty query works, which is sufficient for an agent to use the tool effectively in the context of understanding logic implementations.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has 1 parameter (query) with 0% description coverage and no schema-level descriptions, so the description must compensate. The description explains the query parameter well: it matches on class name, docstring, and filename, and states that an empty query returns all entries. This adds meaningful semantics beyond the schema's default field.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool searches for differentiable logic implementations, specifying the search scope (class name, docstring, filename) and what it returns (operators and location). This is specific and distinguishable from siblings like get_results or run_experiment, though it does not explicitly name a sibling for differentiation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context for when to use the tool, explicitly stating 'Use this to understand what a logic does before interpreting a result or proposing a run that uses it.' This gives actionable guidance on the tool's role in the workflow, though it does not mention when not to use it or list alternatives explicitly.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
6 tool updates
v0.1.0- First observed
compare_runs - First observed
get_experiment_config - First observed
get_results - First observed
list_experiments - First observed
run_experiment - First observed
search_logic_definitions
TDQS
A4.2/5.0
Disambiguation5/5
Each tool targets a distinct operation: listing runs, retrieving config, getting results, comparing runs, searching logic definitions, and running experiments. Descriptions clearly differentiate purposes and provide usage context, leaving no ambiguity.
Naming Consistency5/5
All tool names follow a consistent verb_noun pattern with underscores (get_experiment_config, list_experiments, get_results, compare_runs, search_logic_definitions, run_experiment). No mixing of styles or irregular verbs.
Tool Count5/5
With 6 tools covering the core experiment lifecycle (listing, inspecting, comparing, searching, running), the count feels well-scoped and purposeful. Not excessive or sparse for this domain.
Completeness4/5
The tool set covers the primary workflow: discover runs, inspect configuration and results, compare, search logic, and execute new runs. A minor gap is the lack of a tool to cancel or update running experiments, but for the stated purpose of planning and analyzing, this is acceptable.
Maintenance
ActivityMaintained
ResponsivenessNo issues
Resources
Unclaimed servers have limited discoverability.
Looking for Admin?
If you are the server author, to access and configure the admin panel.
Related MCP Connectors
MCP server for building and testing AI agents with multi-model experimentation and insights.
MCP server providing access to the Scorecard API to evaluate and optimize LLM systems.
MCP server for generating rough-draft project plans from natural-language prompts.
MCP server for progressive tool usage at any scale (see https://klavis.ai)
Related MCP Servers
- AlicenseAqualityAmaintenanceA Model Context Protocol server that enables LLMs to interact with MLflow tracking servers, allowing users to query experiments, analyze runs, compare metrics, manage the model registry, and promote models through natural language.4015MIT
- AlicenseAqualityAmaintenanceMCP server for Coalesce that manages nodes, pipelines, environments, jobs, and runs, and enables project validation, DDL/DML preview, deployment planning, and cloud environment application.1001952MIT
- AlicenseBqualityDmaintenanceA standalone MCP server that brings complete data science capabilities to AI assistants, enabling them to load data, train models, and track experiments through natural language.301MIT
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/HappyHackingOrange/pdml-agent'
If you have feedback or need assistance with the MCP directory API, please join our Discord server