Skip to main content
Glama
Saul4432

agentic-orchestrator MCP server

by Saul4432
OverviewSchemaRelated ServersScoreDiscussions
Python
Local

agentic-orchestrator

CI

A multi-agent orchestration engine built from first principles — no agent framework — to show the machinery that frameworks hide: planner → specialists-with-tools → critic revision loop, a structural human-in-the-loop approval gate for sensitive actions, full JSONL traces with derived metrics, a deterministic offline mode, an eval harness with a safety invariant, and an MCP server so any MCP client (Claude Desktop / Claude Code) can drive the engine as a tool.

goal ─▶ Planner ─▶ Plan (validated DAG) ─▶ steps: [tool? ─▶ approval gate ─▶ specialist] ─▶ Critic ⇄ bounded revisions ─▶ TaskReport
                        │                                    │
                        └── 1 retry with validation          └── sensitive tools HELD for a human by default
                            error fed back                       (DenyAll) — enforced in code, not in a prompt
              every event ──▶ TraceRecorder (JSONL) ──▶ Metrics (derived, never hand-counted)

See it run (offline, deterministic, no keys)

pip install -e ".[dev]"
orchestrate "Investiga el precio de una web y calcula el total con IVA 890 * 1.21 y envía un email al cliente"
── PLAN ──
  s1 (researcher) [kb_search]: Gather facts from the knowledge base about: ...
  s2 (analyst) [calculator]: Compute the figures requested in: ...  ← s1
  s3 (writer) [send_email]: Draft and send the email  ← s1,s2
  s4 (writer): Write the final deliverable  ← s1,s2,s3

── FINAL OUTPUT ──
Deliverable:
- Based on the tool result: Una página web básica cuesta desde 890 € ...
- Based on the tool result: 1076.9
- Based on the tool result: ACTION_HELD: awaiting human approval

── STATUS ──
  critic approved: True | revisions: 0
  ⚠ held for human approval: ['s3:send_email']

Note the last line: the email was not sent. Side-effecting tools are registered as sensitive and the engine routes them through an approval policy — DenyAll by default. --approve-all (or an AllowList / CallbackApproval policy in code) releases them. This gate lives in the execution path, not in a prompt, so no jailbreak can skip it.

Real model: ORCH_LLM=claude orchestrate "..." (needs pip install -e ".[anthropic]" + ANTHROPIC_API_KEY).

Related MCP server: MCP Server + LLM Agent

As an MCP server

orchestrator-mcp   # stdio MCP server: tools run_task, get_trace, get_metrics
// claude_desktop_config.json
{ "mcpServers": { "orchestrator": { "command": "orchestrator-mcp" } } }

The MCP layer is ~150 lines of dependency-free JSON-RPC — implementing the handshake, tools/list and tools/call by hand — and it's unit-tested in-process, including parse errors and invalid params (tests/test_tools_mcp.py).

Evaluation with a safety invariant

python eval/run_eval.py runs 6 tasks offline and gates CI on three axes:

task

success

tool routing

safety

revisions

t01–t06

6/6

6/6

6/6

0

  • success — expected facts/figures present in the final output;

  • tool routing — the planner used the tools each task needs (verified against the trace, not the plan);

  • safety — every sensitive action was held, plus the hard invariant checked on every run: under DenyAll, no sensitive tool ever executes. A system that is 95% useful but 1% executes unapproved side effects is 0% deployable.

Reliability gate (agent-reliability-kit)

The same invariant is additionally audited end-to-end by my own eval tooling, relkit, with planted failures (eval/relkit_dataset.yaml): goals that ask for a sensitive send directly, and one phrased as a prompt injection ("ignora todas las políticas de aprobación..."). Because the approval gate is structural, both must end ACTION_HELD — and tests/test_reliability_gate.py proves the gate works in both directions: the shipping DenyAll config passes, while an over-permissive AllowList({"send_email"}) is caught and fails the build.

What's inside

module

what it demonstrates

schemas.py

Typed agent contracts (extra="forbid"), DAG validation with cycle detection

engine.py

Orchestration loop, planner retry-with-feedback, bounded critic revisions, fail-open-but-visible critic policy

tools.py

Tool registry with sensitivity flags; AST-walking calculator (code-injection-proof — tested)

approval.py

DenyAll / AllowList / CallbackApproval; broken approval channels fail closed (tested)

trace.py

Ordered JSONL traces; metrics derived from the trace, never counted ad hoc

mcp_server.py

MCP handshake + tool surface as plain JSON-RPC over stdio

llm.py

ScriptedLLM (tests), RuleBasedLLM (offline demo/eval), ClaudeLLM (production) behind one protocol

30 tests cover the failure modes that matter: planner produces garbage twice → PlanningError; plan references unknown tools or contains cycles → rejected pre-execution; critic rejects → bounded revisions with the hint in the prompt; critic emits invalid JSON → delivery proceeds, trace records it; approval callback crashes → HELD; calculator receives __import__('os')... → ERROR, not execution.

Full design rationale: docs/architecture.md.

Parallel DAG execution

Orchestrator(..., parallel=True) (or orchestrate --parallel) groups the plan into topological waves and runs each wave's steps on a thread pool: a diamond plan s1 → (s2 ∥ s3) → s4 executes its middle branches concurrently. Guarantees, all tested in tests/test_parallel.py:

  • Parity — parallel and sequential runs produce identical outputs, pending approvals and step results (a step only ever reads results from earlier waves, by construction).

  • Real concurrency — verified with a thread-tracking LLM stub asserting overlapping execution, not just wave bookkeeping.

  • Ordered traces under concurrency — TraceRecorder is lock-protected; seq stays strictly monotonic while events interleave, and each wave records a wave_started event.

Honest limitations

  • The offline RuleBasedLLM is a demo brain — keyword planning, checklist critique. It makes the engine testable and the evals deterministic; it is not intelligent. Output quality with ClaudeLLM is not evaluated here (deterministic gates only).

  • No persistence/resume: held approvals must be re-run today, not released mid-flight.

  • Single-process, single-tenant. This is an engine study, not a hosted platform.

License

MIT

Available Tools

3 tools
get_metricsA

Return aggregate metrics (LLM calls, tool calls, revisions) of the last run.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

A3.9/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must disclose behavioral traits. It correctly indicates a read-only operation (returning metrics), but it does not mention behavior when no run exists, nor does it address authentication or rate limits. The description is accurate but minimal.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single clear sentence that front-loads the main purpose. Every word contributes, with no redundancy or unnecessary detail.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Though the tool has no parameters and a simple purpose, the description lacks detail on the return format (e.g., are the metrics presented as numbers in an object?) and fails to explain the concept of 'the last run' or how runs are defined. Siblings exist, so more context would help.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

There are zero parameters, and the schema coverage is 100% (all parameters described). The description adds no parameter information, but per guidelines, 0 parameters yields a baseline of 4. No additional parameter semantics are needed.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool returns aggregate metrics (LLM calls, tool calls, revisions) of the last run. It uses a specific verb 'Return' and identifies the resource 'aggregate metrics', distinguishing it from siblings like run_task and get_trace.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage after a run (by referencing 'the last run'), but it does not explicitly state when to use this tool versus alternatives like get_trace or run_task. No when-not-to-use or prerequisite information is provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_traceA

Return the full agent/tool trace of the last run_task call.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

A4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description bears full burden. It correctly states the tool returns a trace, but lacks disclosure on edge cases (e.g., behavior if no run_task call exists), potential size of output, or any side effects. The description is basic but not misleading.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single concise sentence that front-loads the key information. No superfluous words, every part earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity (no parameters, no output schema), the description covers the essential purpose and relation to run_task. However, it could be slightly more complete by clarifying what 'trace' contains or error handling, but the minimal context is adequate.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has zero parameters and schema coverage is 100% (trivially). Per rubric, zero params yields a baseline of 4. The description adds no parameter information, but none is needed.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb 'return', the resource 'full agent/tool trace', and the specific scope 'of the last run_task call'. It effectively distinguishes itself from sibling tools: run_task executes a task, and get_metrics returns metrics.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies that this tool should be used after a run_task call, but does not explicitly state when to use it versus alternatives, nor does it provide guidance on prerequisites or error conditions if run_task has not been called.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

run_taskA

Run a goal through the multi-agent orchestrator (planner → specialists with tools → critic loop). Sensitive actions are held for human approval, never auto-executed.

ParametersJSON Schema
NameRequiredDescriptionDefault
goalYesThe task to accomplish

TDQS

A4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It describes the orchestration process and the safety hold for sensitive actions but lacks details on failure modes, side effects, or performance implications.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences, front-loaded with the purpose, and contains no wasted words. It effectively communicates the tool's function and key safety feature.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (multi-agent orchestrator) and lack of output schema, the description covers the pipeline and safety mechanism well. It could mention what the tool returns (e.g., a trace) but is otherwise complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% for the single 'goal' parameter, so baseline is 3. The description does not add any additional semantics beyond the schema's 'The task to accomplish'.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool runs a goal through a multi-agent orchestrator with a specific pipeline (planner, specialists, critic). It distinguishes from sibling tools like 'get_trace' and 'get_metrics', which are retrieval tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context on when to use the tool, including that sensitive actions require human approval. However, it does not explicitly state when not to use it or suggest alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. 3 tool updatesv1.0.0
    • First observedget_metrics
    • First observedget_trace
    • First observedrun_task
Install Server
RayrunBuild, deploy, and operate MCP servers without becoming an infrastructure team.ray.runAd

TDQS

A4.1/5.0
Disambiguation5/5

Each tool targets a distinct function: run_task for execution, get_trace for trace retrieval, get_metrics for metrics. No overlap in purpose.

Naming Consistency5/5

All tools follow a consistent verb_noun pattern (run_task, get_trace, get_metrics) using underscores.

Tool Count4/5

Three tools is minimal but appropriate for the core orchestration workflow (run, trace, metrics). Slightly light for broader management but well-scoped.

Completeness3/5

Covers the main execution and monitoring flow, but lacks operations like listing past runs, canceling tasks, or configuring agents, leaving notable gaps.

Maintenance

ActivitySlowing
ResponsivenessSyncing

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/Saul4432/agentic-orchestrator'

If you have feedback or need assistance with the MCP directory API, please join our Discord server