Skip to main content
Glama
arvindcr4

veriloop

by arvindcr4
OverviewSchemaRelated ServersScoreDiscussions
Python

veriloop

Verifier-scored agent runtime. A minimal ReAct loop you can fully inspect: every step is budgeted, scored by a verifier function, and written to a replayable JSONL trace. The tools are exposed once over MCP so they plug into any framework. No LangChain, no LangGraph, no CrewAI — just the primitives, typed.

ci

The problem

Agent demos hide their failure modes. A polished screencast shows the run that worked; it does not show the runs that looped, hallucinated a tool name, sent malformed arguments, or burned forty steps on a two-step task. Framework-first builds make this worse: when the loop belongs to someone else's abstraction, you can't see why a run went wrong, only that it did.

The position of this repo: the first agent worth building is a minimal loop you fully understand. Frameworks earn their place later, for durable stateful orchestration — not as a substitute for knowing what your agent actually does at each step.

veriloop makes every decision inspectable and scoreable:

  • Hard step budget — the loop terminates, provably (there's a test for it).

  • Verified tool calls — args are schema-checked before execution; bad calls are blocked, logged, and fed back so the model can self-correct.

  • A verifier score on every step — verifier functions return scored judgments (reward-function discipline), logged alongside the step, not bolted on after.

  • Full JSONL trace — every run is a replayable, diffable artifact. Traces and scores are first-class outputs, not debug noise.

  • Kill switch — a threading.Event stops the run cleanly from outside.

  • Tools over MCP — the same three-tool registry the loop executes is served over MCP stdio, so any MCP-capable client reuses it unchanged.

Related MCP server: @agentage/mcp-memory

Approach

The loop is the classic ReAct cycle — think → act → observe — kept deliberately small (a few hundred lines of typed Python across loop.py, verifiers.py, trace.py, tools.py, llm.py):

  1. Think — an LLMClient (protocol; any model plugs in — a deterministic FakeLLM ships for tests/CI, an OpenRouterClient with per-call token/cost accounting and a hard cost cap ships for live runs) looks at the task and the step history and emits a Decision: a thought plus exactly one of tool_call / answer.

  2. Verify — pre-act verifiers judge the decision. The built-in SchemaVerifier validates tool args against the tool's pydantic schema; a failing judgment blocks execution.

  3. Act — the tool runs (calculator, sandboxed file_read, offline web_fetch stub), or the blocked call becomes an error observation.

  4. Observe & score — post-act verifiers score the step (the built-in BudgetVerifier scores remaining headroom). Decision + judgments + observation are appended to the JSONL trace as one step record.

  5. Repeat until the model answers, the budget is exhausted, the kill switch fires, or too many consecutive failures trip the fallback stop.

Retry policy is budget-honest: a rejected step consumes a step and its error is fed back as the observation — there are no free retries, so traces never lie about cost.

Verifiers are the extension point: implement the Verifier protocol (a name, a phase, and judge(ctx) -> Judgment) to add task-specific checks, and their scores land in the same trace.

Evaluation

The eval plan is 30 cases; 10 seed cases are committed in eval/cases.seed.jsonl (arithmetic, sandboxed file tasks, and recovery/adversarial cases: malformed args, unknown tools, sandbox escapes, budget traps). The remaining 20 follow the same schema: 10 more multi-step arithmetic/file compositions and 10 more adversarial cases.

Metrics, measured by eval/harness.py (scripted) and eval/run_live.py (live):

Metric

Definition

Scripted (CI)

Live: openai/gpt-4o-mini (2026-07-19)

Task completion rate

runs ending completed

9/10

9/10

Expected-outcome pass rate

all of a case's checks pass

10/10

5/10

Mean steps-to-completion

steps used, completed runs only

2.33

1.89

Mean verifier score

all judgments, all steps

0.883

0.945

Verifier-blocked steps

tool calls blocked pre-execution

2 (scripted by design)

0

Budget-exhaustion rate

runs ending budget_exhausted

1/10 (by design)

0/10

Total cost

from OpenRouter usage accounting

$0

$0.0016 (8,066 in / 666 out tokens)

Honesty note: the live column is one run of the 10-case seed set — single repetition, temperature 0, max 512 tokens/call, ≤6 steps/case, via OpenRouter. The raw artifacts for that exact run — per-case JSONL traces, summary.json with per-case token/cost accounting, and a failure analysis — are committed at eval/results/live-gpt4omini-2026-07-19/. The 5/10 live pass rate is signal, not embarrassment: the misses are adversarial cases where the model behaved reasonably (refused a sandbox-escape without calling the tool, declined an infinite-loop task, never produced the malformed calls the recovery cases script for) — dissected case-by-case in results.md. Cases were not tuned to make the model pass.

Scripted mode exercises the runtime's plumbing deterministically (CI reruns it; output goes to the gitignored eval/results.md); live mode measures model capability, and its numbers come only from committed run artifacts.

Sample trace

Illustrative format example — not output from a recorded run. Generate a real one with uv run python -m veriloop (written to traces/demo.jsonl).

{"type":"run_start","task":"What is 17 * 23?","max_steps":8,"ts":1789700000.01}
{"type":"step","step":0,"decision":{"thought":"Arithmetic; use the calculator.","tool_call":{"tool":"calculator","args":{"expression":"17 * 23"}},"answer":null},"judgments":[{"verifier":"schema","phase":"pre_act","score":1.0,"passed":true,"reason":"args match calculator schema"},{"verifier":"budget","phase":"post_act","score":1.0,"passed":true,"reason":"step 1/8; headroom 1.00"}],"observation":{"ok":true,"content":"391"},"ts":1789700000.02}
{"type":"step","step":1,"decision":{"thought":"The observation has the product.","tool_call":null,"answer":"17 * 23 = 391"},"judgments":[{"verifier":"schema","phase":"pre_act","score":1.0,"passed":true,"reason":"final answer step; no tool call to validate"},{"verifier":"budget","phase":"post_act","score":0.88,"passed":true,"reason":"step 2/8; headroom 0.88"}],"observation":null,"ts":1789700000.03}
{"type":"run_end","status":"completed","answer":"17 * 23 = 391","steps_used":2,"mean_score":0.97,"error":null,"ts":1789700000.03}

Run it

uv run python -m veriloop              # scripted demo; prints the trace it wrote
uv run pytest                          # real tests: budget, verifiers, trace round-trip, tools
uv run ruff check .                    # lint
uv run python eval/harness.py          # scripted eval; writes eval/results.md
OPENROUTER_API_KEY=... uv run python eval/run_live.py   # live eval (gpt-4o-mini); writes eval/results/live-*/

Docker one-liner:

docker build -t veriloop . && docker run --rm veriloop

MCP server

The tool registry is served over MCP stdio:

uv run python -m veriloop.mcp_server

Plug it into any MCP client — e.g. Claude Code:

{
  "mcpServers": {
    "veriloop": {
      "command": "uv",
      "args": ["run", "--directory", "/path/to/veriloop", "python", "-m", "veriloop.mcp_server"]
    }
  }
}

Same tools, same schemas, zero duplication: the loop and the MCP server share one ToolRegistry.

Limitations

  • Scripted CI numbers measure plumbing, not intelligence. FakeLLM replays fixed decision paths. The live column is measured but thin: one model, one run, 10 cases — not a benchmark.

  • Verifiers are heuristic functions, not learned reward models — dense signal, but only as good as the checks you write.

  • web_fetch is an offline stub by default; enabling live fetches without an allowlist is an SSRF risk (flagged in tools.py).

  • Single-threaded, one tool call per step — no parallel tool fan-out, no streaming.

  • Traces are replayable but the loop is not resumable — replay reconstructs what happened; it does not restart a run mid-flight.

  • The sandbox fences file_read only; the calculator and fetch stub have their own guards, but there is no process-level isolation.

Layout

src/veriloop/     loop.py  verifiers.py  trace.py  tools.py  llm.py  mcp_server.py
eval/             cases.seed.jsonl  harness.py  sandbox/
tests/            budget, verifier, trace round-trip, tool-safety tests
docs/             DECISIONS.md
ARCHITECTURE.md   state machine, verifier contract, MCP layering

MIT — see LICENSE.

Available Tools

3 tools
calculatorA

Evaluate a pure-arithmetic expression (+ - * / // % **, parentheses)

ParametersJSON Schema
NameRequiredDescriptionDefault
expressionYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.3/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the burden of behavioral disclosure. It discloses supported operators and parentheses, but does not mention error behavior (e.g., division by zero), precision, or whether the operation is side-effect-free. The word 'pure' hints at a pure function but is not explicit.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single, front-loaded sentence that immediately states the tool's purpose and then provides necessary syntax details. No redundant or filler content.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple calculator tool with a single string parameter and an output schema, this description is sufficiently complete. It defines the input format and scope, and the output schema covers return values, so no additional behavior needs to be explained.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema only defines 'expression' as a string, with 0% coverage of its format. The description compensates by specifying the allowed operators (+ - * / // % **) and parentheses, giving the agent a clear understanding of the expected input syntax.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ('Evaluate') and a resource ('pure-arithmetic expression'), and it distinguishes itself from sibling tools (file_read, web_fetch) by clearly indicating a math evaluation function. The operator list further specifies the scope.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It clearly implies when to use it (for arithmetic calculations) and the 'pure-arithmetic' qualifier excludes non-math uses. However, it lacks explicit 'when not to use' or named alternatives, though the sibling tools are obviously different in purpose.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

file_readA

Read a UTF-8 text file from inside the sandbox directory

ParametersJSON Schema
NameRequiredDescriptionDefault
pathYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are present, so the description carries the full burden. It explicitly states this is a read operation (non-destructive) and restricts use to UTF-8 text files inside the sandbox, which is meaningful disclosure. It lacks details on error handling or path traversal, but these are less critical for a simple read tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, front-loaded sentence with no filler. Every word contributes meaning, making it highly concise and well-structured.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool is simple, and the description covers the essential aspects: purpose, file type, and scope. An output schema exists, so return values are covered. It doesn't mention error cases, but for a basic read tool this is adequate.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has 0% description coverage, but the description compensates by specifying that the path refers to a UTF-8 text file inside the sandbox. This provides crucial context for the 'path' parameter, making its meaning clear beyond just a bare string.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states a specific verb ('Read') and resource ('UTF-8 text file') with a clear scope ('inside the sandbox directory'). It distinguishes the tool from siblings (calculator, web_fetch) by focusing on local file reading.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context that this tool reads files within the sandbox directory, implying it is not for external resources. However, it does not explicitly mention alternatives or when not to use it, so it falls short of a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_fetchA

Fetch a URL over HTTP GET (offline stub by default)

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A3.6/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations present, the description carries the transparency burden. It discloses the HTTP method and the crucial 'offline stub by default' behavior, which is valuable caveat. However, it does not detail response handling, error behavior, or potential side effects, leaving some gaps.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence, front-loaded with the primary action and resource. Every word adds value, including the important 'offline stub' caveat. It is appropriately concise with no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple one-parameter tool with an output schema, the description covers the essential purpose and a key behavioral trait. However, it lacks usage guidance and alternative differentiation, and the absence of annotations increases the need for more contextual details about expected behavior.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema provides no description for the 'url' parameter (coverage 0%), so the description must compensate. It only says 'Fetch a URL,' which essentially restates the parameter name and adds no information about URL format, encoding, or expected constraints.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a clear action ('Fetch a URL') with a specified resource and method ('over HTTP GET'), which distinguishes it from sibling tools like calculator and file_read. The additional caveat 'offline stub by default' adds useful specificity about the tool's behavior.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no guidance on when to use this tool versus alternatives, no prerequisites, and no exclusions or limitations beyond the stub note. It only implies usage for fetching URLs, but does not offer explicit contextual direction.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. 3 tool updatesv0.1.0
    • First observedcalculator
    • First observedfile_read
    • First observedweb_fetch
Install Server
RayrunBuild, deploy, and operate MCP servers without becoming an infrastructure team.ray.runAd

TDQS

A4.1/5.0
Disambiguation5/5

Each tool performs a completely different function: arithmetic evaluation, file reading, and web fetching. There is no overlap, so an agent can easily select the right tool.

Naming Consistency4/5

Two tools follow verb_noun pattern (file_read, web_fetch), but 'calculator' is a noun, which is a minor deviation. Still, the names are clear and predictable overall.

Tool Count5/5

With only 3 tools, the server is well-scoped as a small utility set. Each tool serves a distinct purpose, and the count feels appropriate for the intended lightweight functionality.

Completeness4/5

The tools cover basic arithmetic, file reading, and web fetching, but missing complementary operations like file_write or web_post create minor gaps. These are workable for typical sandbox use cases.

Maintenance

ActivitySlowing
ResponsivenessSyncing

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

  • Server MCP

    SeydinaBANE
    F
    license
    Not graded
    quality
    D
    maintenance
    Provides math and weather tools accessible via LangGraph agent using MCP protocol with stdio and streamable HTTP transports.
    1
    -

  • MCP Demo

    VictorC55
    F
    license
    Not graded
    quality
    C
    maintenance
    A model-agnostic MCP server exposing example tools (add1, multiply2, greet) for learning purposes, working with any LLM through stdio transport.
    -

  • MPC-PoC-1

    mzoffoli85
    F
    license
    A
    quality
    C
    maintenance
    A learning-oriented MCP server that exposes basic tools (echo, add, reverse) over stdio transport to validate MCP handshake and tool calls.
    3
    -

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/arvindcr4/veriloop'

If you have feedback or need assistance with the MCP directory API, please join our Discord server