Skip to main content
Glama

Bankstatemently

Evaluate Benchmark

evaluate_benchmark
Read-only

Score parsed bank statement transactions against the Bankstatemently benchmark ground truth. Accepts a statement_id (e.g. "bsb-001") or content_hash, plus your parsed transactions. Returns extraction accuracy, integrity score, and an overall score. Only statements marked published: true in the catalog can be evaluated — held-out statements return an error. transactions[].originalData is optional but strongly recommended: fetch it via get_statement with data_mode: "original" and pass it through verbatim — an absent originalData scores that transaction's raw-fidelity (parsed) dimension 0; never fabricate a value. Free to use — no credits consumed. Read the benchmark://catalog resource first to see available statements and their published status.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
accountsNoOptional account roster for multi-account statements. Each transaction references one via accountId.
content_hashNoSHA-256 hex digest of the PDF. Use statement_id instead if you know it.
statement_idNoBenchmark statement ID (e.g. "bsb-001"). Preferred over content_hash.
transactionsYesParsed transactions (1-2000)

Output Schema

TableJSON Schema
NameRequiredDescriptionDefault
idYes
challengesYes
difficultyYes
parsedScoreYes
normalizedScoreYes

Schema Changelog

Changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. Changed18 schema fields changed
    • changedInput schema / $schema
      Previous value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"
    • addedInput schema / properties / accounts / description
      Added value: +"Optional account roster for multi-account statements. Each transaction references one via accountId."
    • addedInput schema / properties / accounts / items / properties / accountNumber / description
      Added value: +"Verbatim printed account number — never normalized by the submitter."
    • addedInput schema / properties / accounts / items / properties / currency / description
      Added value: +"ISO 4217 currency code."
    • addedInput schema / properties / accounts / items / properties / id / description
      Added value: +"Submission-internal handle, referenced by transactions[].accountId."
    • addedInput schema / properties / accounts / items / properties / name / description
      Added value: +"Verbatim printed account label."
    • addedInput schema / properties / content_hash / description
      Added value: +"SHA-256 hex digest of the PDF. Use statement_id instead if you know it."
    • addedInput schema / properties / statement_id / description
      Added value: +"Benchmark statement ID (e.g. \"bsb-001\"). Preferred over content_hash."
    • addedInput schema / properties / transactions / description
      Added value: +"Parsed transactions (1-2000)"
    • addedInput schema / properties / transactions / items / properties / accountId / description
      Added value: +"References accounts[].id — the account this transaction belongs to. Omit for single-account statements."
    • addedInput schema / properties / transactions / items / properties / amount / description
      Added value: +"Transaction amount. Negative = debit, positive = credit (or use direction)."
    • addedInput schema / properties / transactions / items / properties / balance / description
      Added value: +"Running balance after this transaction, if known."
    • addedInput schema / properties / transactions / items / properties / currency / description
      Added value: +"ISO 4217 currency code for this transaction, if known."
    • addedInput schema / properties / transactions / items / properties / date / description
      Added value: +"ISO 8601 date (YYYY-MM-DD)"
    • addedInput schema / properties / transactions / items / properties / description / description
      Added value: +"Transaction description as printed on the statement."
    • addedInput schema / properties / transactions / items / properties / direction / description
      Added value: +"Explicit direction. If omitted, inferred from amount sign."
    • addedInput schema / properties / transactions / items / properties / originalData / description
      Added value: +"Raw column values as on the PDF. Omit if unavailable — never fabricate a value; an absence scores the parsed dimension's raw fields 0 rather than polluting the measurement."
    • changedOutput schema / $schema
      Previous value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"
  2. Changed16 schema fields changed
    • removedInput schema / properties / accounts / description
      Removed value: -"Optional account roster for multi-account statements. Each transaction references one via accountId."
    • removedInput schema / properties / accounts / items / properties / accountNumber / description
      Removed value: -"Verbatim printed account number — never normalized by the submitter."
    • removedInput schema / properties / accounts / items / properties / currency / description
      Removed value: -"ISO 4217 currency code."
    • removedInput schema / properties / accounts / items / properties / id / description
      Removed value: -"Submission-internal handle, referenced by transactions[].accountId."
    • removedInput schema / properties / accounts / items / properties / name / description
      Removed value: -"Verbatim printed account label."
    • removedInput schema / properties / content_hash / description
      Removed value: -"SHA-256 hex digest of the PDF. Use statement_id instead if you know it."
    • removedInput schema / properties / statement_id / description
      Removed value: -"Benchmark statement ID (e.g. \"bsb-001\"). Preferred over content_hash."
    • removedInput schema / properties / transactions / description
      Removed value: -"Parsed transactions (1-2000)"
    • removedInput schema / properties / transactions / items / properties / accountId / description
      Removed value: -"References accounts[].id — the account this transaction belongs to. Omit for single-account statements."
    • removedInput schema / properties / transactions / items / properties / amount / description
      Removed value: -"Transaction amount. Negative = debit, positive = credit (or use direction)."
    • removedInput schema / properties / transactions / items / properties / balance / description
      Removed value: -"Running balance after this transaction, if known."
    • removedInput schema / properties / transactions / items / properties / currency / description
      Removed value: -"ISO 4217 currency code for this transaction, if known."
    • removedInput schema / properties / transactions / items / properties / date / description
      Removed value: -"ISO 8601 date (YYYY-MM-DD)"
    • removedInput schema / properties / transactions / items / properties / description / description
      Removed value: -"Transaction description as printed on the statement."
    • removedInput schema / properties / transactions / items / properties / direction / description
      Removed value: -"Explicit direction. If omitted, inferred from amount sign."
    • removedInput schema / properties / transactions / items / properties / originalData / description
      Removed value: -"Raw column values as on the PDF. Omit if unavailable — never fabricate a value; an absence scores the parsed dimension's raw fields 0 rather than polluting the measurement."
  3. First observed

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the readOnly/destructive annotations, it discloses that absent originalData zero-scores the raw-fidelity dimension, warns against fabricating values, notes the published-only restriction, and states that no credits are consumed. This is significant operational behavior an agent needs to know.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is dense but every sentence earns its place: purpose, identifier preference, originalData guidance, scoring consequence, pricing, and prerequisite resource. It is front-loaded with the action and then adds necessary operational context without padding.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a benchmark evaluation tool, it covers inputs, output metrics, prerequisites, error case, optional-data handling, and credit cost. With an output schema present, no return-shape detail is missing. An agent has everything needed to call this correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Even though schema coverage is 100%, the description adds decision-critical semantics: statement_id is preferred over content_hash, originalData is strongly recommended and should be passed through verbatim from get_statement, and omission has scoring consequences. This goes well beyond the schema field names.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb ('Score') and resource ('parsed bank statement transactions against the Bankstatemently benchmark ground truth'), and names the concrete output metrics (extraction accuracy, integrity score, overall score). This clearly separates it from sibling tools like list_statements or rate_statement.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives explicit preconditions — only published:true statements can be evaluated, held-out statements return an error — and directs the agent to read the benchmark://catalog resource first. It also recommends fetching originalData via get_statement with data_mode 'original', but does not explicitly enumerate when to choose this over sibling scoring/evaluation tools.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

TDQS

A4/5.0
Disambiguation4/5

Core tools are cleanly separated by resource: statements, transactions, transfers, credits, and benchmark all have dedicated entry points. The five analytics tools (aggregate, group_by, compare, time_series, top_n) share metric/filter language, but their distinct output shapes and careful descriptions prevent major confusion, with only group_by vs. time_series/top_n occasionally overlapping.

Naming Consistency4/5

Most tools follow a clear snake_case verb_noun pattern such as convert_statement, list_transactions, rate_statement, and dismiss_statement. The analytics tools (aggregate, compare, group_by, time_series, top_n) break that pattern, but they are still recognizable, consistently styled, and form a meaningful cluster.

Tool Count4/5

At 16 tools, this is slightly above the typical 3-15 well-scoped range, but the server covers a broad workflow: upload, conversion, retrieval, categorization, analytics, transfer matching, rating, credits, and benchmarking. Each tool maps to a distinct capability, so the count feels justified rather than bloated.

Completeness4/5

The tool surface covers the full statement lifecycle from upload and conversion through retrieval, categorization, analysis, rating, and dismissal, plus useful side capabilities like credits and benchmark evaluation. Minor gaps exist—no permanent deletion and no way to manually edit category mappings—but dismiss_statement and categorize_statement provide adequate workarounds.