What can you do with this server?

The llmprobe server lets you probe and monitor LLM API endpoints to measure performance metrics. * Run a full health check (probe): Test all LLM endpoints defined in a probes.yml config file (or a custom path), returning TTFT (time to first token), total latency, throughput (tokens/sec), and health status (healthy/degraded/error) for every configured model. * Check a single model (check_model): Probe a specific model on-demand without a config file by specifying the provider (openai, anthropic, google, azure, bedrock, or OpenAI-compatible APIs like Groq/Together), the model identifier (e.g. gpt-4o, claude-sonnet-4-20250514), and the environment variable holding the API key — returning the same metrics. Results can also be used to gate CI/CD deployments based on provider health.

Which integrations are available for this server?

Probes Google models via the streamGenerateContent API, measuring TTFT, latency, tokens/sec, with configurable thresholds. Probes Ollama models (OpenAI-compatible) via local endpoint, measuring TTFT, latency, tokens/sec, with configurable thresholds. Probes OpenAI models like gpt-4o and gpt-4o-mini, measuring TTFT, latency, tokens/sec, with configurable thresholds.

de en es ja ko ru zh

llmprobe

by Jwrede

Overview Schema Related Servers Score Discussions

Local

llmprobe

Synthetic monitoring and CI smoke tests for LLM inference endpoints. Measure TTFT, latency, throughput, and errors. Single binary, zero SDKs.

License: MIT llmprobe MCP server

llmprobe is a CLI tool for LLM serving reliability. It probes hosted APIs or OpenAI-compatible inference servers, then reports the metrics that matter for production user experience: time to first token (TTFT), total latency, generation throughput (tokens/sec), and error rates.

Use it as a one-off health check, a continuous monitor, or a CI gate that blocks deploys when your LLM provider is degraded.

demo

Public benchmark

llm-bench uses llmprobe to run a continuous public benchmark of major LLM APIs. It publishes a live dashboard at bench.jonathanwrede.de and raw JSONL data in Jwrede/llm-bench-data.

This is the intended use case: repeated synthetic probes that make LLM latency, TTFT regressions, throughput drops, and provider degradation visible before users report them.

Related MCP server: LLM API Benchmark MCP Server

Install

Download a prebuilt binary from the latest release (Linux, macOS, Windows; amd64 and arm64).

Or install from source:

go install github.com/Jwrede/llmprobe@latest
llmprobe version

Claude Code plugin

Install as a Claude Code plugin for /llmprobe skill and MCP tools:

claude plugin install Jwrede/llmprobe

Or register the MCP server directly:

claude mcp add --transport stdio llmprobe -- llmprobe mcp

llmprobe runs locally and only contacts LLM endpoints you configure. See PRIVACY.md for details.

Quick start

llmprobe works with OpenAI, Anthropic, Google, Azure OpenAI, AWS Bedrock, and OpenAI-compatible endpoints such as vLLM, Ollama, OpenRouter, Groq, Together AI, Fireworks, DeepSeek, and Mistral.

Create a probes.yml (or copy the included example):

providers:
  - name: openai
    api_key: ${OPENAI_API_KEY}
    models:
      - name: gpt-4o
        thresholds:
          max_ttft: 2s
      - name: gpt-4o-mini
        thresholds:
          max_ttft: 500ms

  - name: anthropic
    api_key: ${ANTHROPIC_API_KEY}
    models:
      - name: claude-sonnet-4-20250514
        thresholds:
          max_ttft: 1s

Run a probe:

$ llmprobe probe

Provider   Model                    Status    TTFT    Latency  Tok/s  Tokens  Error
--------   -----                    ------    ----    -------  -----  ------  -----
openai     gpt-4o                   healthy   312ms   2100ms   68.4   42
openai     gpt-4o-mini              healthy   98ms    814ms    112.3  56
anthropic  claude-sonnet-4-20250514 healthy   420ms   2831ms   52.1   38
azure      gpt-4o                   healthy   289ms   1950ms   71.2   44
bedrock    anthropic.claude-3-5...  degraded  1820ms  4510ms   28.1   38

4 healthy, 1 degraded, 0 errors

What it measures

Metric	What it means
TTFT	Time from request send to first content token. This is what users feel as "lag" before the response starts streaming.
Latency	Total time from request to stream close.
Tok/s	Generation throughput: tokens produced per second after the first token. Calculated as `token_count / (latency - ttft)`.
Tokens	Total output tokens. Prefers provider usage metadata when available, falls back to SSE event counting.
Status	`healthy` if all thresholds pass, `degraded` if any threshold is exceeded, `error` if the request failed.

Commands

`llmprobe probe`

One-off health check. Probes all configured endpoints and prints results.

llmprobe probe                        # table output
llmprobe probe -f json                # JSON output
llmprobe probe --fail-on degraded     # exit 1 if any endpoint is degraded
llmprobe probe -c custom-config.yml   # custom config path

Exit codes for CI:

`--fail-on`	Exit 0	Exit 1
`error` (default)	healthy or degraded	any error
`degraded`	healthy only	degraded or error
`none`	always	never

`llmprobe watch`

Continuous monitoring. Probes all endpoints on an interval and prints a summary line per iteration.

llmprobe watch                          # default 60s interval
llmprobe watch --interval 30s           # custom interval
llmprobe watch --tui                    # live terminal dashboard with TTFT chart
llmprobe watch --tui --load data.jsonl  # load historical data into the dashboard
llmprobe watch -f json                  # JSONL output (one line per result)
llmprobe watch --prometheus :9090       # expose Prometheus metrics
llmprobe watch --otel localhost:4317     # export OpenTelemetry metrics via OTLP/gRPC

The --tui flag launches a live terminal dashboard with a TTFT chart, color legend, and statistics table. Use --load to import historical JSONL data (from llmprobe watch -f json > data.jsonl).

llmprobe

$ llmprobe watch --interval 30s

Watching 4 endpoints every 30s (Ctrl+C to stop)

[14:01:02] All 4 endpoints healthy.
[14:01:32] All 4 endpoints healthy.
[14:02:02] 3 healthy, 1 degraded, 0 errors. DEGRADED: openai/gpt-4o (TTFT 1820ms)
[14:02:32] All 4 endpoints healthy.

`llmprobe report`

Generate a Markdown summary from JSONL probe data with p50/p95/p99 percentiles for TTFT, latency, and throughput per endpoint.

llmprobe report data.jsonl

Output:

| Provider | Model | Probes | Errors | TTFT p50 | TTFT p95 | ... | Tok/s p50 | ...
|----------|-------|--------|--------|----------|----------|-----|-----------|----
| openai   | gpt-4o | 100  | 2      | 115ms    | 188ms    | ... | 46.9      | ...

`llmprobe baseline`

Create a baseline file from historical JSONL data for regression detection.

llmprobe baseline data.jsonl -o baseline.json

Reference the baseline in your config to use multiplier-based thresholds:

baseline: baseline.json

providers:
  - name: openai
    api_key: ${OPENAI_API_KEY}
    models:
      - name: gpt-4o
        thresholds:
          max_ttft_multiplier: 2.0       # fail if TTFT > 2x baseline p50
          max_latency_multiplier: 2.5    # fail if latency > 2.5x baseline p50

This lets you detect regressions relative to your own historical data rather than setting absolute thresholds.

`llmprobe version`

Print the installed binary version.

llmprobe version

CI integration

Use llmprobe probe as a pre-deploy gate:

# .github/workflows/deploy.yml
- name: Check LLM providers
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
  run: |
    go install github.com/Jwrede/llmprobe@latest
    llmprobe probe --fail-on degraded

This blocks the deploy if any LLM provider is experiencing degraded performance right now.

When a probe fails, the output shows only the failing endpoints:

Failed endpoints (1/4):
  openai/gpt-4o  DEGRADED  TTFT=280ms  Latency=950ms  Tok/s=32.1

MCP server

llmprobe MCP server

llmprobe includes a built-in Model Context Protocol server, allowing Claude Code and other MCP hosts to check LLM API health directly from an agent workflow.

Running the server

llmprobe mcp

This starts the MCP server over stdio.

Registering with Claude Code

claude mcp add --transport stdio llmprobe -- llmprobe mcp

Once registered, Claude Code can call llmprobe tools during any conversation.

Available tools

Tool	Description
`probe_all`	Probe all configured endpoints from `probes.yml`. Returns TTFT, latency, throughput, and health status for every model. Accepts an optional `config` parameter for a custom config path.
`probe_model`	Probe a single model without a config file. Requires `provider`, `model`, and `api_key_env`. Supports optional `base_url` for OpenAI-compatible endpoints and optional `label` for display.
`list_providers`	List all providers and models in the config file with their thresholds. Use this to discover available models before probing.
`get_config`	Return the full parsed configuration including defaults, providers, models, and thresholds.

Example use case: An agent calls list_providers to see what models are configured, then probe_all to verify they are healthy before deploying changes.

Configuration

defaults:
  prompt: "Hello"                                # probe prompt
  max_tokens: 20                                 # max output tokens
  timeout: 30s                                   # per-probe timeout
  concurrency: 5                                 # max parallel probes

providers:
  - name: openai                    # openai, anthropic, google, azure, bedrock
    label: openai-prod              # optional display name; useful for multiple OpenAI-compatible endpoints
    api_key: ${OPENAI_API_KEY}      # env var expansion
    base_url: https://custom.api    # optional, override endpoint
    models:
      - name: gpt-4o
        prompt: "Say hello."        # override default prompt
        max_tokens: 10              # override default max_tokens
        response_format: json       # optional; OpenAI-compatible JSON mode
        validate_json: true         # optional; mark degraded if returned content is not valid JSON
        thresholds:
          max_ttft: 2s              # alert if TTFT exceeds this
          max_latency: 10s          # alert if total latency exceeds this
          min_tokens_per_sec: 20    # alert if throughput drops below this
          max_ttft_multiplier: 2.0  # optional; compare against baseline p50
          max_latency_multiplier: 2.5

  - name: azure
    api_key: ${AZURE_OPENAI_API_KEY}
    base_url: https://your-resource.openai.azure.com
    api_version: "2024-10-21"       # optional, defaults to 2024-10-21
    models:
      - name: gpt-4o               # deployment name

  - name: bedrock
    access_key: ${AWS_ACCESS_KEY_ID}
    secret_key: ${AWS_SECRET_ACCESS_KEY}
    region: us-east-1
    models:
      - name: anthropic.claude-3-5-sonnet-20241022-v2:0

API keys and AWS credentials support ${ENV_VAR} syntax. Only credential fields are expanded, so env var references in prompts or model names are left as-is.

OpenAI-compatible providers

Many providers (Groq, Together AI, Fireworks, DeepSeek, Mistral, OpenRouter, Ollama, vLLM) expose an OpenAI-compatible API. These work out of the box by setting base_url. Use the label field to distinguish multiple OpenAI-compatible blocks:

providers:
  # Groq
  - name: openai
    label: groq
    api_key: ${GROQ_API_KEY}
    base_url: https://api.groq.com/openai
    models:
      - name: llama-3.3-70b-versatile

  # DeepSeek
  - name: openai
    label: deepseek
    api_key: ${DEEPSEEK_API_KEY}
    base_url: https://api.deepseek.com
    models:
      - name: deepseek-chat

  # Together AI
  - name: openai
    label: together
    api_key: ${TOGETHER_API_KEY}
    base_url: https://api.together.xyz
    models:
      - name: meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo

  # Local Ollama
  - name: openai
    label: ollama
    api_key: unused
    base_url: http://localhost:11434
    models:
      - name: llama3.2

See examples/ for ready-to-use configs for vLLM, SGLang, and Ollama.

JSON response validation

For OpenAI-compatible endpoints, set response_format: json to request JSON mode and validate_json: true to mark the probe as degraded if the streamed content is not valid JSON.

providers:
  - name: openai
    label: vllm-json
    api_key: unused
    base_url: http://localhost:8000
    models:
      - name: meta-llama/Llama-3.1-8B-Instruct
        prompt: 'Return {"ok": true} as JSON.'
        response_format: json
        validate_json: true

Prometheus metrics

Run with --prometheus to expose metrics for scraping:

llmprobe watch --interval 30s --prometheus :9090

Available metrics at /metrics:

Metric	Type	Labels
`llmprobe_ttft_seconds`	gauge	provider, model
`llmprobe_latency_seconds`	gauge	provider, model
`llmprobe_tokens_per_second`	gauge	provider, model
`llmprobe_token_count`	gauge	provider, model
`llmprobe_status`	gauge	provider, model
`llmprobe_probes_total`	counter	provider, model
`llmprobe_errors_total`	counter	provider, model
`llmprobe_ttft_seconds_hist`	histogram	provider, model
`llmprobe_latency_seconds_hist`	histogram	provider, model
`llmprobe_tokens_per_second_hist`	histogram	provider, model

The llmprobe_status gauge encodes health as: 1 = healthy, 0.5 = degraded, 0 = error. Use this for alerting in Grafana or Alertmanager.

OpenTelemetry metrics

Run with --otel to export probe metrics to an OTLP/gRPC collector.

llmprobe watch --interval 30s --otel localhost:4317

Exported metric names:

Metric	Description
`llmprobe.ttft.seconds`	Time to first token in seconds
`llmprobe.latency.seconds`	Total request latency in seconds
`llmprobe.tokens_per_second`	Generation throughput
`llmprobe.token_count`	Output token count from the last probe
`llmprobe.status`	1 = healthy, 0.5 = degraded, 0 = error
`llmprobe.probes.total`	Total probes executed
`llmprobe.errors.total`	Total probe errors

All metrics include provider and model attributes.

Architecture

probes.yml
  -> Config loader (YAML + env var expansion)
    -> Probe engine (concurrent goroutines per provider/model)
      -> Provider clients (raw HTTP + SSE parsing, no SDKs)
        -> Results (TTFT, latency, tokens/sec, status)
          -> Output (table, JSON, JSONL)

Each provider client is a thin HTTP wrapper that sends a streaming request and parses the response. No LLM SDKs are imported. The SSE parser handles both data-only events (OpenAI, Google) and named events (Anthropic). The Bedrock client implements SigV4 signing and AWS binary event stream parsing from scratch.

TTFT is measured from the moment the HTTP request is sent to the first event that contains actual content text (not role assignments or metadata).

Providers

Provider	Endpoint	Auth	Streaming format
OpenAI	`/v1/chat/completions`	`Authorization: Bearer`	SSE, `[DONE]` sentinel
Anthropic	`/v1/messages`	`x-api-key` header	named-event SSE
Google	`/v1beta/models/{model}:streamGenerateContent?alt=sse`	`key` query param	SSE
Azure OpenAI	`/openai/deployments/{model}/chat/completions`	`api-key` header	SSE, `[DONE]` sentinel
AWS Bedrock	`/model/{model}/converse-stream`	SigV4	AWS binary event stream
OpenAI-compat	`/v1/chat/completions` (custom base_url)	`Authorization: Bearer`	SSE

OpenAI-compatible covers: Groq, Together AI, Fireworks, DeepSeek, Mistral, OpenRouter, Ollama, vLLM, and any endpoint that speaks the OpenAI chat completions API.

Roadmap

More provider-specific examples for self-hosted OpenAI-compatible endpoints
More report formats for long-running monitoring windows
Optional runbook templates for common LLM endpoint failures

License

MIT

Available Tools

4 tools

get_configA

Return the full parsed configuration including defaults, providers, models, and thresholds. Useful for understanding the current probe setup or debugging configuration issues.

ParametersJSON Schema

Name	Required	Description	Default
`config`	No	path to probes.yml config file

TDQS

A4.1/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Describes return contents but does not mention side effects, auth requirements, or rate limits. No annotations exist to supplement.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two efficient sentences, no redundancy, front-loaded with purpose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Adequately covers purpose, return content, and common use cases for a simple tool with one optional parameter and no output schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema covers the single parameter with description. Description adds no extra meaning beyond 'full parsed configuration'; baseline 3 due to high coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states it returns the full parsed configuration, listing included elements (defaults, providers, models, thresholds). Distinct from siblings like list_providers or probe_all.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Indicates usefulness for understanding setup or debugging, implying context. Lacks explicit when-not-to-use or comparison to siblings.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_providersA

List all providers and models defined in the config file. Returns provider names, model identifiers, and any configured thresholds. Use this to discover what models are available before probing.

ParametersJSON Schema

Name	Required	Description	Default
`config`	No	path to probes.yml config file

TDQS

A4/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Implicitly a read operation, but with no annotations, the description should explicitly state it is read-only or disclose any side effects. It lacks explicit non-destructive guarantee.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two concise sentences: first states what it does, second tells when to use it. No wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple single-parameter tool with no output schema, the description is sufficiently complete, covering purpose, returns, and usage context. Minor gap in behavioral transparency.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with a single parameter described. The description adds no additional meaning beyond what the schema provides.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states it lists providers and models from a config file, specifying the exact information returned (names, identifiers, thresholds). This distinguishes it from sibling tools like 'probe_all' or 'get_config'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly advises using it 'before probing', providing clear usage context. However, it does not contrast with siblings or specify when not to use it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

probe_allA

Probe all configured LLM API endpoints. Returns TTFT (ms), total latency (ms), throughput (tokens/sec), and health status for every model in the config file.

ParametersJSON Schema

Name	Required	Description	Default
`config`	No	path to probes.yml config file

TDQS

A3.9/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so description carries full burden. It discloses return values and that it uses a config file, but does not mention side effects, error handling, or whether it's read-only. Adequate but not comprehensive.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Single sentence that is concise, front-loaded, and contains no filler. Every word adds value.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description adequately explains the return values. The single optional parameter is well-described. Sibling tool context implies complementarity with 'probe_model'. Complete for a probing tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with a clear description for the 'config' parameter. The description adds context by explaining that the config file determines which endpoints are probed, enhancing the schema's meaning.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Probe all configured LLM API endpoints') and specifies the exact metrics returned (TTFT, latency, throughput, health status). It distinguishes from sibling 'probe_model' which likely targets a single model.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool versus alternatives like 'probe_model' or 'get_config'. The description does not specify prerequisites or scenarios where this tool is appropriate.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

probe_modelA

Probe a single LLM model by provider and model name. Use this for ad-hoc checks without a config file. Returns TTFT (ms), total latency (ms), throughput (tokens/sec), and health status.

ParametersJSON Schema

Name	Required	Description
`provider`	Yes	provider name (openai, anthropic, google, azure, bedrock)
`model`	Yes	model identifier (e.g. gpt-4o, claude-sonnet-4-20250514)
`api_key_env`	Yes	environment variable name containing the API key
`base_url`	No	optional base URL for OpenAI-compatible endpoints (e.g. http://localhost:8000)
`label`	No	optional display name for the endpoint (e.g. vllm-local)

TDQS

A4/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries full burden. It discloses return values (TTFT, latency, throughput, health status) and states it probes a model, but fails to mention side effects (e.g., real API call) or safety properties (read-only vs. destructive). This is adequate but not comprehensive.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, each carrying essential information: first sentence states purpose and parameters, second sentence clarifies usage context and return values. No redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 5-parameter tool with no output schema, the description adequately covers return values and usage context. It could mention that the tool makes a live API call, but overall completeness is high for a simple diagnostic tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so baseline is 3. The description does not add parameter-level details beyond what the schema already provides; it only mentions 'provider and model name' generically.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ('Probe') and explicitly states the resource ('single LLM model by provider and model name'). It also distinguishes from siblings (probe_all, list_providers) by noting ad-hoc single-model use without a config file.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description clearly recommends use for 'ad-hoc checks without a config file', implying when to use this tool. Sibling names (probe_all, list_providers, get_config) provide contrast, but no explicit exclusions or when-not-to-use guidance are given.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

6 tool updatesv1.4.0May 29, 2026
- Removedcheck_model
- Addedget_config
- Addedlist_providers
- Removedprobe
- Addedprobe_all
- Addedprobe_model
2 tool updatesv0.1.0May 7, 2026
- First observedcheck_model
- First observedprobe

Install Server

RayrunBuild, deploy, and operate MCP servers without becoming an infrastructure team.ray.runAd

TDQS

A4.3/5.0

Disambiguation5/5

Each tool has a distinct purpose: configuration retrieval, provider listing, bulk probing, and single model probing. No overlap.

Naming Consistency5/5

All tool names follow a consistent verb_noun pattern (get_config, list_providers, probe_all, probe_model) with appropriate verbs.

Tool Count5/5

4 tools is well-scoped for the LLM probing domain, covering necessary operations without bloat.

Completeness5/5

The tool set covers all essential probe operations: viewing config, listing providers, probing all endpoints, and probing a single endpoint, with no obvious gaps.

Maintenance

ActivityInactive

ResponsivenessSyncing

Resources

GitHub Repository

Need Help?

Related Servers

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

LLM Latency Tracker A
dev.llmlatency
Measured latency, time to first token and uptime for ~45 AI inference APIs, by region.
Monitoring Observability
221
AI API Radar A
com.dropwatchhq
Live LLM API price + status radar across 11 providers, with public per-model price HISTORY.
AI & Machine Learning Monitoring
42
traced-multi-llm-proxy
ai.getvda
Traced Multi-LLM Proxy
1
LLM Provider Intelligence A
org.memoryapi
LLM provider intelligence: recommendations, pricing, and status
AI & Machine Learning Monitoring Developer Tools
32

Related MCP Servers

LLM MCP Bridge
Developer Tools Testing & QA Tools Code Execution
ramgeart
F
license
B
quality
D
maintenance
Provides a universal bridge to interact with any OpenAI-compatible LLM API (local or cloud), enabling model testing, benchmarking, quality evaluation, and chat operations with performance metrics.
Updated 9 months ago (2025-12-06 01:13 UTC)
8
-
LLM API Benchmark MCP Server
Testing & QA Tools Monitoring Developer Tools
Yoosu-L
A
license
Not graded
quality
Not graded
maintenance
Enables benchmarking of Large Language Model APIs by measuring performance metrics such as generation throughput, prompt throughput, and Time To First Token (TTFT) with configurable concurrency levels and parameters.
Updated a year ago (2025-06-26 05:01 UTC)
1
-
ppb-mcp
AI & Machine Learning Research & Data
paulplee
A
license
Not graded
quality
D
maintenance
Exposes queryable GPU inference benchmark data (quantization, throughput, VRAM, concurrent users) as tools for LLM clients.
Updated 3 months ago (2026-05-27 16:47 UTC)
MIT
qa-probeofficial
Testing & QA Tools API Testing
LS-SIEM-LLP
A
license
A
quality
A
maintenance
Probes your live API and classifies why each endpoint failed (root cause, evidence, and a calibrated confidence level), exposed over MCP so your AI assistant debugs from evidence instead of guessing. Works with FastAPI, Express, Next.js, tRPC, and GraphQL.
Updated 2 months ago (2026-06-25 08:01 UTC)
8
32
2
Apache 2.0

Latest Blog Posts

Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)
By Om-Shree-0709 on July 25, 2026.
mcp
Agent Identity
OAuth 2.1
Your AI Chatbot Just Exposed Your CEO's Salary to an Intern
By Om-Shree-0709 on July 2, 2026.
Agent Identity
MCP Security
OAuth Delegation
Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)
By Om-Shree-0709 on June 30, 2026.
Agentic Ai
Prompt Injection
WebAssembly

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/Jwrede/llmprobe'

If you have feedback or need assistance with the MCP directory API, please join our Discord server

llmprobe

Public benchmark

Install

Claude Code plugin

Quick start

What it measures

Commands

llmprobe probe

llmprobe watch

llmprobe report

llmprobe baseline

llmprobe version

CI integration

MCP server

Running the server

Registering with Claude Code

Available tools

Configuration

OpenAI-compatible providers

JSON response validation

Prometheus metrics

OpenTelemetry metrics

Architecture

Providers

Roadmap

License

Available Tools

Tool Schema Changelog

Maintenance

Resources

Looking for Admin?

Related MCP Connectors

Related MCP Servers

qa-probeofficial

Latest Blog Posts

MCP directory API

`llmprobe probe`

`llmprobe watch`

`llmprobe report`

`llmprobe baseline`

`llmprobe version`