Skip to main content
Glama

Get forecast skill

get_forecast_skill
Read-onlyIdempotent

How accurate our forecasts have actually been near a location, measured against observed analysis truth. Returns bias (positive = the model runs high), mean absolute error, RMSE, and a skill score against local climatology, per model, weather variable, and forecast lead time; continuous and vector entries also carry persistenceSkillScore, skill against the analysis at forecast issue time (null means not enough persist pairs, not zero skill -- do not compare it to skillScore as if they shared a denominator), and analysisDisagreementMae, the analyses' own disagreement at that lead -- a floor on how good the forecast can look, not a skill score and not an excuse (null means the sibling row is missing or below minimumSamples); for probability forecasts, the Brier score and a reliability breakdown. Use this to qualify a forecast rather than assert it -- "NBM has been running 1.8F warm at 3-day leads near you, so treat that 72 as around 70" -- and to answer "how much should I trust this forecast", "is the model biased here", or "how accurate were you last month". Evidence is reported at three scopes side by side: the exact point (strongest, slowest to accumulate), the ~50km neighborhood, and the ~300km region. Prefer the most specific scope that has samples. Metrics below minimumSamples observations are withheld and listed under insufficientHistory with their count -- say that history is still accumulating rather than treating thin numbers as evidence. Coverage is a rolling recent window over verified US variables, not all of history. Entries are per model and their samples are not matched, so never conclude that one model beats another by comparing their numbers here. Each entry states the truth field it was measured against -- one designated analysis per variable -- so never compare numbers carrying different truth values either. Each entry also states the regime it was measured under: ALL for every observation regardless of weather, or a conditioned tier such as SEA:DJF (winter), SCN1:WINDY / SCN1:WET / SCN1:QUIET (what the forecast was showing), or JC1:NW (a circulation pattern). Pass the regime parameter to ask for a conditioned track record. It falls back, so asking for SCN1:WINDY and getting back regime ALL is a successful answer, not a missing one -- always read the regime field and qualify the claim with it, because "NBM runs warm here when it shows windy" and "NBM runs warm here" are different statements. Regimes overlap by construction across families, so entries under different regimes are alternative answers to one question and must never be compared or added; within SCN1: the labels are mutually exclusive. Entries with a categorical block answer a yes/no question instead of an error magnitude -- did it rain, at the thresholdMm stated on the entry -- with pod (of the times it happened, how often we called it), far (of the times we called it, how often it did not happen), and frequencyBias (above 1 = we call it too often). Use these for "will it actually rain" questions, where a small average error means nothing if the rain lands in the wrong hour. A null rate means the sample cannot answer it -- the event has not happened, or been forecast, enough times to divide by -- and must be reported as unknown, never as zero. The counts beside it are still evidence, and for a rare event they are often the whole answer: "it has only rained twice here in the record" is a useful thing to say.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
latNoLatitude in decimal degrees (-90 to 90). Most tools also accept a `location` place-name string instead of lat/lon.
lonNoLongitude in decimal degrees (-180 to 180). For continental US use negative values (west of the prime meridian).
unitNoUnits for the error magnitudes. Default imperial (bias/MAE/RMSE in °F, mph, in).
modelNoNarrow to one model, e.g. nbm or rrfs.
truthNoMeasure against a named truth source instead of the default one for each variable, e.g. urma. Only pass this if the user asked which analysis was used or named one; the default is already the designated source, and the analyses disagree, so switching changes the numbers.
regimeNoAsk for a track record measured only under particular conditions, as a comma-separated preference chain, most specific first, e.g. "SCN1:WINDY,SEA:JJA". SEA: is the meteorological season (DJF, MAM, JJA, SON); SCN1: is a forecast-conditioned scenario (WINDY, WET, QUIET — mutually exclusive within the family); JC1: is a circulation pattern. The most specific tier with enough observations answers and the unconditioned record is the last resort, so this never empties a result the way truth does -- it degrades. Read the regime field on each entry to see which tier actually answered. Pass this when the question is conditional ("is it worse in winter", "how does it do when the model shows windy"); omit it otherwise, since conditioned tiers are thinner and slower to earn numbers.
locationNoFree-text place: city ("Denver"), city+state ("Portland, OR"), US ZIP ("50219"), or "lat,lon" ("39.74,-104.99"). Provide either this OR explicit lat+lon, not both.
variableNoNarrow to one variable, e.g. temperature_2m, dew_point_2m, wind_speed_10m, wind_gusts_10m, wind_vector_10m, cloud_cover, precipitation, precipitation_probability, or a thresholded rain event such as precipitation_gt_0p254mm (any measurable rain) or precipitation_gt_2p54mm. Omit for everything measured at the location.
lead_hoursNoNarrow to the lead time being asked about, in hours; the containing lead bucket is selected for you (60 gives the 48-72h bucket). Use the lead of the forecast you are qualifying: ~24 for tomorrow, ~72 for three days out. Never approximated -- a lead we have not verified returns no entries rather than a nearby bucket, so an empty result means we cannot speak to that range.

Output Schema

TableJSON Schema
NameRequiredDescriptionDefault
cellsNo
skillYes
unitsYes
trackedYesWhether this exact coordinate is one the verification pipeline tracks.
locationYes
minimumSamplesYes
insufficientHistoryYes

Schema Changelog

Changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. Changed3 schema fields changed
    • changedInput schema / properties / model / description
      Previous value: -"Narrow to one model, e.g. nbm."New value: +"Narrow to one model, e.g. nbm or rrfs."
    • changedOutput schema / properties / insufficientHistory / items / additionalProperties
      Previous value: -falseNew value: +true
    • changedOutput schema / properties / skill / items / additionalProperties
      Previous value: -falseNew value: +true
  2. Changed4 schema fields changed
    • addedOutput schema / properties / skill / items / properties / continuous / properties / analysisDisagreementMae
      Added value: +{
      +  "type": [
      +    "number",
      +    "null"
      +  ]
      +}
    • addedOutput schema / properties / skill / items / properties / continuous / properties / analysisDisagreementSamples
      Added value: +{
      +  "type": [
      +    "number",
      +    "null"
      +  ]
      +}
    • addedOutput schema / properties / skill / items / properties / vector / properties / analysisDisagreementMae
      Added value: +{
      +  "type": [
      +    "number",
      +    "null"
      +  ]
      +}
    • addedOutput schema / properties / skill / items / properties / vector / properties / analysisDisagreementSamples
      Added value: +{
      +  "type": [
      +    "number",
      +    "null"
      +  ]
      +}
  3. Changed2 schema fields changed
    • changedInput schema / properties / regime / description
      Previous value: -"Ask for a track record measured only under particular conditions, as a comma-separated preference chain, most specific first, e.g. \"SCN1:WINDY,SEA:JJA\". SEA: is the meteorological season (DJF, MAM, JJA, SON); SCN1: is a forecast-conditioned scenario (WINDY, WET, QUIET — mutually exclusive within the family); JC: is a circulation pattern. The most specific tier with enough observations answers and the unconditioned record is the last resort, so this never empties a result the way truth does -- it degrades. Read the regime field on each entry to see which tier actually answered. Pass this when the question is conditional (\"is it worse in winter\", \"how does it do when the model shows windy\"); omit it otherwise, since conditioned tiers are thinner and slower to earn numbers."New value: +"Ask for a track record measured only under particular conditions, as a comma-separated preference chain, most specific first, e.g. \"SCN1:WINDY,SEA:JJA\". SEA: is the meteorological season (DJF, MAM, JJA, SON); SCN1: is a forecast-conditioned scenario (WINDY, WET, QUIET — mutually exclusive within the family); JC1: is a circulation pattern. The most specific tier with enough observations answers and the unconditioned record is the last resort, so this never empties a result the way truth does -- it degrades. Read the regime field on each entry to see which tier actually answered. Pass this when the question is conditional (\"is it worse in winter\", \"how does it do when the model shows windy\"); omit it otherwise, since conditioned tiers are thinner and slower to earn numbers."
    • addedOutput schema / properties / skill / items / properties / vector / properties / persistenceSkillScore
      Added value: +{
      +  "type": [
      +    "number",
      +    "null"
      +  ]
      +}
  4. Changed2 schema fields changed
    • changedInput schema / properties / variable / description
      Previous value: -"Narrow to one variable, e.g. temperature_2m, dew_point_2m, wind_speed_10m, wind_gusts_10m, cloud_cover, precipitation, precipitation_probability, or a thresholded rain event such as precipitation_gt_0p254mm (any measurable rain) or precipitation_gt_2p54mm. Omit for everything measured at the location."New value: +"Narrow to one variable, e.g. temperature_2m, dew_point_2m, wind_speed_10m, wind_gusts_10m, wind_vector_10m, cloud_cover, precipitation, precipitation_probability, or a thresholded rain event such as precipitation_gt_0p254mm (any measurable rain) or precipitation_gt_2p54mm. Omit for everything measured at the location."
    • addedOutput schema / properties / skill / items / properties / vector
      Added value: +{
      +  "additionalProperties": false,
      +  "properties": {
      +    "samples": {
      +      "type": "number"
      +    },
      +    "skillScore": {
      +      "type": [
      +        "number",
      +        "null"
      +      ]
      +    },
      +    "vectorRmse": {
      +      "type": "number"
      +    }
      +  },
      +  "required": [
      +    "samples",
      +    "vectorRmse"
      +  ],
      +  "type": "object"
      +}
  5. Changed1 schema field changed
    • addedOutput schema / properties / skill / items / properties / continuous / properties / persistenceSkillScore
      Added value: +{
      +  "type": [
      +    "number",
      +    "null"
      +  ]
      +}
  6. Changed1 schema field changed
    • changedInput schema / properties / regime / description
      Previous value: -"Ask for a track record measured only under particular conditions, as a comma-separated preference chain, most specific first, e.g. \"JC:CNW,SEA:DJF\". SEA: is the meteorological season (DJF, MAM, JJA, SON); JC: is a circulation pattern. The most specific tier with enough observations answers and the unconditioned record is the last resort, so this never empties a result the way truth does -- it degrades. Read the regime field on each entry to see which tier actually answered. Pass this when the question is conditional (\"is it worse in winter\", \"how does it do in this pattern\"); omit it otherwise, since conditioned tiers are thinner and slower to earn numbers."New value: +"Ask for a track record measured only under particular conditions, as a comma-separated preference chain, most specific first, e.g. \"SCN1:WINDY,SEA:JJA\". SEA: is the meteorological season (DJF, MAM, JJA, SON); SCN1: is a forecast-conditioned scenario (WINDY, WET, QUIET — mutually exclusive within the family); JC: is a circulation pattern. The most specific tier with enough observations answers and the unconditioned record is the last resort, so this never empties a result the way truth does -- it degrades. Read the regime field on each entry to see which tier actually answered. Pass this when the question is conditional (\"is it worse in winter\", \"how does it do when the model shows windy\"); omit it otherwise, since conditioned tiers are thinner and slower to earn numbers."
  7. Changed2 schema fields changed
    • changedInput schema / properties / variable / description
      Previous value: -"Narrow to one variable, e.g. temperature_2m, dew_point_2m, wind_speed_10m, wind_gusts_10m, cloud_cover, precipitation, precipitation_probability. Omit for everything measured at the location."New value: +"Narrow to one variable, e.g. temperature_2m, dew_point_2m, wind_speed_10m, wind_gusts_10m, cloud_cover, precipitation, precipitation_probability, or a thresholded rain event such as precipitation_gt_0p254mm (any measurable rain) or precipitation_gt_2p54mm. Omit for everything measured at the location."
    • addedOutput schema / properties / skill / items / properties / categorical
      Added value: +{
      +  "additionalProperties": false,
      +  "properties": {
      +    "baseRate": {
      +      "type": "number"
      +    },
      +    "baseVariable": {
      +      "type": "string"
      +    },
      +    "correctNegatives": {
      +      "type": "number"
      +    },
      +    "csi": {
      +      "type": [
      +        "number",
      +        "null"
      +      ]
      +    },
      +    "ets": {
      +      "type": [
      +        "number",
      +        "null"
      +      ]
      +    },
      +    "falseAlarms": {
      +      "type": "number"
      +    },
      +    "far": {
      +      "type": [
      +        "number",
      +        "null"
      +      ]
      +    },
      +    "frequencyBias": {
      +      "type": [
      +        "number",
      +        "null"
      +      ]
      +    },
      +    "hits": {
      +      "type": "number"
      +    },
      +    "misses": {
      +      "type": "number"
      +    },
      +    "pod": {
      +      "type": [
      +        "number",
      +        "null"
      +      ]
      +    },
      +    "samples": {
      +      "type": "number"
      +    },
      +    "thresholdMm": {
      +      "type": "number"
      +    }
      +  },
      +  "required": [
      +    "samples",
      +    "thresholdMm",
      +    "hits",
      +    "misses",
      +    "falseAlarms",
      +    "correctNegatives",
      +    "baseRate"
      +  ],
      +  "type": "object"
      +}
  8. Changed3 schema fields changed
    • addedInput schema / properties / regime
      Added value: +{
      +  "description": "Ask for a track record measured only under particular conditions, as a comma-separated preference chain, most specific first, e.g. \"JC:CNW,SEA:DJF\". SEA: is the meteorological season (DJF, MAM, JJA, SON); JC: is a circulation pattern. The most specific tier with enough observations answers and the unconditioned record is the last resort, so this never empties a result the way truth does -- it degrades. Read the regime field on each entry to see which tier actually answered. Pass this when the question is conditional (\"is it worse in winter\", \"how does it do in this pattern\"); omit it otherwise, since conditioned tiers are thinner and slower to earn numbers.",
      +  "type": "string"
      +}
    • addedOutput schema / properties / insufficientHistory / items / properties / regime
      Added value: +{
      +  "type": "string"
      +}
    • addedOutput schema / properties / skill / items / properties / regime
      Added value: +{
      +  "type": "string"
      +}
  9. Changed3 schema fields changed
    • addedInput schema / properties / truth
      Added value: +{
      +  "description": "Measure against a named truth source instead of the default one for each variable, e.g. urma. Only pass this if the user asked which analysis was used or named one; the default is already the designated source, and the analyses disagree, so switching changes the numbers.",
      +  "type": "string"
      +}
    • addedOutput schema / properties / insufficientHistory / items / properties / truth
      Added value: +{
      +  "type": "string"
      +}
    • addedOutput schema / properties / skill / items / properties / truth
      Added value: +{
      +  "type": "string"
      +}
  10. Added

TDQS

A4.6/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations declare readOnlyHint/openWorldHint/idempotentHint and the description agrees with no contradiction. Beyond annotations it discloses exceptional behavioral depth: null means 'not enough persist pairs, not zero skill — do not compare it to skillScore'; regime queries degrade to ALL rather than erroring; metrics below minimumSamples are withheld into insufficientHistory; coverage is a rolling recent window over verified US variables, not all history; samples are not matched across models so cross-model comparison is invalid; entries under different regimes are alternative answers that must never be compared. This far exceeds what annotations provide.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Core purpose, return values, and the usage example are front-loaded before the caveat sections, and nearly every sentence carries real interpretive weight given the tool's complexity. It loses a point for sheer length — roughly 700 words, at the extreme end of what an agent can scan efficiently — where tighter bulleted structure would improve parsability. Every section earns its place, but the whole is longer than ideal.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 9-parameter tool with no required parameters and an output schema present, the description covers everything needed for correct invocation and result interpretation: three scopes with a preference rule ('prefer the most specific scope that has samples'), minimumSamples withholding, null semantics for multiple field types, regime fallback behavior, and three separate incomparability constraints (model, truth, regime). The output schema carries return structure, so nothing critical for correct use is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so baseline is 3. The description adds genuine interpretive semantics beyond the field-level text: model and truth carry comparability constraints ('never conclude that one model beats another', 'never compare numbers carrying different truth values'), regime gains cross-family overlap and within-family mutual exclusivity ('within SCN1: the labels are mutually exclusive'), and lead_hours gains the 'never approximated — a lead we have not verified returns no entries' warning. This is meaningful added value, though not organized parameter-by-parameter.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The opening sentence states a specific verb+resource — measuring forecast accuracy near a location against observed analysis truth — and enumerates the concrete outputs (bias, MAE, RMSE, skill score). The phrase 'qualify a forecast rather than assert it' plus the worked example ('NBM has been running 1.8F warm at 3-day leads') sharply separates this from forecast-content siblings like get_forecast or get_hourly_forecast. An agent can identify what this tool is for without inspecting sibling schemas.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives explicit when-to-use signals: qualify a forecast rather than assert it, and answer 'how much should I trust this forecast', 'is the model biased here', or 'how accurate were you last month'. It also provides an example of how the answer should be phrased. However, it never names alternatives or states when-not-to-use conditions (e.g., nothing says 'for the actual forecast values, use get_forecast'), so routing relies on clear context rather than explicit exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

TDQS

A4.1/5.0
Disambiguation4/5

Most tools have clearly distinct purposes, and the detailed descriptions generally prevent misselection. A few near-overlapping pairs exist — get_current_conditions vs get_observations, and get_forecast already bundling current conditions, alerts, and outlooks — so some ambiguity remains.

Naming Consistency4/5

The overwhelming majority of tools follow a get_<object>_<modifier> pattern in snake_case, and the non-get tools still use an imperative verb_noun form. The mix of verbs (get, list, describe, find, query, search, reverse) is a minor inconsistency, but the overall pattern is predictable.

Tool Count2/5

At 32 tools, the surface is heavy and exceeds the 25+ threshold for a large tool set. The weather domain justifies much of the breadth, but several tools overlap in scope and could plausibly be consolidated, making the count feel higher than necessary.

Completeness5/5

The tool set comprehensively covers current conditions, forecasts, hourly data, climate, alerts, severe weather, air quality, tropical systems, upper-air soundings, maps, model data, geocoding, and platform status. There are no obvious dead-end workflows, and raw access via query_dataset fills most remaining gaps.