kaigo_mcp

An MCP server that exposes Japan's long-term care supply-demand data as tools that AI agents can use.

When connected from Claude Code or Claude Desktop, it can answer questions like "Does this city have enough special nursing homes (tokuyo)?" based on public data.

Status: Server, agent, and eval all working. Column A (local 7B), Column B (local 9B), and Column D (Claude Haiku 4.5) have been measured.

What it does

MCP (Model Context Protocol) is a standard for giving LLMs tools. This repository builds the tool side; it does not include the LLM itself. The server alone makes no API calls and incurs no charges.

[考える側]                    [このリポジトリ]
Claude Code / 自作エージェント  ←stdio→  kaigo-gap サーバー
「どの道具を使うか」を判断              呼ばれたらデータを返す

Related MCP server: MCP Statistics Server

Tools

The primary indicator is the number of tokuyo beds per 100 certified people with care level 3 or higher (national: 24.6, median: 26.7). Definitions and sources are on the kaigo_gap_analysis side.

We don't do the aggregation here

The data is written out by Plan 4 (kaigo_gap_analysis) via export_web.py, and we use a copy of the same insurers.json that the public dashboard serves. We don't rewrite the division here.

If the indicator definition lived in two places, then when the agent's answer and the dashboard's numbers disagreed, there would be no way to tell which was correct. Plan 4's dataset.py is written with a "definition in one place" policy, and we maintain that across repositories. Update with python scripts/sync_data.py.

Design considerations for the tools

Returning just a number doesn't let anyone judge it. Even if you hand an LLM "12.4," it can't tell whether that's high or low, so we include the national rank and the national value in the tool descriptions.

Don't hard-code same-name municipalities to a single match. "Fuchu City" exists in both Tokyo and Hiroshima Prefecture. We return both candidates and leave the narrowing down to the caller.

Don't let a tokuyo bed count of 0 be read as "worst in the nation." 90 insurers qualify, and all of them would tie for first place. We always attach the number of tied entries, and add a note that "this is not uncommon in small municipalities, and residents often use facilities in neighboring municipalities." Without this, rank_insurers results would lead to the false conclusion that "Shingo Village is the worst region in Japan."

If domain knowledge isn't embedded in the tool responses, the same misreading recurs every time the calling model changes. That's why it goes on the tool side rather than in the prompt.

Usage

pip install -r requirements.txt
python scripts/sync_data.py     # 案4からデータを取り込む
python scripts/smoke_test.py    # 通信の疎通確認

To use it from Claude Code, launching it in this directory will read .mcp.json. Settings for registering with other clients:

{
  "mcpServers": {
    "kaigo-gap": {
      "command": "python",
      "args": ["-m", "kaigo_mcp"],
      "cwd": "/path/to/kaigo_mcp"
    }
  }
}

When writing a Windows path, separate with \ ("C:\projects\kaigo_mcp"). In JSON, \p becomes an invalid escape and the entire config file becomes unreadable. This repository's .mcp.json was actually broken by exactly that.

Agent

python -m kaigo_mcp.agent --list
python -m kaigo_mcp.agent "尼崎市は特養が足りてる？" --verbose

Tools are called through the MCP server (not by importing functions directly). If we just imported them, it wouldn't be going through MCP, and the claim "we built an MCP server" would go unverified.

Columns being compared

The columns are arranged so that the difference between each pair is exactly one factor.

We keep both B and B' because keeping only one would hide a finding. With the same model and same weights, changing only num_ctx splits the correct-answer rate between 83% and 100%. If we only published the 8k result, it would look like "the local 9B is excellent," and if we only published the 4k result, it would look like "local models aren't practical." Neither is true.

Column C is on hold. It originally referred to Qwen3-235B-A22B, but that was generation 3.0 (not matching column B) and had already been discontinued. The free-tier catalog changes, so before using it, verify it actually exists via /models. We also tried the NVIDIA free tier (C-alt-nvidia), but one round trip took 168 seconds on one occasion, making it unusable for eval.

Write only one loop. If one side uses the SDK's tool runner and the other is hand-written, you can no longer separate whether the difference between columns comes from the models or from the loop implementation. The backend is responsible only for history conversion.

Measured results (columns A and B)

Measured 6 times each with python scripts/probe_tool_calling.py.

Since columns A→B change hardware and model generation simultaneously, we also ran the 7B on the GPU machine for separation (middle column).

Temperature 0 being needed was specific to the 7B

With qwen2.5:7b at the default temperature, the opening tag of <tool_call> breaks (a few characters like olith or pering get prepended), ollama's parser can't recognize it, and the call leaks into the body text. The telltale sign is that only the closing tag </tool_call> remains. Temperature 0 fixes it.

With qwen3.5:9b, it's 6/6 even at the default temperature. This breakage isn't a general problem with local execution; it was specific to this model's generation and size.

Even running the same 7B on the GPU machine, it leaked at the default temperature (5/6), and the leak even had the same shape (brtc {"name": "lookup_insurer", ...} </tool_call> — a few characters sit in the opening tag position, leaving only the closing tag). Changing hardware doesn't fix it. It's a quirk of the model.

The bigger difference was "whether it follows instructions"

The system prompt instructs: "Before answering, check the national baseline with get_national_baseline." The 9B followed 6/6 times; the 7B never followed, 0/6. Moreover, the 9B calls two tools in parallel in a single step.

The 7B got the right answer only because it picked up the national value (24.6) embedded in the tool descriptions — it did not follow the instructed procedure. This is a difference in instruction following, not stability, and it matters more as the number of tools grows.

What worked was the model generation; the GPU only affected speed

Putting the 7B on the GPU moved neither quality metric. The tag breakage at the default temperature remained (4/6 → 5/6, within noise for n=6), and instruction following at temperature 0 stayed at 0/6. The 7B on the GPU machine fetched the baseline only 2 out of 6 times at the default temperature, and never at temperature 0.

Only the seconds moved: 7.6 sec → 0.4 sec (about 19x). So of the A→B difference, success rate and instruction following read as effects of model generation, and speed as an effect of hardware. We were able to separate them within the same table.

Incidentally, the 9B is 8x slower than the 7B (3.2 sec vs 0.4 sec). Even on the same GPU, it calls two tools in parallel and produces longer output because it follows instructions. Speed and instruction following are in direct trade-off here.

Further confirmation of the previous correction about seconds

Column B's "default 18.9 sec → temperature 0 4.4 sec" was not a temperature effect but the first-load effect. Re-measured with the improved script: first call 19.2 sec, subsequent 3.2 sec (3.9 sec after warm-up at temperature 0), with no difference by temperature once warm. The same structure as column A's first call 102.6 sec, subsequent 7.6 sec — an average that mixes in the first call looks like a temperature difference or a hardware difference.

When run end-to-end, context ran out before intelligence did

Results of running the agent through a full question, rather than tool calls in isolation.

What hit the ceiling first wasn't VRAM but ollama's default num_ctx = 4096. Since the 9B takes 6.6GB, we'd assumed the KV cache would be the constraint, but it stops at the default context length before ever getting there.

The way it stops is hard to spot. The second step ends with stop_reason='length', with prompt ~1,150 + output 2,946 = exactly 4,096. The output has 3,121 tokens, yet the body is empty.

The trigger was the model's own incorrect argument. Even though Amagasaki City is in Hyogo Prefecture, it sent lookup_insurer({'name': '尼崎市', 'pref': '大阪'}), got zero matches, then spent 2,946 tokens thinking about it and ran out before starting to write the answer. The tokens needed for self-recovery consume the context.

Raising num_ctx to 8192 resolves it, but then ollama ps shows 12%/88% CPU/GPU. It doesn't fit in 8GB. At 4k no answer comes out; at 8k it doesn't fit on the GPU — that's what running a 9B on an RTX 5050 8GB means.

A harder question (name 3 municipalities in Aomori Prefecture and compare with the national figures, --max-steps 8) completed in 3 steps and 41.06 seconds even at 4k. What determines the context isn't the difficulty of the question but whether recovery is needed.

It was a flaw in the measurement itself (now fixed)

The old loop.py set stopped_by = "end" whenever there was no tool call. Even when cut off by length with an empty body, it recorded "end," so those 4 runs were logged as "completed in 2 steps," looking identical to column A's success. If an empty answer wears the same face as a completed run, lining up the columns is meaningless.

We now set truncated when length occurs or the body is empty. Aggregation looks at RunRecord.answered. The CLI prints the reason and exits with code 1.

Flaws like this aren't found until you actually run a column that fails. Running only column A would never have revealed it.

And this fix itself had two holes. Both take effect the moment you run columns C and D.

• The value indicating truncation differs by vendor. OpenAI-compatible uses length; Anthropic uses max_tokens. Since we only checked length, column D (Claude) would fail to detect truncation. The same breakage that counted empty answers as completed in column B would recur in column D as-is.

• We didn't handle truncation mid-tool-call. The check was inside the "when there is no tool call" branch. If it proceeds with a truncated argument JSON, only a record of the tool failing remains, and the cause looks like "the model is bad."

TRUNCATED_REASONS = {"length", "max_tokens"} is now checked before the presence of a tool call. We actually exercised it with max_tokens set to 8 (truncated before the tool) and 40 (truncated mid-tool-call), and also confirmed that normal cases don't get mislabeled as truncated.

Eval results (columns A, B, and D)

6 cases, measured with python scripts/eval_agent.py. All judgments use only conditions that can be programmed (no LLM-as-judge). Column B ran 3 rounds (18 runs).

num_ctx wasn't a speed setting; it was a correct-answer-rate setting

All 3 losses at 4k are truncated/length on amagasaki-basic. The mechanism is the same one seen in the previous section (When run end-to-end, context ran out before intelligence did) showing up in the eval — not a new kind of breakage.

What the eval made clear is that the same model with the same weights splits between 83% and 100% correct answers based on settings alone. The 8k setting merely provides room for recovery; the model hasn't changed at all.

We can't write "local beat Haiku"

In these 6 cases, column B' (8k) came out ahead (18/18 vs 15/18 correct, 18/18 vs 17/18 instruction following). However, that's too weak a basis to claim it.

• Only 6 cases × 3 rounds. The sample is small

• Haiku's losses aren't performance-related. As described in "The note worked too well" below, one round included it reading the note embedded in the tool and excluding the 4 zero-bed cases. Counting losses caused by our design as model weakness is wrong

• 3x the seconds (24.0 sec vs 6.5–8.5 sec). Not the same playing field

What we can say is "for these 6 questions, the 8k local 9B was sufficient." Given the condition of 0 yen and not leaving your own machine, that's already a meaningful result.

Same correct-answer rate, different content (column A)

Columns A, B, and D all have 83% correct answers, but the breakdown differs. Column A happened to get it right by picking up the national value embedded in the tool descriptions, without doing the instructed baseline check (instruction following 2/6). Columns B and D fetch it every time. This difference matters more as the number of tools grows.

Column B's losses are the truncated cases above — not that it couldn't answer, but that it couldn't finish writing the answer. Even at the same 83%, the three columns failed for three different reasons. Lining up a single number like the correct-answer rate erases this difference.

The note on the tool side worked, but worked too well

rank_insurers responses embed "a bed count of 0 is not uncommon in small municipalities; don't judge on that alone." Column D read this and excluded the 4 zero-bed cases from its answer (once in 3 rounds). The numbers match the actual data; nothing was fabricated.

The misreading was prevented, but information was hidden instead. "Don't judge" was interpreted as "don't mention." Embedding domain knowledge in tool responses works, but it can work too well.

Next steps

• Column B will include both 4k and 8k (B and B'). The point is that settings alone split the results

• Column C is on hold. DeepInfra registration may require a card

• The NVIDIA free tier can't be used for eval (speed isn't guaranteed; one round trip took 168 seconds)

Related

• kaigo_matching — Structured extraction of care records (Plan 1)

• kaigo_gap_analysis — Supply-demand gap analysis by insurer (Plan 4, the source of this data)