Skip to main content
Glama

GameDevBench

A benchmark suite for evaluating LLM agents on game development tasks.

Paper: GameDevBench: A Comprehensive Benchmark for Game Development

Overview

GameDevBench contains 132 game development tasks to evaluate LLM agents' ability to complete game development problems in the Godot game engine.

Related MCP server: PeepIt MCP

Installation

Prerequisites

  1. Godot 4.x - Download and install from godotengine.org

    • Ensure godot is available in your PATH, or set GODOT_EXEC_PATH environment variable

  2. Python 3.10+ - Required for all agents

    • Python 3.12+ - Required for OpenHands agent

Install Agents

Install the agent(s) you want to use:

Setup Tasks

Before running the benchmark, unzip the tasks folder:

unzip tasks.zip

Note: The tasks are distributed as a zip file to prevent accidental data leakage.

Configuration

Environment Variables

You can use the built-in plans for claude-code, codex, and gemini-cli, or provide API keys directly. For OpenHands you must provide your own API keys. See .env.example for a complete list of optional environment variables.

Usage

Running the Benchmark

uv run python gamedevbench/src/benchmark_runner.py \
  --agent AGENT \
  --model MODEL \
  run --task-list tasks.yaml

Available Agents

  • claude-code - Anthropic's Claude Code CLI

  • codex - OpenAI Codex

  • gemini-cli - Google Gemini CLI

  • openhands - OpenHands (requires Python 3.12+)

Command-Line Options

  • --agent AGENT - Agent to use (required)

  • --model MODEL - Model name (e.g., claude-sonnet-4.5-20250929)

  • --enable-mcp - Enable MCP (Model Context Protocol) server for supported agents

    • Provides screenshot capabilities to the agent

    • Note: MCP server requires macOS (see limitations below)

  • --use-runtime-video - Enable runtime video mode

    • Appends Godot runtime instructions to prompts

    • Helps agents understand how to run and test their changes

  • --skip-display - Skip tasks that require display

  • run --task-list FILE - Run tasks from YAML file (e.g., tasks.yaml)

Platform Limitations

macOS-only Features:

  • MCP server screenshot functionality (--enable-mcp) currently only works on macOS

    • Uses AppleScript for display capture

    • Requires setting GODOT_SCREENSHOT_DISPLAY environment variable to correct display number

Results

Benchmark results are saved to results/ directory with the following information:

  • Task success/failure status

  • Token usage and costs

  • Execution time

  • Validation results

Citation

@misc{chi2026gamedevbenchevaluatingagenticcapabilities,
      title={GameDevBench: Evaluating Agentic Capabilities Through Game Development},
      author={Wayne Chi and Yixiong Fang and Arnav Yayavaram and Siddharth Yayavaram and Seth Karten and Qiuhong Anna Wei and Runkun Chen and Alexander Wang and Valerie Chen and Ameet Talwalkar and Chris Donahue},
      year={2026},
      eprint={2602.11103},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2602.11103},
}

License

gamedevbench-bp

Available Tools

1 tool
godot-screenshotC

Launch Godot editor on specified display and take a screenshot

ParametersJSON Schema
NameRequiredDescriptionDefault
project_dirYesPath to the Godot project directory
displayNoNo need to specify this, it will be automatically determined by the MCP server. Do not specify this if you are not sure about the display number.

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. While it mentions launching the editor (a side effect), it fails to disclose critical behavioral traits: whether the editor process remains open after screenshotting, the output format/destination of the screenshot, cleanup behavior, or required environment setup.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, efficient sentence with no redundant words. The action verbs are front-loaded, and every word contributes to understanding the tool's function.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the complexity (external process launch, GUI automation, image capture) and absence of output schema or annotations, the description is incomplete. It fails to specify the return value format (critical for an agent to use the screenshot) or side effect persistence (editor state).

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, establishing a baseline of 3. The description mentions 'specified display' (referencing the display parameter) and implies project_dir is used to launch that project, but adds no semantic information about valid formats, constraints, or the warning (from schema) that display should typically be omitted.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the specific actions: 'Launch Godot editor' and 'take a screenshot'. However, it omits what happens to the screenshot (returned as data? saved to file?) and doesn't distinguish scope since there are no sibling tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this versus alternatives, prerequisites (e.g., Godot installation), or when not to use it. The phrase 'specified display' contradicts the parameter schema which advises against specifying the display.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. 1 tool updatev0.1.0
    • First observedgodot-screenshot

TDQS

B3/5.0
Disambiguation5/5

With only one tool, there is no possibility of ambiguity or overlap between tools. The tool's purpose is clearly defined and distinct by default.

Naming Consistency5/5

A single tool inherently has perfect naming consistency, as there are no other tools to compare it against. The name 'godot-screenshot' follows a clear noun-verb pattern.

Tool Count2/5

A single tool for a server named 'GameDevBench MCP' suggests an extremely limited scope that does not align with the implied breadth of game development tasks. This feels thin and incomplete for the domain.

Completeness1/5

The tool set is severely incomplete for game development, covering only a niche screenshot function. There are obvious gaps in core operations like asset management, testing, or build processes, making it inadequate for the stated purpose.

Maintenance

ActivityInactive
ResponsivenessNo issues

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/SeeleAI/gamedevbench-mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server