2D
16 × 16256 cells
60 of 256 cells touch the boundary.
Grid-Bench
Can LLMs Predict How Discrete Worlds Change?
Watch a world relabel
A model reads a grid, the rules, and up to 64 events, then writes the final world. A simulator checks each cell.
A real 3D benchmark trajectory, executed by the reference simulator · drag to rotate
Author affiliations
The leaderboard
Six LLMs, the same prompts, one answer each and no tools. A grid counts only if every cell is right. T is the number of events in the history.
Exact match, main benchmark
Share of instances answered with every cell right, at each history length. Rows are sorted by the pooled column, T = 8–64.
One change shapes the next
Spatial instructions describe how to move objects, transform regions, or change their surroundings. Applied in sequence, each one acts on a world that earlier ones have already changed.
From the paper’s Figure 1: the entity moves right, a SET adds an obstacle in its column (pink), and the next move, allowed two cells, stops after one (✕). Cyan is the entity (1), slate an obstacle (2), dark empty (0).
What we ask, and why
Instructions like these show up in tasks from object manipulation to geometric design, where an earlier change can alter what a later one does. Can a language model follow such a history and say exactly where everything ends up?
In Grid-Bench, the model reads the starting grid, the rules, and the events as text, and writes the complete final grid in one response. Eight operation families cover geometry, neighborhoods, connected regions, and movement, in 2D, 3D, and 4D. Histories run up to 64 events, which separates getting one operation right from keeping track of many.
The design
Grids make space explicit. Coordinates and adjacency are spelled out, so a reflection inside a box and the growth of a connected region share one representation and one answer format.
The answer is the whole world. Many spatial benchmarks check selected relations or entity states; here every cell must be right, including the many that should not change. Exact match asks for complete recovery; cell-level checks show how far a wrong answer is off, and where.
The rules execute. A simulator computes the exact final state for any history, so every prediction is checkable, and the same rules can supply training examples with exact targets.
Explicit rules do not make this easy. The best model we tested gets 95% of grids exactly right after one event, and 13% after 64.
Eight ways a world changes
Each history applies one family, interleaved with single-cell SET events. Cards replay real benchmark trajectories; select one for its rule and a worked example.
One instance, start to finish
The history from section 02, one step at a time: what the model reads, what it must write, and how the answer is scored.
The paper's Figure 1 example, not a benchmark instance.
-
A bounded world
The model starts from W0, here the 4×4 world from the paper's Figure 1, where 1 is an entity, 2 an obstacle, and 0 empty. Coordinates start at zero and do not wrap; benchmark worlds are 16×16, 6×6×7, or 4×4×4×4.
-
One rule
Every instance uses one operation family. This one is MOVE: move selected values along an axis around obstructions.
Exact rule text from the prompt
-
Events, in order
Benchmark histories hold 0, 1, 8, 16, 32, or 64 events; this example has 3. Each event sees the world the previous one left.
-
One complete answer
The model lists every nonzero cell once, in coordinate order. Only this final grid is scored.
Reference answer, built from the simulator's W3 -
Checked cell by cell
The simulator's W3 is the reference, compared at every coordinate. One wrong cell sets exact match to 0.
Hand-specified, as in the paper's Figure 1, not a model output.
See the full prompt the model reads
Same size, three geometries
Worlds hold about 256 cells in 2D, 3D, and 4D, and every trajectory is cut at six history lengths.
3D
6 × 6 × 7252 cells
172 of 252 cells touch the boundary.
4D
4 × 4 × 4 × 4256 cells in 16 slices
240 of 256 cells touch the boundary.
Each card replays the opening events of a RELABEL trajectory from the main benchmark, with states from the reference simulator.
Six prefixes, one trajectory
Every trajectory has 64 events and is cut at six lengths. Each cut is a separate request, scored against its own reference final state.
The six prefixes of a trajectory share one history, so they are not independent rows. Confidence intervals therefore resample whole trajectories, not rows, keeping each trajectory's prefixes together.How the numbers add up
(
+336workload instances7 extra 2D conditions, 56 trajectories
)
8operations
×3geometries
×8trajectories each
×6history lengths
1,152 main instances from 192 trajectories
1,488 instances from 248 trajectories
Beyond the leaderboard
Where models pull apart as histories grow, how far their wrong grids miss, and what those grids still get right.
Close at one event, far apart by eight
Kimi K2.5 and MiniMax M2.5 solve more one-event instances than Claude 5 Sonnet (77.1% and 78.1% vs. 71.4%), yet at T = 8 Claude 5 Sonnet leads, 47.4% to 17.2% and 16.1%.
Similar scores, different errors
Kimi K2.5 and MiniMax M2.5 are nearly tied at T = 8 (17.2% vs. 16.1%), yet where both return a valid wrong grid (43.8% of instances), the median wrong grid misses 5.5% of cells for Kimi K2.5 and 26.2% for MiniMax M2.5.
Wrong grids can still capture real change
At T = 64, the six models miss the exact grid on 96.4% of their attempts, yet GPT-5.6 Sol's valid answers still get 81.6% of cells right, against 51.0% for copying the initial world.
- Each length is scored on its own. A model sometimes solves a longer prefix after failing a shorter one: on 2.6% (GLM-5) to 26.6% (Claude 5 Sonnet) of trajectories.
- Missing answers count as wrong. Empty responses range from 0% of requests (GLM-5) to 16.7% (Claude 5 Sonnet; 98.8% of those stopped at the length limit).
- References are double-checked. A separately written implementation of the rules recomputes every reference state by replaying its events; 100% match.
Your turn
Paint the final grid you expect, then check it against the reference cell by cell.
MOVE
Exact rule
Events, in orderp = [row, column]
Practice worlds use the same simulator but are not benchmark instances.
Your prediction
Arrow keys move between cells. Enter or Space paints the selected value. A digit key paints that value.
Test your model on Grid-Bench
The simulator and scorer use only the Python standard library, and scoring saved answers needs no API key.
git clone https://github.com/grid-bench/Grid-Bench.git
cd Grid-Bench
python -m pip install -e .
python scripts/quickstart.py
Needs Python 3.11 or newer. Runs and scores a small example with no credentials or model calls.
grid-bench evaluate \
--data /path/to/Grid-Bench-data \
--predictions answers.jsonl \
--out evaluation
Write answers.jsonl with one JSON line per response: {"public_id": "...", "response": "..."}
The default main view scores 1,152 instances; --view all scores all 1,488. Missing answers stay in the denominator.
from huggingface_hub import snapshot_download
snapshot_download(
repo_id="xzx34/Grid-Bench",
repo_type="dataset",
local_dir="Grid-Bench-data",
)
Install huggingface_hub first. The download holds simulator references and recorded measurements, not model answers.