Language models are being asked to act as urban planners. They draft zoning maps, argue with simulated residents, and check simulated futures against ecological redlines. Whether they can actually track how a landscape changes over many steps is still unmeasured. We propose to measure it with a well-understood generator of ground truth: the patch-growth rules used in land-use simulators. A cellular automaton is a grid model in which each cell updates according to its neighbors. In the patch-generating variant, under-allocated land classes plant seeds in suitable places and grow contiguous patches. Classes compete for each cell, and a class that falls behind its area target becomes more aggressive. These rules are deterministic once seeded, inherently two-dimensional, and multi-class. That lets us produce unlimited reasoning problems with exact answers at three depths: reading the map, applying one update rule, and running many coupled updates under area quotas.
The point is not to replace the simulators. The point is to use them as a diagnostic instrument for two open questions. The first is whether the sharp "cliff" in multi-step reasoning, where small models score near zero and large ones suddenly succeed, is simply step-wise errors compounding. The second is why step-by-step prompting (chain-of-thought, where the model writes out intermediate reasoning before answering) helps large models but hurts small ones. Our central bet is that one measurable quantity explains the second question: how faithfully a model can restate the local map in text. We think chain-of-thought starts helping at the scale where a model can copy a cell's neighborhood correctly. Parameter count matters only through that copying ability.
Core Claim
We make two claims, each falsifiable. First, on spatial state-tracking tasks, the probability of answering correctly after $N$ automaton steps is predicted by the product of per-step success rates, each measured when the model is handed the correct state:
$$\hat{S}_N = \prod_{k=1}^{N} p_k ,$$
Here $p_k$ is the model's accuracy on step $k$ given the gold state after step $k-1$, and $\hat{S}_N$ is the predicted end-to-end accuracy. This is the compounding argument the threshold literature states informally. If observed accuracy falls well below $\hat{S}_N$ at some scale, errors interact: a wrong cell makes later cells wrong more often than chance would. That would point to a genuine qualitative failure rather than an artifact of how accuracy is scored.
Second, the sign of the chain-of-thought effect at each tier is predicted by the model's accuracy at re-stating the grid in its own reasoning (tier-1 transcription), rather than by its size. If we hand a sub-threshold model a pre-serialized, correct neighborhood, the prompt-induced harm should shrink or reverse. This is a causal test that size alone cannot pass.
Grounding in Prior Work
The motivating evidence comes from three benchmarks that pair task tiers with model-scale and prompting studies, none of them spatial-dynamic. ChessQA builds a five-level chess ladder. It finds that models reach up to 97% on board mechanics but a mean of 17.4% on short tactics. It attributes the failure to "board-state hallucination" and reports a +14.7-point average gain from explicit reasoning, at costs above 11,000 tokens per problem. GEOTEMP tiers time-zone reasoning and finds near-total failure below 7B. Llama 3-70B reaches only 25.4% on its hardest tier despite about 90% on direct offset lookup, and chain-of-thought hurts on the easy tiers. TMBench isolates rule-following on one-dimensional tag systems and finds models below 4B fail at the first steps. Our work takes their logic of tiers and step-wise decay into a two-dimensional, multi-class setting with neighborhood effects and quotas, which none of them touches.
Chess is a closed 8×8 game. GEOTEMP's geography is zero-dimensional point lookups. TMBench uses a one-dimensional queue. Measuring Taiwanese Mandarin Language Understanding supplies the constraining observation that step-by-step prompting helped GPT-4 by 15% in STEM while degrading smaller models. Our second claim proposes and tests a mechanism for that contradiction, which none of these papers isolate.
The task generator comes from the land-use literature. The Patch-generating Land Use Simulation model, as used in the Nanchang carbon study and the Kriya River risk study, combines two components. A random-forest suitability map is mined from historical change. A growth engine then plants seeds, weights conversions by neighborhood density, and applies self-adaptive inertia, under which a lagging class gains competitive strength. That inertia behavior is described in the Loess Plateau study. The ecological-network coupling adds corridor constraints as resistance surfaces, which we reuse to pose connectivity questions. We reimplement a simplified, fully deterministic version of these rules so that every answer is exact.
The closest LLM-and-land-use work is Large Language Model for Participatory Urban Planning. It uses GPT-4V planners and GPT-3.5 resident agents and reports 11.2–12.9% satisfaction gains over cellular-automaton and reinforcement-learning baselines. It evaluates only aggregate satisfaction with two fixed models, with no tiered diagnosis and no scale sweep. Our work asks whether the spatial reasoning such agents implicitly depend on exists at the scales people would deploy.
Research Program
We will build a task generator around the simplified patch-growth engine. Its inputs are grids cropped from 30 m land-cover rasters of the kind used in the studies above, plus driver layers such as slope and road distance. Grids range from 8×8 to 32×32 and have three to six land classes. Each instance is serialized as text, with a coordinate-indexed grid, a rule card, and the area targets. Rendered images are a secondary modality.
Tier one asks grounding questions: which class a cell holds, a driver value, or how many neighbors of a class lie in a 3×3 window. Tier two asks for one rule application: compute a cell's combined score from suitability, neighborhood weight and inertia, and say which class wins the competition for that cell. Tier three asks the model to run $N$ coupled steps, with $N$ from 1 to 20, including seed growth and inertia updates. It then answers a patch-level question: the final class of a target cell, whether a class met its target, how many patches exist, or whether a corridor remains connected.
Every tier-three instance comes with oracle-conditioned versions that supply the gold intermediate state at step $k$. This gives us the per-step $p_k$ needed for the compounding test. It also addresses the inter-tier cascade problem the taxonomy literature flags: separating a broken high-level procedure from a procedure starved of correct inputs.
Land-use change is sparse, so most cells never change. We therefore score tier three on changed cells only, by analogy to Figure of Merit, the land-use metric that counts agreement only where change actually occurred. This matters. The Shenzhen three-dimensional simulation reports overall accuracy of 0.9561 alongside a Figure of Merit of just 0.1907, which shows how badly cell-wise accuracy flatters a model that predicts no change.
To address the "artifact versus real transition" debate directly, we also record the log-probability the model assigns to the gold answer. This continuous measure lets us check whether an apparent cliff in accuracy sits on a smooth underlying curve.
Baselines
The naive reference is the persistence baseline: predict that every cell keeps its current class. It is trivially strong on cell-wise accuracy, which is why we report changed-cell accuracy. Any model that does not beat persistence on changed cells has learned nothing about the dynamics. Alongside it we report chance at each tier.
The core comparison runs four prompting conditions on the same items, the same decoding settings, and an equal few-shot example budget. The conditions are run on open-weight families spanning roughly 1B to 72B, mirroring GEOTEMP's Llama and Qwen sweep, plus a small number of frontier reasoning models.
The first condition is direct answering. The second is free-form chain-of-thought. The third is a structured procedural trace modeled on the five-stage protocol of SoS1, which reports up to a 21% gain from guided traces. Our trace instructs the model to transcribe the neighborhood, compute scores, resolve competition, update inertia, and then move to the next cell. It is the strongest prompting competitor because it targets exactly the failure we hypothesize. The fourth condition is oracle-transcription chain-of-thought, where the neighborhood serialization is supplied correctly. This is the ablation that isolates our key ingredient. If oracle transcription does not change the sign of the chain-of-thought effect in small models, our second claim is wrong.
The strongest existing method for pushing small models past a floor is task-specific fine-tuning. SoS1 reports SoS-7B at 70% against DeepSeek-V3's 69% on SoS-1K after four hours of training. We will fine-tune a 7B model on generator traces under a matched budget, with no overlap between training and test grids. Our test distribution includes larger grids and longer horizons than training, so we can see whether the gains transfer to the dynamics or merely fit the training format.
We also run a tool arm, in which the model writes code that executes the rules. It is the practical upper bound and a different question, so we report it separately.
For headline numbers to beat, there is no reported result found on Lacuna for language models on land-use automaton tasks. The nearest published references are ChessQA's 79.3% for its top thinking model and TMBench's above-90% at step 30 for Gemini-2.5-Pro. The relevant reference result for a chain-of-thought dividend is ChessQA's +14.7 points. "Beating" the strongest baseline here means the structured trace or the fine-tuned 7B closes at least half the gap to the frontier models on changed-cell tier-three accuracy at $N \geq 10$.
Why not just…?
Why not just use TMBench or ChessQA, which already measure state tracking? They may suffice, and we will check this directly. We will compute rank correlation between model scores on our tier three and TMBench pass rates over the shared model set. If the correlation is near perfect and per-step decay curves have the same shape, two-dimensional multi-class competition adds no new failure mode, and we will say so. We expect divergence for a specific reason. Neighborhood counting and quota-driven inertia require attending to many scattered positions at once. A tag system's head-and-tail operations do not.
Why not just report flat end-to-end accuracy across model sizes? Flat accuracy cannot tell a model that misreads the grid from one that misapplies a correct reading. GEOTEMP's knowledge paradox, with 90% offset lookup but 25.4% multi-hop accuracy, is exactly the kind of gap that only tiering exposes. The oracle-conditioned versions are what let us estimate $p_k$ at all. Without them the compounding test is impossible.
Why not just use a smooth metric and declare the cliff an artifact? We do record gold-answer log-probability. But smoothness in that metric would not settle whether errors compound independently. The $\hat{S}_N$ comparison asks a different question. If the curve is smooth and observed accuracy matches $\hat{S}_N$, the cliff is pure compounding. If the curve is smooth but observed accuracy falls below $\hat{S}_N$, errors are correlated in a way neither framing captures.
Why not just let the model call the simulator as a tool? For running land-use projections, this is the right engineering answer, and we expect the tool arm to dominate. It does not answer whether a model can audit a map it is shown, or reason about a proposed change without running code. Those are the situations in which agents like the participatory-planning system make judgments. RS-Agent also found that tool orchestration fails without curated procedural workflows. That suggests tool use does not remove the need for procedural competence. Reporting the tool arm alongside the others makes the size of this gap visible.
Decisive Checks
The first check is the compounding test. For each model, we compare observed tier-three accuracy at horizons 1 to 20 with $\hat{S}_N$ built from oracle-conditioned $p_k$, using bootstrap intervals over items. The emergent-cliff-as-compounding account predicts agreement within intervals across all scales.
The second check is the chain-of-thought crossover. We plot the free-form chain-of-thought gain against tier-one transcription accuracy and against log parameter count. We then test which predictor better explains the gain on held-out model families. The prediction to confirm is that transcription accuracy wins.
The third check is causal. In models below roughly 7B, oracle transcription should remove most of the harm from chain-of-thought. If it does not, the harm comes from rule application rather than from copying the map, and the second claim falls.
Throughout, we report token cost per correct answer. A prompting method that wins only at ChessQA-scale token budgets has limited value for a planning agent that must check many cells.
References
- ChessQA: Evaluating Large Language Models for Chess Understanding
- Around the World in 24 Hours: Probing LLM Knowledge of Time and Place
- Computational Reasoning of Large Language Models
- Measuring Taiwanese Mandarin Language Understanding
- Dynamic Simulation and Prediction of Carbon Storage Based on Land Use/Land Cover Change from 2000 to 2040: A Case Study of the Nanchang Urban Agglomeration
- Landscape Ecological Risk Assessment of Kriya River Basin in Xinjiang and Its Multi-Scenario Simulation Analysis
- Quantitatively Assessing and Attributing Land Use and Land Cover Changes on China's Loess Plateau
- Coupling an Ecological Network with Multi-Scenario Land Use Simulation: An Ecological Spatial Constraint Approach
- Large Language Model for Participatory Urban Planning
- Three-Dimensional Simulation Model for Synergistically Simulating Urban Horizontal Expansion and Vertical Growth
- SoS1: O1 and R1-Like Reasoning LLMs are Sum-of-Square Solvers
- RS-Agent: Automating Remote Sensing Tasks through Intelligent Agent