TL;DR
Large Language Models (LLMs) often fail when explicit in-context rules contradict their pre-trained commonsense knowledge—a flaw termed "Semantic Inertia" that surprisingly worsens as models scale up. This paper introduces Code-Grounded Vistas (LCV), a framework that overcomes this limitation by training models to translate mutable rules into executable Python code, effectively decoupling logical reasoning from semantic biases and restoring positive scaling laws.
What Problem Does This Paper Address?
Autoregressive LLMs excel at leveraging commonsense knowledge because they rely on distributional semantics—the statistical correlations between words learned during pre-training. However, this strength becomes a liability when a task requires fluidly overriding these learned associations. For instance, if an environment dictates that "Lava is safe" or "Walls are passable," an LLM will often hallucinate standard physics (avoiding lava or treating walls as obstacles) despite explicit instructions to the contrary.
The authors term this phenomenon Semantic Inertia: the tendency for parametric memory to override in-context reasoning. Crucially, they document an inverse scaling effect: larger models, having absorbed more human-centric data, possess more deeply entrenched priors and perform worse than smaller models when forced to reason using counterintuitive natural language rules. This creates a computational analog to the human Stroop effect, where descriptive text semantics entangle with and disrupt logical execution.
Methods
To break the grip of semantic inertia, the authors propose Code-Grounded Vistas (LCV). Instead of using natural language to predict actions directly, LCV shifts the LLM's role to an active theorist that maps the environment state and rules into an executable Python transition kernel ($T_t \leftarrow f_\theta(s_t)$). A classical symbolic planner then uses this generated code as a dynamics oracle to search for the optimal path.
To ensure the model strictly respects mutable rules rather than visual semantics, LCV introduces Counterfactual Contrastive Alignment during Supervised Fine-Tuning (SFT). The training corpus is structured in pairs: two instances share the exact same visual grid layout ($v$) but feature completely contradictory rules ($r^+$ and $r^-$), yielding distinct ground-truth programs ($c^+$ and $c^-$):
- Instance A: $\langle v, r^+ = \text{WALL IS STOP}, c^+ \rangle$
- Instance B: $\langle v, r^- = \text{WALL IS PASS}, c^- \rangle$
By optimizing a contrastive disentanglement loss, gradients relying solely on visual appearance oscillate and cancel out. The model is forced to suppress the ambiguous visual signals and focus entirely on the active text rules, effectively orthogonalizing logical dynamics from visual appearance.
Datasets
The framework is evaluated on BABABENCH, a newly introduced benchmark based on the logic puzzle game Baba Is You. In this environment, physical laws exist as tangible text blocks that can be manipulated to rewrite the game's logic in real-time. BABABENCH features three tiers of increasing semantic friction:
- Tier 1: Semantic Alignment (Control): Rules align with pre-training priors (e.g.,
WALL IS STOP,FLAG IS WIN). - Tier 2: Semantic Conflict (Inhibitory Control): Rules violate commonsense physics (e.g.,
LAVA IS SAFE,WALL IS YOU). - Tier 3: Dynamic Plasticity (Rule Adaptation): Rules shift mid-episode, requiring multi-stage planning and real-time rule rewriting.
The authors evaluated three model families (Llama-3, Pythia, Qwen2.5) spanning sizes from 160M to 70B parameters.
Key Insights and Ideas
- The Failure of Scale in NL: When prompting models using natural language, larger models exhibit stronger semantic inertia. For example, Llama-3-70B shows a worse logit difference ($\Delta P = -0.18$) than its 8B counterpart when dealing with counterintuitive rules.
- Code as a Semantic Filter: Expressing environment physics as variable assignments in code strips symbols of their descriptive baggage. This shifts the model's internal processing from surface-level pattern matching to strict logical inference, restoring positive scaling laws.
- Amortized vs. Inference-Time Compute: Rather than burning tokens at inference time to "debug" its way out of a prior bias (a strategy used by frameworks like TheoryCoder), LCV compiles inhibitory control directly into the model's weights during training. This turns a complex reasoning task into a rapid, single-pass forward retrieval.
Key Results
- Reversing the Scaling Trend: As shown in Figure 2, code grounding completely flips the inverse scaling curve. Under code grounding, Llama-3-70B achieves a positive logit difference of $+0.29$, a $0.47$ improvement over its natural language performance.
- Success Rate Dominance: On Tier 2 (Semantic Conflict), LCV achieved a 75.56% Success Rate (SR), vastly outperforming massive base models like Claude 4.5 Sonnet (13.33%) and GPT-4o with a single API call (13.33%).
- Efficiency Gains: LCV achieved a 4x reduction in inference latency compared to iterative inference-time search methods like TheoryCoder, while demonstrating superior robustness against error accumulation in dynamic settings (Tier 3).
Limitations and Future Work
- Discrete State Spaces: LCV currently relies on discrete state abstractions (grid worlds). Transferring this approach to continuous, high-dimensional visual domains would require integrating an additional robust perception module.
- Data Curation Costs: Counterfactual Contrastive Alignment requires paired data with explicitly contradictory rules. While this is straightforward to procedurally generate in logic puzzles, curating such contrastive pairs in naturalistic or real-world environments could prove highly complex and expensive.
Summary updated