Language models now generate furniture layouts one object at a time. They write each object's category, position, size and rotation as text tokens, conditioned on the room and on everything placed so far. These models are trained on large collections of professionally designed rooms, so their weights encode strong conventions: beds against walls, nightstands beside beds, desks facing away from doors. Users sometimes ask for rooms that break these conventions. We propose to study what happens then, and our hypothesis is specific. The model often obeys the unusual instruction for the object it names. Then, over the following steps, the objects that depend on it "snap back" to where convention says they belong. A bed placed in the middle of the room as asked still gets its nightstands against the wall, where the bed would normally have been. The conflict between memorized convention and the user's request is not settled once. It is refought at every later object.
This matters for two reasons. For scene generation, it names a failure that current benchmarks cannot see, because they score collision-free rooms that follow convention. For the study of knowledge conflict (the tension between facts stored in a model's weights and facts supplied in its prompt), it moves the question out of single-answer factual QA into long, structured, self-conditioning generation. In that setting, the model's own earlier outputs become part of the "context" it must trust or override. We will build a benchmark of counter-conventional room requests. We will measure where in the generated sequence compliance breaks. We will test whether a linear readout of the model's internal state at the start of each new object predicts snap-back. Finally, we will test whether a steering intervention that switches on only when that readout fires repairs dependent objects without degrading ordinary rooms.
Core Claim
In autoregressive layout models, instruction violations that contradict learned conventions concentrate on dependent objects rather than on the object the instruction names. This happens because each dependent object has its own memorized placement, tied to where the anchor usually is rather than where it actually is. We further claim that the model internally represents this three-way tension, between the weights, the user's text and its own earlier placements, at the first token of each new object. If so, a detection signal read from that position can gate a cheap intervention step by step, instead of applying a fixed correction everywhere.
Where This Sits
The scene-generation side supplies the setting. ReSpace casts indoor synthesis and editing as next-token prediction over a JSON scene description. It reports that its zero-shot editor degrades as that description grows, but it does not ask whether the cause is distraction or conflicting priors. OptiScene fine-tunes Qwen3-8B and then applies preference optimization toward expert-consensus layouts. That is exactly the training that should harden conventions into the weights, yet it never tests requests that contradict consensus. DirectLayout and Reason-3D handle spatial errors after generation, through evaluator loops or programmatic collision checks followed by re-prompting. DirectLayout reports about 185 seconds per scene. These papers motivate our question but do not pose it. None of them separates the named object from its dependents, and none looks inside the model.
The knowledge-conflict side supplies the method and the priors about failure. Wang et al. and Song et al. show that frequent entities make models stubborn against context. We carry this over by binning requests according to how atypical they are. Hong et al. document the opposite failure, gullibility. That constrains us: any intervention must be checked for damage to ordinary rooms, not only for gains on unusual ones. Analysing the Residual Stream Under Knowledge Conflicts finds that linear probes on Llama3-8B detect conflict at about 90% accuracy by layer 14 and predict the context-versus-memory decision at layers 16–17 with AUROC above 0.85. We reuse this probing recipe, but at object-boundary tokens rather than answer tokens. Cutting Off the Head Ends the Conflict identifies competing "memory heads" and "context heads" using path patching, a technique that swaps activations between two runs to trace which components cause an output. We reuse it to ask whether coordinate tokens have analogous heads. ContextFocus steers frozen models toward their context with a vector built from about 1.5k contrastive examples, but it applies a fixed strength. Adaptive Contrastive Decoding adapts strength per token from output entropy, at the cost of two forward passes per token. Our gating sits between these two: adaptive like ACD, single-pass like ContextFocus, and driven by an internal conflict readout rather than by entropy.
Research Program
Models. We will fine-tune Llama3-8B and Qwen3-8B on 3D-FRONT rooms (3D-FRONT), using a ReSpace-style JSON serialization. We will produce two variants: one supervised only, and one with an added OptiScene-style consensus preference stage. This lets us test whether consensus alignment increases snap-back. Released ReSpace or OptiScene checkpoints will be evaluated as well if available.
Benchmark. We will build the counter-conventional benchmark from 3D-FRONT and 3D-SynthPlace in two ways that yield ground truth without human design.
The first way uses real rooms. We will mine existing professionally designed rooms that sit in the tail of the layout distribution, such as a centered bed or a desk facing the door. These are coherent unusual rooms, and each gets an instruction describing only its anchor object.
The second way uses rigid edits. We will take an ordinary room and move an anchor together with its functional group, for example the bed and its nightstands, to an atypical pose, then repair collisions. The instruction mentions only the anchor. The ground truth says the dependents should keep their relative relation to it.
To measure how unusual a request is, we will use the negative log-likelihood of the anchor pose under the unconditioned model. This is the spatial analogue of entity frequency, and every result will be reported per bin. Instructions will be split into those a program can check exactly ("bed centered, headboard facing east") and relational or stylistic ones that cannot be parsed into hard constraints.
Measurements. For each generated scene, we will record three things: whether the anchor is honored, whether each dependent keeps its relation to the anchor, and, for each dependent that fails, whether it moved toward its conventional position. We define the snap-back ratio as
$$\rho = \frac{\Pr(\text{dependent violation} \mid \text{anchor honored})}{\Pr(\text{anchor violation})},$$
where both probabilities are estimated over the benchmark. Here $\rho > 1$ means failures pile up downstream rather than at the named object. We will also track violation rate as a function of how many objects were placed between the anchor and the dependent. This tests whether the conflict compounds over steps, a question ReSpace raises but does not analyze.
Mechanism. At the first token of each object, we will train linear probes on the residual stream (the running internal representation the model updates layer by layer) to predict whether that object will snap back. Path patching on coordinate tokens will then locate the heads that read anchor coordinates from the model's own earlier output versus the heads that write conventional positions.
Intervention. We will build a steering vector $v$ ContextFocus-style, contrasting activations when the model follows the scene history against when it snaps back. At layer $\ell$ we will apply
$$h_\ell \leftarrow h_\ell + \lambda\, \sigma\big(w^\top h_{\ell'} + b\big)\, v,$$
where $h_\ell$ is the residual-stream state at layer $\ell$, $w, b$ are the probe weights read at an earlier layer $\ell'$, $\sigma$ is the logistic function, and $\lambda$ is a single global strength tuned on a validation split. The gate is computed once per object and applied to that object's tokens.
Baselines
The naive reference point is the fine-tuned layout model with the instruction in the prompt and no intervention. It defines how much snap-back exists at all. No reported result on counter-conventional layouts was found on Lacuna for any method, so every baseline number on the new benchmark will be produced by us.
The strongest existing layout systems anchor our standard-room comparisons. DirectLayout reports an out-of-bound rate of 8.74% and a collision rate of 6.31% on its 3D-FRONT split, and OptiScene reports a 75% collision-free success rate in bedrooms (OptiScene). Our models must stay within a few points of these figures on ordinary rooms for any gain on unusual rooms to count. We will re-run DirectLayout-style checks on our own outputs with identical scripts.
The strongest competitor for the conflict question is augmented fine-tuning. This means supervised fine-tuning plus preference optimization on counter-conventional training scenes built by the same rigid-edit procedure, at the same data budget and GPU-hours as all our tuning combined. It is the method most likely to simply solve the problem. Beating it means comparable dependent-relation fidelity with less loss on ordinary rooms, or equal fidelity from 1.5k examples on a frozen model.
Among decoding-time methods, ACD is the strongest reference. It reports 42.06 exact match on noisy TriviaQA with Llama 2 7B, against 24.75 for fixed-weight context-aware decoding. We will run it on coordinate tokens with scene history as the "context" and report its doubled latency.
We will also run ungated ContextFocus steering with its fixed multiplier. For ContextFocus, the file gives no numerical faithfulness score; it reports only qualitative superiority over ContextDPO and COIECD on ConFiQA. Its strength will be tuned on the same validation split as ours.
The key ablation isolates the gate. It compares gated steering, ungated steering at matched average strength, and steering gated by random per-object coin flips at the same firing rate. If the random gate does as well, the probe is not doing work.
Why not just…?
Why not just enforce the constraint with a solver after generation? For instructions a program can check, a solver or rejection sampler will reliably fix the anchor, in the spirit of Reason-3D's collision-aware refinement or RLSS's greedy placement with 96–98% constraint satisfaction (RLSS). Our hypothesis, though, is that the anchor is usually already right and the dependents are wrong. A solver given only "bed centered" has no constraint for the nightstands. The benchmark's split between checkable and relational instructions, together with the dependent-fidelity metric, measures this directly. If solver repair of the anchor followed by regeneration of the dependents closes the gap, we will report that as the practical answer.
Why not just place the constrained object first so the model conditions on it? This is the cheapest serious threat. It turns the conflict into ordinary conditioning on history, which is what autoregressive layout models already do well. We include it as a baseline. It also tests the core claim, because snap-back with the anchor first means the model is overriding its own history. That is precisely the stubbornness pattern known from QA, and it would show that ordering does not remove the conflict.
Why not just fine-tune on counter-conventional examples? It may well work. This is why augmented fine-tuning is our strongest baseline at a matched budget. The risk is the gullibility trade-off documented by Hong et al.: a model trained to break convention may start breaking it unprompted. ContextFocus also argues that steering reaches comparable faithfulness from far less data. We will measure quality on ordinary rooms for both approaches. If fine-tuning dominates on both axes, the intervention claim fails and the contribution reduces to the benchmark and the diagnosis.
Why not just prompt with chain-of-thought? Chain-of-Note shows that structured intermediate reasoning helps with noisy context, but at 12.0 s versus 0.6 s per example. ContextFocus reports that prompting alone falls short of steering plus prompting. We will include a DirectLayout-style reasoning prompt with explicit anchor-relative steps, and we will report both fidelity and latency.
Decisive Checks
The first check is whether $\rho$ exceeds 1 and rises with how unusual the request is, and whether it is larger after consensus preference training. If dependents fail no more often than anchors, the snap-back hypothesis is false and the study becomes a plain compliance benchmark.
The second is whether the object-start probe predicts snap-back with AUROC comparable to the 0.85 reported for QA decisions. Patching the identified heads should then causally shift dependents between conventional and anchor-relative positions.
The third is whether gated steering improves dependent fidelity over the anchor-first and ungated baselines while staying near the DirectLayout and OptiScene reference points on ordinary rooms, with single-pass latency. It must also beat the random-gate ablation.
Each of the three checks can fail independently, and each failure tells us which part of the story to discard.
References
- ReSpace: Text-Driven Autoregressive 3D Indoor Scene Synthesis and Editing
- OptiScene: LLM-driven Indoor Scene Layout Generation via Scaled Human-aligned Data Synthesis and Multi-Stage Preference Optimization
- Direct Numerical Layout Generation for 3D Indoor Scene Synthesis via Spatial Reasoning
- Text-to-Scene with Large Reasoning Models
- A Causal View of Entity Bias in (Large) Language Models
- Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refuse
- Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noise
- Analysing the Residual Stream of Language Models Under Knowledge Conflicts
- Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Models
- ContextFocus: Activation Steering for Contextual Faithfulness in Large Language Models
- Adaptive Contrastive Decoding in Retrieval-Augmented Generation for Handling Noisy Contexts
- 3D-FRONT: 3D Furnished Rooms with layOuts and semaNTics
- RLSS: A Deep Reinforcement Learning Algorithm for Sequential Scene Generation
- Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Models