TL;DR
Current text-to-image (T2I) models struggle when spatial relations are described from an object’s point of view rather than the camera’s. FoR-SALE solves this by using vision models to detect 3D attributes (bounding boxes, depth, and facing orientation), prompting an LLM to translate non-camera spatial descriptions into camera-centric coordinates, and applying targeted latent-space diffusion edits—including new operations for adjusting depth and facing direction.
What Problem Does This Paper Address?
Humans naturally switch between different Frames of Reference (FoR) when describing spatial scenes:
- Relative (camera-centric): "The dog is to the left of the car" (from the viewer's vantage point).
- Intrinsic (object-centric): "The dog is to the left of the car from the car's perspective."
While state-of-the-art models like FLUX.1 and GPT-4o handle camera-relative descriptions reasonably well, their performance collapses when faced with intrinsic perspectives. In tests on the synthetic FoR-LMD benchmark, GPT-4o achieves 94.76% accuracy on relative descriptions, but plummets to 24.35% on intrinsic descriptions. Current self-correcting editing pipelines (such as SLD) fail here because they do not model perspective shifts or support 3D orientation and depth adjustments.
Methods
FoR-SALE builds on the Self-correcting LLM-controlled Diffusion (SLD) paradigm, introducing a perspective-aware, closed-loop edit cycle:
Initial Image + Text Prompt
│
▼
[Visual Perception] (OWLv2 BBoxes + DPT Depth + OrientAnything 3D Facing)
│
▼
[FoR-Interpreter LLM] (Applies 32 symbolic rules to rewrite prompt into Camera View)
│
▼
[Layout Interpreter LLM] (Detects mismatches & proposes corrected target layout)
│
▼
[Latent Diffusion Editing] (Performs Addition, Deletion, Reposition, Depth, or Facing Edits)
- Fine-Grained 3D Perception: Beyond 2D bounding boxes (OWLv2), FoR-SALE extracts depth maps (DPT) and 8-way directional facing angles (OrientAnything).
- FoR-Interpreter: Rather than forcing the LLM to reason directly about non-camera space, this module uses 32 geometric conversion rules (covering 8 facing orientations $\times$ 4 cardinal directions) to rewrite the prompt into a unified camera perspective.
- Novel Latent Operations: Standard pipelines only support 2D addition, deletion, repositioning, and color/attribute modification. FoR-SALE adds:
- Facing Direction Modification: Uses SAM segmentation and DiffEdit with orientation-specific text prompts to reorient objects in latent space.
- Depth Modification: Rescales depth maps and leverages ControlNet to regenerate objects at the correct depth layer before blending back via backward diffusion.
Datasets
The authors evaluate on two benchmarks of 500 prompts each:
- FoR-LMD: An extension of the LLM-grounded Diffusion benchmark, augmented by the authors with explicit reference cues (
relativevs.intrinsic) for directional relations (left, right, front, back). - FoREST (Premsri & Kordjamshidi, 2025): A synthetic spatial benchmark sampling from the C-split, featuring explicit facing directions and reference perspectives.
Key Insights and Ideas
- Perspective Conversion Simplifies Reasoning: LLMs excel at camera-centric spatial planning but fail at mental coordinate rotation. Decoupling perspective conversion (via explicit symbolic rules) from layout planning yields a +38.5% boost in intrinsic layout accuracy for open-weight models like Qwen3-32B.
- The "Depth-Duplication" Trap: Earlier frameworks lacking depth awareness (like SLD) frequently attempt to fix front/back errors by mistakenly hallucinating a second duplicate object, trading a spatial error for an object-count error.
- More Aggressive Plans Degrade Image Quality: In ablations, OpenAI's o3 produced higher-accuracy raw layout plans than o4-mini, but resulted in lower final image accuracy. Why? o3 tended to re-architect the entire scene (requiring 59% repositioning actions) rather than proposing minimal edits. Heavy latent blending damages the generation quality.
Key Results
- Substantial Overall Gains: A single round of FoR-SALE improves initial image accuracy across all tested generators:
- SD 3.5-Large: 28.80% $\rightarrow$ 31.40% (1 round) $\rightarrow$ 36.80% (3 rounds).
- FLUX.1: 29.00% $\rightarrow$ 34.30% (1 round) $\rightarrow$ 38.90% (3 rounds).
- GPT-4o: 52.20% $\rightarrow$ 54.20% (1 round) $\rightarrow$ 55.60% (3 rounds).
- Fixing Intrinsic FoR: On intrinsic prompts in FoR-LMD, FoR-SALE boosts FLUX.1 from 25.83% to 28.04% and GPT-4o from 24.35% to 38.37% (a +14.02% jump over 3 rounds).
- Ablations: Disabling facing direction modification drops FLUX.1 accuracy by 2.8%, while omitting depth modification reduces multi-round refinement gains by 2.0%.
Limitations and Future Work
- Computational Overhead: Running the LLM parser, interpreters, and vision models adds roughly 2.5$\times$ generation time (~128 seconds total per image on an A6000 GPU vs. ~50 seconds for FLUX.1 alone).
- Diffusion Model 3D Bottlenecks: Around 28% of remaining failure cases come from diffusion models failing to generate the correct facing direction, even when prompted explicitly with DiffEdit.
- Latent Blending Artifacts: Multiple overlapping edits occasionally trigger generation failure (23% of errors) or detector occlusion (18% of errors).
- Scope: Currently restricted to English and synthetic template-based spatial descriptions.
Summary updated