TL;DR
Current image generators are masters of local texture but struggle with global spatial reasoning. The authors introduce GJBench, a benchmark that asks models to generate scattered, rotated jigsaw pieces of a single scene, revealing that even state-of-the-art models fail to maintain scene continuity across non-adjacent regions of the canvas.
What Problem Does This Paper Address?
When an AI generates an image of a bridge, the two halves of the bridge are usually adjacent on the canvas. Because "belonging to the same object" and "being next to each other" are almost always synonymous in training data and benchmarks, it’s unclear if models truly understand a scene as a global entity or if they are just very good at making each pixel look consistent with its immediate neighbors.
The authors propose to disentangle these two properties by asking models to draw a scattered jigsaw puzzle. In this task, pieces of a single scene are moved apart and rotated on a plain background. To succeed, the model must "hold the scene together" in its internal representation even though the pieces are spatially disconnected on the output canvas.
Methods
The authors developed GJBench, which evaluates models on three distinct requirements:
- Scatter (RQ1): Does the image contain the requested number of separate pieces ($N=4, 9, 16$)?
- Interlock (RQ2): Do the shapes of the pieces actually fit together? The authors use a mixed-integer program (the HiGHS solver) to find the best possible physical arrangement of the generated pieces and score the fit using a Dice overlap metric.
- Reconstruction (RQ3): Does the content of the pieces, when reassembled, match a reference scene? This is measured using Multi-Scale Structural Similarity (MS-SSIM) and panoptic quality metrics.
The benchmark tests models under four conditions:
- TEXT: Only a text prompt describing the scene.
- REF: Text prompt plus a reference image of the whole scene.
- LAYOUT: Text prompt plus a target layout showing piece outlines.
- REF+LAYOUT: Both a reference image and a target layout.
Crucially, the evaluation is deterministic and pixel-based, avoiding the biases and "blind spots" often found in LLM-based or VLM-based judges.
Datasets
- GJBench: A new benchmark consisting of 32 curated scenes from Wikimedia Commons (CC0). The scenes are divided into four categories to test different spatial properties: lines and topology (roads/rivers), object parts, multi-object relations (counting/relative positions), and scene geometry/lighting.
- Models Evaluated: FLUX 3 Image Klein, Qwen-Image-2.1, GPT Image 2.6, Gemini Nano Banana 2.1, and Muse Image 1.0.
Key Insights and Ideas
- Local vs. Global Coherence: Every model evaluated could draw the scenes perfectly as a "whole" image. However, every model failed to maintain that coherence once the scene was broken into scattered pieces.
- Layout Helps Shape, Not Content: Providing a target layout (the outlines of the pieces) significantly improved the Interlock score, suggesting models can copy shapes. However, it did not improve Reconstruction, meaning models still couldn't place the right part of the scene into the right piece.
- The Reference Image Paradox: Surprisingly, providing a reference image of the whole scene did not help models count pieces more accurately or make them fit together better. In some cases, it even made performance worse.
- Failure Modes: Models failed in distinct ways. Some (like Qwen) tended to merge pieces together, while others (like Nano Banana) generated extra pieces. A common "cheat" was "thumbnailing"—drawing the entire scene in miniature inside each individual puzzle piece.
Key Results
- GPT Image 2.6 was the most reliable at following the "scatter" and "interlock" instructions, but its reconstruction of scene content was still poor.
- FLUX 3 struggled the most with interlocking; its pieces rarely had matching tabs and sockets unless a layout was explicitly provided.
- Quantitative Gap: While the "control" (exact cuts of real images) scored ~0.94 on interlock and reconstruction, the best models typically scored below 0.50 on reconstruction.
- Content Placement is Hardest: Even when models were given square tiles (no complex tabs) and told exactly where they belonged, they still couldn't accurately map the reference scene onto those tiles.
Limitations and Future Work
- Proprietary "Black Boxes": Some models (like FLUX 3) use prompt upsamplers that rewrite user instructions. This makes it difficult to determine if the failure lies in the language understanding stage or the image generation stage.
- Scale: The benchmark currently uses 32 scenes. Expanding this to more diverse content families would provide more robust statistics.
- Future Directions: The authors suggest developing "reference-free" measures of spatial continuity, such as checking if the edges of scattered pieces are "mutual best matches" for each other, similar to how jigsaw puzzle solvers operate.
Summary updated