TL;DR
BrickFlow is a generative SE(3) flow-matching model designed to reconstruct LEGO assemblies from a single image and a known inventory of parts. By pretraining on a massive procedural dataset and employing a novel, analytic "connectivity-field" guidance at test time, the system can infer complex 3D structures that satisfy discrete physical constraints.
What Problem Does This Paper Address?
Reconstructing a LEGO set from a photo is significantly harder than standard 3D reconstruction. While a typical AI might generate a "LEGO-like" mesh or voxel grid, a true reconstruction must identify the exact 6D pose (position and orientation) of every individual brick in a provided inventory.
This presents two major challenges:
- Combinatorial Complexity: With thousands of possible part types and near-infinite ways to connect them, the search space is astronomical.
- Discrete Constraints: LEGO bricks only "fit" in specific, discrete configurations (e.g., a stud must align perfectly with a hole). Even a sub-millimeter pose error results in a physically impossible build.
Existing methods often struggle with these "zero-measure" valid configurations or are limited to a very small subset of simple rectangular bricks.
Methods
The authors propose a multi-stage approach called BrickFlow:
- SE(3) Flow Matching: The core model is a Diffusion Transformer (DiT) trained using flow matching. It learns to "denoise" a random pile of bricks into a structured assembly. Because bricks exist in 3D space, the model operates on the $SE(3)$ manifold, handling both 3D translations and rotations.
- Image Conditioning: To guide the assembly toward a specific target, the model is conditioned on image features extracted from a frozen DINOv3 encoder, processed through a Perceiver resampler.
- Connectivity-Field Guidance: This is the paper's most novel contribution. Since a neural network rarely predicts a perfectly aligned pose, the authors introduce an analytic energy function at test time. They model the assembly as an electrical network where bricks are nodes and potential connections are conductances. The gradient of this "connectivity energy" pulls bricks toward valid "seats" (connections) during the sampling process, ensuring the final output actually snaps together.
Datasets
- ProcBrick (New): A massive procedural dataset created by the authors. It contains 100 million unique assemblies of 100 parts each, generated by a "physical grammar" that ensures connectivity. A subset of 1 million rendered image-structure pairs is used for finetuning.
- BrickNet: A validation set of 512 human-designed LEGO models used to test the model's ability to generalize to realistic, non-procedural structures.
- LDraw: The underlying library of digital LEGO part replicas used for geometry and connector definitions.
Key Insights and Ideas
- Learning the "Grammar" of Assembly: The authors found that pretraining on "nonsense" procedural assemblies (which don't look like real objects but follow physical rules) is highly effective. It teaches the model the global "prior" of how LEGO bricks tend to cluster and align before it ever sees a real image.
- Analytic Guidance over Learning: Rather than trying to force the model to learn the exact geometry of every stud and hole (which is computationally expensive and prone to error), they use the known geometry of the connectors to create a differentiable "guidance" field. This allows the model to focus on the "where" while the guidance handles the "exactly how."
- Global vs. Local Connectivity: By modeling the assembly as an electrical network, the connectivity field ensures the entire object is connected, rather than just ensuring individual pairs of bricks touch.
Key Results
- Validity Boost: On the BrickNet benchmark, the largest model (IC DiT-L) with connectivity guidance achieved a 35.1% validity rate, compared to much lower scores for unguided or smaller models.
- Scaling: The task scales well with model capacity; moving from a "Small" to a "Large" transformer significantly improved the structural coherence of the builds.
- Real-World Generalization: Despite being trained only on synthetic renders, BrickFlow successfully reconstructed LEGO models from actual photographs of physical builds.
- Connectivity: Guidance roughly halved the number of disconnected components in large assemblies, proving that the "electrical network" analogy effectively pulls stray bricks into the main structure.
Limitations and Future Work
- Exact Inventory Assumption: The model currently assumes you have exactly the right bricks. It cannot yet handle "the bucket problem"—where you have extra bricks or are missing a few.
- Dense Builds: The model struggles with very dense internal structures where many identical bricks are packed together, leading to ambiguities.
- Scale: While 100 parts is impressive for current AI, real-world LEGO sets often contain thousands of pieces. Scaling to these sizes will require more efficient attention mechanisms and perhaps hierarchical assembly strategies.
- Collisions: The current connectivity field doesn't explicitly penalize bricks overlapping (collisions), though the authors note that pulling bricks into valid seats often resolves collisions indirectly.
Summary updated