Category Agnostic Assembly of Fractured Shapes

25 concepts15 papers1 proposal

Imagine dropping a ceramic mug on a concrete floor. It shatters into dozens of jagged, irregular shards. If you were to hand these pieces to a traditional computer vision model tasked with object assembly, it would likely fail. Most standard assembly algorithms are built on semantic priors—they recognize a "chair leg" or a "car wheel" and use template matching to place components where they belong. But a shattered shard carries no semantic label; it is an arbitrary, textureless geometry whose only clue to its original identity lies in the fine-grained topography of its broken edges.

To solve this "jigsaw puzzle from hell," a rapidly growing research direction has emerged around category-agnostic geometric fracture assembly. By abandoning high-level semantic templates and focusing entirely on local boundary features, these methods learn to snap fragments together based on physical fit alone. This shift enables algorithms to generalize to entirely unseen object categories, offering powerful tools for archaeology, robotics, and digital heritage preservation.


The Core Challenge: Moving Beyond Semantics

In standard part assembly, components have predefined structural roles. In geometric fracture assembly, however, shards lack consistent semantic meaning. As Silvia Sellán et al. demonstrate in the benchmark dataset Breaking Bad: A Dataset for Geometric Fracture and Reassembly, when an object breaks, global shape features become useless. Reassembly requires a paradigm shift from global template matching to local surface alignment.

Traditional Assembly:  [Chair Seat] + [Chair Leg]  ---> Uses Semantic Templates
Fracture Assembly:     [Jagged Shard A] + [Shard B] ---> Uses Pure Geometric Fit

This problem becomes even more pronounced in real-world scenarios. In Re-assembling the past: The RePAIR dataset and benchmark for real world 2D and 3D puzzle solving, Theodore Tsesmelis et al. introduce a dataset of digitized fresco fragments from Pompeii. Here, algorithms cannot rely on clean, synthetic breaks; they must contend with erosion, missing pieces, and sparse pictorial cues. Without functional labels or known global templates, systems must rely exclusively on low-level feature alignment.


The Lock-and-Key Mechanism: Fracture Surface Matching

If we cannot use semantic labels, how do we determine if two pieces fit together? The answer lies in the mathematical properties of the break itself.

Instead of searching for identical surface patterns, effective models look for complementary geometric features—a "lock and key" mechanism where a convexity on one fragment meets a concavity on another. Jiaxin Lu et al. formalized this in their framework, Jigsaw: Learning to Assemble Multiple Fractured Objects, which jointly learns to segment fracture surfaces and match complementary features across multiple pieces simultaneously.

   Shard A (Convexity)        Shard B (Concavity)
       _______                    _______
      /       \                  |       |
     |    >    |       +         |   <   |   ---> Perfect Fit
      \_______/                  |_______|

To extract these features without being distracted by the flat, unbroken outer walls of an object, researchers isolate the "breaking curves"—the jagged boundaries where the fracture occurred. Ali Alagrami et al. show in Reassembling Broken Objects using Breaking Curves that segmenting fragments based purely on these boundary lines allows an exhaustive search to outperform many deep learning models that try to process the entire global shape. By focusing on the mathematical properties of the break, the system avoids the generalization issues common to category-specific models.


Disentangling Shape and Pose via SE(3) Equivariance

One of the steepest hurdles in geometric assembly is that fragments are observed in random initial orientations. A neural network must be able to recognize that two surfaces match regardless of how they are rotated or translated in 3D space.

To solve this, researchers leverage SE(3) equivariance—a mathematical property ensuring that if an input shape rotates, its learned feature representation rotates in the exact same manner. Ruihai Wu et al. utilize this in Leveraging SE(3) Equivariance for Learning 3D Geometric Shape Assembly to completely disentangle the intrinsic shape of a fragment from its spatial pose.

This mathematical formulation enables correspondence-free assembly. Traditional geometric registration algorithms (like Iterative Closest Point) rely on finding overlapping surface areas. However, fractured fragments do not overlap; they touch at a zero-thickness boundary. As Ziming Wang et al. demonstrate in SE(3)-bi-equivariant Transformers for Point Cloud Assembly, learning global and local geometric features instead of point-to-point correspondences allows networks to precisely align 3D point clouds even when they share zero overlapping surface area.


Scaling Up: Multi-Fragment Agglomeration

While mating two pieces is a well-defined pairwise problem, scaling to dozens of fragments introduces a massive combinatorial explosion. Early learning-based approaches, such as the Fragment Relation Networks (FRN) introduced by Jinhwi Lee et al. in Fragment Relation Networks for Geometric Shape Assembly, addressed this by using permutation-equivariant architectures to learn selection and placement order for textureless 2D fragments.

In 3D, the state-of-the-art has evolved toward hierarchical and generative strategies:

  1. Proxy-Level Transformers: Processing raw point clouds for dozens of fragments is computationally prohibitive. Wenting Cui et al. introduced PHFormer: Multi-Fragment Assembly Using Proxy-Level Hybrid Transformer, which condenses fragments into local geometric "proxies." A hierarchical transformer then estimates poses based on these proxies, making the assembly orders of magnitude faster.
  2. Auto-Agglomerative Diffusion: Instead of predicting poses in a single step, recent work treats assembly as a denoising process. In PuzzleFusion++: Auto-agglomerative 3D Fracture Assembly by Denoise and Verify, Zhengqing Wang proposes a framework that uses a diffusion model to align fragments, paired with a transformer-based verifier that iteratively merges pieces into larger groups, mimicking how humans solve puzzles by clustering small segments first.

Real-World Impact: From Archaeology to Robotics

The ability to reason about shapes without semantic priors opens up profound applications across domains.

In cultural heritage preservation, as reviewed by Xinda Liu et al. in Review on Virtual Reality Technology in Cultural Heritage Protection, automated fracture-surface matching algorithms provide researchers with computational fit suggestions. This drastically reduces the physical handling of delicate, ancient artifacts.

[Fractured Shards] ---> [Geometric Assembly Model] ---> [Robotic Bimanual Grasping]

Furthermore, this geometric understanding is transitioning into physical manipulation. Yan Shen et al. introduced BiAssemble: Learning Collaborative Affordance for Bimanual Geometric Assembly, a framework that teaches dual-arm robots how to reassemble broken objects. By analyzing the fine-grained geometry of fractured edges, the robot learns "foresightful" grasp points—picking up a shard in a way that leaves its matching fracture surface exposed for the subsequent alignment step.

By treating reassembly as a pure problem of geometric compatibility rather than object recognition, these systems demonstrate that machines do not need to know what an object is to understand how its pieces fit back together.

Key Foundational Work

3D Fragment Reassembly Using Integrated Template Guidance and Fracture-Region Matching (2015) — contributed 1 concept elements to this direction.
Fragment Relation Networks for Geometric Shape Assembly (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_geometric_fragment_matching_04cb3bbf,
  title = {geometric fragment matching},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/geometric-fragment-matching/rdr_04cb3bbf036b43c1b92cc0c80ba997df},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_geometric_fragment_matching_04cb3bbf_v1,
  title = {geometric fragment matching},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/geometric-fragment-matching/rdr_04cb3bbf036b43c1b92cc0c80ba997df/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake