SE(n)-Invariant Flow Matching: A General Framework with Application to Object Reassembly

Gaël Heck, Sylvie Le Hégarat-Mascle, Nicolas LerméICML 2026 · OpenReview

TL;DR

This paper introduces SE(n)-Invariant Flow Matching, a novel geometric framework for reconstructing objects from fragmented parts. It tackles the challenge of global rigid motion ambiguity by enforcing invariance through "Global Gauge Fixing" and a "quotient-invariant Flow Matching" objective, leading to more accurate and robust object reassembly.

What Problem Does This Paper Address?

Reassembling fragmented objects, whether 2D puzzles or 3D fractures, is a fundamental problem in computer vision and graphics. A key challenge is that the absolute position and orientation of the reconstructed object don't matter, only its shape. This means many configurations that differ only by a global rigid transformation (translation and rotation) are equivalent. Standard machine learning approaches often struggle with this "global pose ambiguity," leading to ill-posed training problems where models learn to predict spurious global drifts instead of intrinsic shape changes. Existing methods try to circumvent this by using "anchoring" (fixing certain parts) or extensive data augmentation, but these approaches can break important symmetries and introduce biases.

Methods

The authors propose a two-pronged approach to enforce SE(n) invariance by construction:

  1. Global Gauge Fixing (GGF): This is a deterministic process that aligns configurations to a unique "canonical section" across all possible global transformations. Think of it as finding a standard way to represent each object's shape, regardless of its overall pose. They use a PCA-based approach to align principal inertia axes, effectively removing global translation and rotation degrees of freedom.
  2. Quotient-Invariant Flow Matching (FM) Objective: Standard Flow Matching trains a model to predict the velocity field that transforms random noise into a target shape. This paper modifies the FM loss by projecting the predicted velocity residuals onto the "horizontal tangent bundle." This bundle represents motions that change the shape but not the global pose. By only penalizing errors in this subspace, the model is forced to learn intrinsic shape dynamics, ignoring irrelevant global drifts. This projection is derived from the inertia-momentum equations, providing a principled way to remove global motion.

Datasets

The framework is evaluated on a variety of 2D and 3D reassembly benchmarks:

  • 2D Synthetic Puzzles:
    • SVJP (Square Voronoi Jigsaw Puzzle): Synthetic dataset with varying noise levels.
  • SVFP (Square Voronoi Fractal Puzzle): Similar to SVJP but with more realistic, fractal break contours.
  • RPLAN & MagicPlan: Real-world floor plan assembly datasets, involving room polygons and semantic labels.
  • Archaeological Reassembly:
    • RePAIR: Dataset of 2D fresco fragments with irregular geometries and erosion.
  • 3D Synthetic Fractures:
    • Breaking Bad: Large-scale synthetic dataset for 3D fracture reassembly, with "Everyday" and "Artifact" subsets.
  • Mixed Synthetic/Real Scientific Fractures:
    • FRACTURA: Dataset combining synthetic and real-world 3D fractures from various scientific domains (ceramics, bones, lithics).

Key Insights and Ideas

  • Invariance by Construction is Key: Instead of trying to "fix" symmetry issues with ad-hoc methods like anchoring or augmentation, building invariance directly into the framework (via GGF and quotient-invariant loss) leads to more principled and effective solutions.
  • Global Gauge Fixing Simplifies the Problem: By mapping all equivalent configurations to a single canonical representation, GGF effectively reduces the dimensionality of the problem, making it easier for the model to learn.
  • Focusing on Shape Dynamics: Projecting the learning objective onto the "horizontal tangent bundle" ensures the model learns only the motions that change the object's shape, ignoring irrelevant global transformations. This is crucial for accurate reassembly.
  • Geometric Rigor Pays Off: Leveraging concepts from differential geometry (tangent bundles, Lie groups) provides a solid theoretical foundation for handling complex geometric transformations and symmetries.

Key Results

The proposed SE(n)-Invariant Flow Matching framework demonstrates significant improvements across various benchmarks:

  • 2D Puzzles: Achieves state-of-the-art results on SVJP and SVFP, showing robustness to noise and degradation. For example, on SVJP with 1% noise, it achieves 0.78 Overlap, outperforming the strongest SE(2) baseline (0.69). It also shows strong performance on real-world floor plan assembly (RPLAN, MagicPlan) and archaeological reassembly (RePAIR).
  • 3D Fractures: Consistently outperforms strong baselines like GARF on Breaking Bad and FRACTURA datasets. For instance, on Breaking Bad (Everyday), it reduces RMSE(R) from 6.43 to 5.80 and RMSE(T) from 1.26 to 1.12, while increasing Part Accuracy (PA).
  • Ablation Studies: The paper shows that both Global Gauge Fixing and the quotient-invariant loss are critical for performance, with GGF providing the largest gain and the quotient loss further improving accuracy. Test-time projection also provides a consistent boost.

Limitations and Future Work

  • Computational Cost: While the framework is efficient, training can still be computationally intensive, especially for 3D tasks.
  • Degenerate Configurations: The paper acknowledges that near-degeneracies can arise in datasets, particularly with rotational symmetries or elongated structures. While their method performs well even on these subsets, it's an area for continued investigation.
  • Future Work: The authors suggest integrating domain-specific constraints into their framework, which could further enhance its applicability to real-world problems. They also highlight the potential for this geometric approach to be extended to other areas of geometric deep learning.

Summary updated

Abstract

Reassembling $N$ fragments in $n$-dimensional space is a shape reconstruction task that is invariant to global rigid motions. Training directly on $\mathcal{M}=\mathrm{SE}(n)^N$ can be ill-posed: standard losses penalize solutions that differ only by a global transform. Existing methods often address this with ad-hoc anchoring which breaks permutation invariance across fragments and can introduce biases that must be mitigated with extensive and costly data augmentation. We propose a geometric framework that enforces invariance by construction. First, a **Global Gauge Fixing** (GGF) strategy deterministically aligns configurations using an intrinsic generalized-inertia rule. Second, we introduce a **quotient-invariant Flow Matching objective** that operates via orthogonal projection onto the horizontal tangent bundle. This construction factors out global pose at each timestep, enabling the model to learn only shape-changing dynamics on the quotient space $\mathcal{M}/\mathrm{SE}(n)$. Our unified $\mathrm{SE}(n)$-invariant framework admits efficient closed-form 2D/3D instantiations and improves accuracy on polygonal jigsaw puzzles and 3D fracture reassembly benchmarks.
BibTeX
@inproceedings{heck2026se,
  author = {Gaël Heck and Sylvie Le Hégarat-Mascle and Nicolas Lermé},
  title = {{SE(n)-Invariant Flow Matching: A General Framework with Application to Object Reassembly}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=FN7KIzIQiA},
}
Figure