Linear Optimal Transport Subspaces for Point Set Classification

Mohammad Shifat‐E‐Rabbi, Naqib Sad Pathan, Shiying Li, Yan Zhuang, Abu Hasnat Mohammad Rubaiyat, Gustavo K. Rohde

TL;DR

This paper introduces a novel framework for classifying point sets, even when they undergo significant spatial deformations like affine transformations. By leveraging the Linear Optimal Transport (LOT) transform, they create a convex data space that simplifies classification, leading to a highly efficient and robust method that outperforms existing approaches, especially in challenging out-of-distribution scenarios.

What Problem Does This Paper Address?

Classifying point sets, which represent 3D object geometry, is crucial for many computer vision tasks. However, point sets are inherently challenging to work with due to their unordered nature, permutation invariance, and susceptibility to spatial deformations (like rotations, translations, and scaling). Existing methods often struggle with these deformations, require large amounts of training data, and can be vulnerable to variations in data distribution. This paper aims to address these limitations by developing a classification method that is robust to spatial deformations and efficient in terms of data requirements.

Methods

The core of the proposed method lies in the Linear Optimal Transport (LOT) transform. This transform maps a point set distribution to a linear embedding in a high-dimensional space. The key insight is that under certain conditions, point set classes, when represented in this LOT space, form convex subspaces. This convexity significantly simplifies the classification problem.

The framework involves:

  • LOT Transform: A mathematical tool that linearizes the optimal transport problem, providing a consistent embedding for point sets.
  • Generative Model: A model that defines point set classes as instances of a template pattern under unknown spatial deformations.
  • Convex Subspace Approximation: The LOT space allows for the approximation of these data classes as convex subspaces.
  • Nearest Subspace Classifier: A classification algorithm that finds the closest subspace to a given test point set in the LOT space.
  • Invariance Encoding: The method explicitly incorporates mechanisms to handle common affine transformations (translation, isotropic/anisotropic scaling, shear) by defining specific spanning sets for these transformations within the LOT space.

This approach is non-iterative and requires no hyper-parameter tuning.

Datasets

The paper evaluates its method on several datasets:

  • Synthetic Datasets: Generated from point cloud MNIST and ShapeNet datasets by applying random translations, anisotropic scaling, and shear transformations.
  • Real Datasets:
    • Point cloud MNIST
    • ModelNet
    • ShapeNet

For experiments, they also created modified versions of these datasets by applying random translations, anisotropic scaling, and shear transformations to both training and test sets to simulate out-of-distribution scenarios.

Key Insights and Ideas

  • LOT Space as a Convex Domain: The most significant insight is that the LOT transform can map complex, non-linear point set data into a linear space where class distributions become approximately convex subspaces. This is a powerful simplification for classification.
  • Data Efficiency: The nearest subspace classifier in the LOT space is shown to be highly data-efficient, achieving good accuracy with fewer training samples compared to other methods.
  • Robustness to Deformations: By explicitly encoding invariances to affine transformations, the method demonstrates remarkable robustness to variations in deformation magnitudes, even when training and test distributions differ significantly (out-of-distribution settings).
  • Unified Framework: The LOT transform provides a unified mathematical framework to handle various types of spatial deformations, making it a versatile tool for point set analysis.

Key Results

  • Synthetic Datasets: The proposed method substantially outperformed other methods (PointNet, DGCNN, FSpool-MLP) on synthetic datasets, achieving higher test accuracy.
  • Real Datasets: On MNIST, ModelNet, and ShapeNet, the proposed method consistently achieved higher test accuracy across different numbers of training samples per class, often with a smoother accuracy-vs-training size curve.
  • Out-of-Distribution (OOD) Robustness: In OOD experiments (where deformation magnitudes differed between training and testing), the proposed method maintained high accuracy, while other methods saw significant declines. This highlights its superior generalization capabilities. For instance, on ShapeNet, the OOD accuracy was close to the standard experimental setup accuracy, whereas other methods' accuracy dropped considerably.
  • Comparison with Set-Embedding Methods: The proposed method also outperformed classical machine learning techniques (LR, k-SVM, MLP, NS) when combined with various set-to-vector embedding methods (GeM, COVpool, FSpool).

Limitations and Future Work

  • Generative Model Assumptions: The effectiveness of the method relies on the assumption that point set classes can be represented by the proposed generative model and that the LOT space allows for convex subspace approximation. If these assumptions are not met, performance might degrade.
  • Future Work:
    • Alternative Subspace Assumptions: Exploring different ways to model subspaces, such as using linear convex hulls instead of linear combinations, could be beneficial for certain problems.
  • Alternative Distances: Investigating other distance metrics, like the sliced-Wasserstein distance with discrete CDT transform, in conjunction with subspace models.
  • Broader Applicability: The authors believe their approach is applicable to many point set classification problems and could inspire further research in 3D object recognition and scene understanding.

Summary updated

Abstract

Learning from point sets is an essential component in many computer vision and machine learning applications. Native, unordered, and permutation invariant set structure space is challenging to model, particularly for point set classification under spatial deformations. Here we propose a framework for classifying point sets experiencing certain types of spatial deformations, with a particular emphasis on datasets featuring affine deformations. Our approach employs the Linear Optimal Transport (LOT) transform to obtain a linear embedding of set-structured data. Utilizing the mathematical properties of the LOT transform, we demonstrate its capacity to accommodate variations in point sets by constructing a convex data space, effectively simplifying point set classification problems. Our method, which employs a nearest-subspace algorithm in the LOT space, demonstrates label efficiency, non-iterative behavior, and requires no hyper-parameter tuning. It achieves competitive accuracies compared to state-of-the-art methods across various point set classification tasks. Furthermore, our approach exhibits robustness in out-of-distribution scenarios where training and test distributions vary in terms of deformation magnitudes.
BibTeX
@article{shifaterabbi2025linear,
  author = {Mohammad Shifat‐E‐Rabbi and Naqib Sad Pathan and Shiying Li and Yan Zhuang and Abu Hasnat Mohammad Rubaiyat and Gustavo K. Rohde},
  title = {{Linear Optimal Transport Subspaces for Point Set Classification}},
  journal = {Journal of Mathematical Imaging and Vision},
  year = {2025},
  doi = {10.1007/s10851-025-01261-x},
}
Figure