Revisiting Transformation Invariant Geometric Deep Learning: An Initial Representation Perspective

Ziwei Zhang, Xin Wang, Zeyang Zhang, Peng Cui, Wenwu Zhu

TL;DR

This paper introduces TinvNet, a novel plug-in for geometric deep learning models that ensures transformation invariance (like rotation, translation, and scaling) by modifying the initial point representations. This approach is simpler and more general than existing methods that require complex network layer designs, offering a strong baseline for future research in this area.

What Problem Does This Paper Address?

Geometric data, such as point clouds and graphs, is prevalent in many applications. A crucial challenge in processing this data with deep learning is ensuring that the models are invariant to various transformations (e.g., rotating an object shouldn't change its classification). While many Graph Neural Networks (GNNs) achieve permutation invariance (meaning the order of nodes doesn't matter), they often fail to handle other geometric transformations. Existing methods to achieve this invariance typically involve designing highly specialized and computationally expensive neural network layers, making them difficult to extend and integrate. This paper aims to address this by proposing a simpler, more general approach that focuses on transforming the initial data representation rather than modifying the core network architecture.

Methods

The core idea of TinvNet is to create "transformation-invariant and distance-preserving initial point representations." The authors achieve this by adapting Multi-Dimensional Scaling (MDS), a dimensionality reduction technique. Specifically, they:

  1. Construct a similarity matrix: This is derived from the pairwise Euclidean distances between points.
  2. Center the similarity matrix: This step prepares it for eigen-decomposition.
  3. Perform eigen-decomposition: This yields eigenvalues and eigenvectors.
  4. Normalize the representation: A crucial step involves normalizing the eigenvectors based on the largest eigenvalue. This normalization ensures that the resulting initial representation is both transformation-invariant and preserves relative distances between points, even under scaling transformations.

This transformed representation is then fed into any existing neural network (like a GNN or a general deep learning model) for further processing. The authors mathematically prove that this approach guarantees transformation invariance. They also extend their method to handle transformation equivariance.

Datasets

The paper evaluates TinvNet on several standard datasets:

  • ModelNet40: A widely used dataset for 3D object classification.
  • ShapeNet: Used for object part segmentation.
  • Synthetic datasets: Generated for testing specific properties like eigenvalue multiplicity.
  • TSP and CVRP instances: Datasets for combinatorial optimization problems (Travelling Salesman Problem and Capacitated Vehicle Routing Problem).

Key Insights and Ideas

  • Initial Representation is Key: The most significant insight is that achieving transformation invariance doesn't necessarily require complex architectural changes in the neural network itself. Instead, focusing on transforming the initial data representation is sufficient and more general.
  • MDS as a Powerful Tool: Adapting Multi-Dimensional Scaling (MDS) proves to be a surprisingly effective and straightforward way to generate these invariant initial representations.
  • Distance Preservation Matters: The method's ability to preserve relative distances between points is crucial for maintaining information content while achieving invariance, especially under scaling.
  • Generality and Simplicity: TinvNet is designed as a "plug-in," meaning it can be easily integrated with various existing neural network architectures without significant modifications, making it a versatile tool.

Key Results

  • Strict Transformation Invariance: TinvNet demonstrably achieves strict invariance to translation, rotation, reflection, and scaling across all tested tasks.
  • Point Cloud Analysis:
    • On ModelNet40 classification, TinvNet combined with DGCNN achieved 89.5% accuracy across all rotation settings (z/z, SO3/SO3, z/SO3), outperforming or matching specialized rotation-invariant models.
  • For object part segmentation on ShapeNet, TinvNet also showed strong performance, particularly in challenging rotation settings.
  • Combinatorial Optimization:
    • For TSP and CVRP, TinvNet maintained consistent performance across different transformations, while the baseline GAT (Graph Attention Network) showed significant performance degradation. For instance, in TSP-20, TinvNet achieved a route length of 3.87±0.01 across all transformations, whereas GAT's performance varied drastically.
  • Efficiency: The time complexity for computing initial representations is O(N²d), which is manageable and can be pre-computed.

Limitations and Future Work

  • Eigenvector Sign Ambiguity: The authors acknowledge that eigenvectors can have arbitrary signs (e.g., u and -u are valid). They propose data augmentation by enumerating all possible sign combinations as a solution, which adds computational cost. They suggest this is a minor issue for typical dimensions.
  • Eigenvalue Multiplicity: While not observed in their real-world datasets, eigenvalue multiplicity can pose challenges for obtaining unique eigenvectors. This is left as future work.
  • Similarity Transformations Only: The current TinvNet method is designed for similarity transformations. Extending it to handle more general affine transformations is a direction for future research.
  • Non-Euclidean Metrics: The current formulation assumes Euclidean distance. Adapting it for non-Euclidean metrics would require further investigation.
  • Equivariance Extension: The paper successfully extends TinvNet to handle transformation equivariance, which is important for tasks like object detection.

Summary updated

Abstract

Deep neural networks have achieved great success in the last decade. When designing neural networks to handle the ubiquitous geometric data such as point clouds and graphs, it is critical that the model can maintain invariance towards various transformations such as translation, rotation, and scaling. Most existing graph neural network (GNN) approaches can only maintain permutation-invariance, failing to guarantee invariance with respect to other transformations. Besides GNNs, other works design sophisticated transformation-invariant layers, which are computationally expensive and difficult to be extended. In this paper, we revisit why general neural networks cannot maintain transformation invariance. Our findings show that transformation-invariant and distance-preserving initial point representations are sufficient to achieve transformation invariance rather than needing sophisticated neural layer designs. Motivated by these findings, we propose Transformation Invariant Neural Networks (TinvNet), a straightforward and general plug-in for geometric data. Specifically, we realize transformation invariant and distance-preserving initial point representations by modifying multi-dimensional scaling and feed the representations into existing neural networks. We prove that TinvNet can strictly guarantee transformation invariance, being general and flexible enough to be combined with the existing neural networks. Extensive experimental results on point cloud analysis and combinatorial optimization demonstrate the effectiveness and general applicability of our method. We also extend our method into equivariance cases. Based on the results, we advocate that TinvNet should be considered as an essential baseline for further studies of transformation-invariant geometric deep learning.
BibTeX
@article{zhang2025revisiting,
  author = {Ziwei Zhang and Xin Wang and Zeyang Zhang and Peng Cui and Wenwu Zhu},
  title = {{Revisiting Transformation Invariant Geometric Deep Learning: An Initial Representation Perspective}},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2025},
  doi = {10.1109/tpami.2025.3626735},
}
Figure