Abstract
Geometric deep learning has achieved significant success in handling geometric data like point clouds and graphs. A critical inductive bias for these models is maintaining invariance toward transformations such as translation, rotation, and scaling. While existing approaches often rely on computationally expensive, custom-built layers or data augmentation, this paper explores whether transformation-invariant and distance-preserving initial representations are sufficient. We propose Transformation Invariant Neural Networks (TinvNN), a general framework that utilizes a modified Multi-Dimensional Scaling (MDS) technique to achieve strict transformation invariance. By normalizing eigenvalues to handle scaling, TinvNN can be combined with any standard neural network backbone. Our results on point cloud analysis and combinatorial optimization demonstrate that TinvNN achieves state-of-the-art performance with minimal architectural complexity.
Introduction
In geometric deep learning, the absolute coordinates of data points are often arbitrary. A 3D model of a chair remains a chair regardless of its orientation, position, or size. However, standard neural networks like PointNet or Graph Neural Networks (GNNs) typically treat transformed coordinates as entirely new inputs. This leads to poor generalization when models encounter rotations or scales not present in the training set.
Historically, this has been addressed through data augmentation, manual feature engineering (e.g., using relative distances), or sophisticated equivariant architectures. These methods are often computationally heavy, lose global structural information, or are difficult to adapt to new baseline architectures. TinvNN addresses these issues by shifting the focus from complex layer design to the initial data representation.
Methodology: The TinvNet Framework
TinvNet acts as a universal pre-processing plug-in that transforms raw coordinates into an invariant space before they reach the neural network backbone. The process involves four key steps:
- Distance Matrix Calculation: Pairwise Euclidean distances between all points are computed to capture the geometric structure.
- Modified MDS: Classical Multi-Dimensional Scaling is applied to convert the distance matrix into a similarity matrix, followed by eigen-decomposition to find a new coordinate basis.
- Eigenvalue Normalization: Standard MDS is invariant to rotation and translation but not scaling. TinvNet introduces eigenvalue normalization to preserve relative distance ratios, ensuring invariance to uniform scaling.
- Backbone Integration: The resulting invariant representations are fed into standard backbones like PointNet, DGCNN, or Graph Attention Networks (GAT).
Additionally, the framework can be extended to achieve equivariance by using the Kabsch-Umeyama algorithm at the output layer to reverse the initial transformation mapping.
Experimental Results
Point Cloud Analysis
The model was tested on ModelNet40 (classification) and ShapeNet (segmentation). In a challenging "extreme rotation" scenario—where models are trained on z-aligned objects but tested on arbitrary 3D rotations—standard models fail significantly. For instance, a standard DGCNN's accuracy drops to 22.3%. With the TinvNet plug-in, the same DGCNN achieves 89.5% accuracy, matching or exceeding the performance of specialized, mathematically complex invariant networks.
Combinatorial Optimization
TinvNet was also applied to the Travelling Salesman Problem (TSP) and Capacitated Vehicle Routing Problem (CVRP). Standard GAT models show extreme sensitivity to coordinate shifts; for TSP-100, a simple translation causes the error gap to jump from 9.4% to over 570%. TinvNet+GAT maintains a stable 9.8% error gap across all similarity transformations, including rotation, scaling, and reflection.
Discussion and Insights
- Decoupling Invariance: TinvNN proves that invariance can be achieved at the representation level, allowing researchers to use the most powerful standard architectures without needing to redesign layers for equivariance.
- Efficiency: The MDS pre-processing is highly efficient, taking less than 50ms for a 2,048-point cloud on a standard GPU.
- Limitations: The current framework is limited to similarity transformations and does not yet handle non-uniform affine transformations (like shearing). Additionally, the sign ambiguity of eigenvectors requires a small amount of sign-flip data augmentation to ensure the network remains robust to different eigen-bases.
In conclusion, TinvNN provides a simple yet mathematically rigorous baseline for transformation-invariant geometric deep learning, suggesting that sophisticated initial representations may indeed be "all you need."
Summary updated