Introduction
In modern machine learning, the ability to quantify the similarity between datasets is fundamental to tasks such as evaluating generative models, assessing transfer learning potential, and analyzing high-dimensional biological data. However, comparing datasets remains a significant challenge when they are unaligned—meaning they may differ in sample size, feature sets, or even dimensionality.
Traditional approaches often rely on finding point-to-point correspondences, a problem that is computationally NP-hard. Spectral methods, such as the Intrinsic Multi-scale Distance (IMD), attempt to circumvent this by comparing the "fingerprints" (eigenvalues) of data-driven heat kernels. While effective, these methods often treat eigenvalue comparison in a pointwise Euclidean fashion, neglecting the underlying geometric structure of the operators. This paper introduces the Log-Euclidean Signature (LES), a theoretically grounded distance metric that treats datasets as points on a Riemannian manifold of symmetric positive-definite (SPD) matrices, offering a more accurate capture of intrinsic geometric structures.
The Geometric Framework: From Euclidean to Log-Euclidean
The core innovation of LES lies in its transition from standard Euclidean distances to the Log-Euclidean (LE) metric. The methodology follows a structured geometric pipeline:
- Diffusion Operators: Each dataset is first represented as a diffusion operator—an SPD matrix that serves as a discrete approximation of the Laplace-Beltrami operator on the underlying data manifold. This operator encodes the local and global connectivity of the data points.
- The Geometry of SPD Matrices: SPD matrices do not reside in a flat Euclidean space; they form a curved Riemannian manifold. Measuring distances on this manifold requires a metric that respects its curvature. While the Log-Euclidean metric is a natural choice, it typically requires matrices to be of the same dimensions and aligned.
- The Lower Bound Trick: To accommodate unaligned data of varying sizes, the authors derive a mathematically rigorous lower bound of the LE distance. This bound is calculated using only the ordered eigenvalues of the diffusion operators, enabling comparison without the need for explicit point-to-point matching.
- Scalability via Nyström Approximation: To ensure the method scales to massive datasets, the authors employ a Nyström-based fixed-rank approximation. This allows for the efficient estimation of the top $K$ eigenvalues, making the computation robust to noise and computationally feasible for large-scale applications.
Experimental Validation
The authors evaluated the LES distance across a diverse range of domains, demonstrating its versatility and robustness compared to existing spectral and optimal transport methods.
Developmental Trajectory Recovery
In a single-cell RNA-sequencing (scRNA-seq) task involving approximately 250,000 reprogrammed mouse embryonic cells, LES was used to recover developmental trajectories over an 18-day period. LES achieved a correlation of 0.74 with the actual developmental time axis. This significantly outperformed the Intrinsic Multi-scale Distance (0.28) and the Geometry Score (0.03). Notably, LES is purely intrinsic; it outperformed Optimal Transport (OT) methods even though OT had access to the gene-space coordinates, while LES relied solely on the internal manifold geometry.
Predicting Task Difficulty in Few-Shot Learning
Using few-shot learning benchmarks like CIFAR-FS and FC100, the authors used LES to measure the distance between training and test tasks. The LES distance showed a near-perfect negative correlation (-0.94) with the model's actual test accuracy. This suggests that LES can serve as a reliable diagnostic tool, warning practitioners when a new task is too geometrically distinct from the training data to expect high performance.
Alignment-Free Layer Matching
In a striking application to neural network interpretability, LES was used to match corresponding hidden layers between two different networks (LeNet on MNIST and All-CNN-C on CIFAR-10) trained with different random seeds. LES achieved a matching accuracy of 96%. This performance nearly matches Centered Kernel Alignment (CKA), but with a critical advantage: LES does not require the networks to process the same input data to perform the alignment.
Discussion and Limitations
While LES represents a significant advancement in dataset comparison, it is subject to certain theoretical and practical constraints. A primary theoretical limit is the "isospectral manifold" problem—the fact that different shapes can occasionally share the same eigenvalues (the classic "Can you hear the shape of a drum?" problem). While such cases are rare in empirical data, they represent a boundary for eigenvalue-based methods.
Additionally, the method's performance can be sensitive to the regularization parameter $\gamma$ and the number of eigenvalues $K$ used in the approximation. While the authors provide guidelines for these parameters, noisy datasets may require careful tuning.
Conclusion
The Log-Euclidean Signature provides a robust, scalable, and theoretically sound framework for comparing unaligned datasets. By leveraging the Riemannian geometry of diffusion operators, LES captures structural nuances that Euclidean spectral methods miss. Its success in biological trajectory recovery and neural network analysis suggests it is a powerful tool for diagnosing out-of-distribution shifts and understanding the geometric evolution of data representations.
Summary updated