Abstract
Quantifying the overlap between high-dimensional datasets is a fundamental challenge in neuroscience and machine learning. We introduce the Effective Number of Shared Dimensions (ENSD), a model-free, variance-aware metric that generalizes the Participation Ratio to paired datasets. ENSD accounts for the distribution of variance across dimensions, providing a more nuanced and computationally efficient alternative to traditional methods like Canonical Correlation Analysis (CCA) and Centered Kernel Alignment (CKA). We demonstrate the utility of ENSD across diverse domains, including artificial neural networks, biological electrophysiology, and connectomics.
Introduction
In fields such as neuroscience and machine learning, researchers frequently need to compare two different "views" of the same underlying phenomenon—for example, neural activity in distinct brain regions or activations in different layers of a deep neural network. Identifying a "shared subspace" is the standard approach, yet existing tools often fall short.
Traditional methods like Canonical Correlation Analysis (CCA) provide a hard integer count of dimensions but ignore variance distribution—treating a dimension with 90% of the variance the same as one with 0.1%. Centered Kernel Alignment (CKA) offers a similarity score but lacks an interpretable "count." Meanwhile, model-based methods like Reduced Rank Regression (RRR) are computationally intensive and difficult to scale to modern "Big Science" datasets. ENSD addresses these gaps by providing a measure that is interpretable as a number of dimensions, sensitive to variance, and computationally efficient.
The ENSD Metric
ENSD is a generalization of the Participation Ratio (PR), which measures the effective dimensionality of a single dataset. For paired datasets, ENSD is defined through the trace of the product of two scaled covariance matrices.
A key advantage of ENSD is its mathematical decomposition into two intuitive components:
- Distribution of Variance (Λ): Reflects the eigenvalues of the datasets, capturing how much information is concentrated in top dimensions.
- Eigenvector Alignment (W): Measures the alignment of the principal axes between the two datasets, ranging from complete alignment to orthogonality.
This decomposition reveals a major theoretical insight: CKA is actually a normalized version of ENSD, representing the fraction of the maximum possible shared dimensionality being utilized.
Key Insights and Applications
Robustness and Efficiency
ENSD is significantly more robust to data sparsity than CCA or RRR. In simulations with high levels of missing data, ENSD maintained accurate dimensionality estimates where other methods' performance degraded significantly. Furthermore, ENSD is approximately two orders of magnitude faster to compute than RRR, making it viable for massive datasets.
Artificial Neural Networks
Analysis of Convolutional Neural Networks (CNNs) using ENSD reveals an "expansion-compression" cycle. Shared dimensionality between layers initially expands to create separable features and subsequently contracts as the network focuses on the specific classification task. While CKA scores often remain flat across layers, ENSD provides a detailed "fingerprint" of representational evolution, showing clear peaks in dimensionality at middle layers.
Biological Systems
In macaque visual cortices (V1 and V2), ENSD quantifies the shared information processing between regions. In fruit fly connectomics, ENSD identified a consistent 1D shared subspace between olfactory processing areas (the LH and MB) across individuals. This shared dimension was specifically tuned to food-related odors like yeast and fruit, suggesting a genetically hardwired functional structure rather than random connectivity.
Conclusion
The Effective Number of Shared Dimensions (ENSD) provides a powerful, efficient, and interpretable framework for comparing high-dimensional representations. By accounting for variance and alignment, it offers a more realistic view of shared information than traditional integer-based counts. Future work may extend ENSD to non-linear structures through kernelization and explore scale-dependent variations in data structure.
Summary updated