In the world of deep learning, "inductive biases"—like the grid structure in images for CNNs or the connectivity in graphs for GNNs—are the secret sauce that allows models to generalize from limited data. But what happens when your data doesn't have an obvious pre-defined structure?
A new paper from researchers at Delft University of Technology, the Indian Institute of Science, and Stanford University introduces CAnonical correlation NEural networks (CANEs). This architecture leverages data statistics—specifically covariance matrices—as a relational structure to process multi-view data more effectively and robustly than classical statistical methods or standard neural networks.
TL;DR
CANEs are a new class of neural networks that use in-view and cross-view covariance matrices as "relational inductive biases" to solve multi-view tasks. By replacing the rigid, often unstable operations of classical Canonical Correlation Analysis (CCA) with learnable polynomial filters, CANEs achieve state-of-the-art performance on brain imaging and motion sensing tasks while remaining remarkably robust to noisy data.
What Problem Does This Paper Address?
Many real-world problems involve "multi-view" data: observing the same phenomenon through different lenses. Think of a brain scan that provides both cortical thickness and volume measurements, or a person wearing both an accelerometer and a gyroscope.
To process this, researchers often turn to Canonical Correlation Analysis (CCA), a century-old statistical tool that finds a shared space where these views are most correlated. However, CCA has three major flaws:
- Task-Agnosticism: It maximizes correlation regardless of whether that correlation is actually useful for your final goal (like predicting a patient's age).
- Instability: It relies on matrix inverses and singular value decompositions (SVD), which can explode if your data is noisy or if you have few samples.
- Linearity: Standard CCA can't capture complex, non-linear relationships.
While deep learning versions of CCA exist, they often ignore the internal "relational" structure within each view or suffer from complex training dynamics.
Methods
CANEs solve these issues by embedding the logic of CCA directly into a neural network architecture. The model operates in two distinct stages:
- In-view Covariance Stage: Each view is processed independently using coVariance Filters (VFs). These are polynomial filters that operate in the eigenspace of the view's own covariance matrix. This stage generalizes the "whitening" step of CCA but allows the network to learn which parts of the internal signal to emphasize.
- Cross-view Covariance Stage: The model computes a cross-view covariance matrix to link the views. It then applies Cross-view coVariance Filters (CVFs) to integrate information across modalities.
The "magic" here is that these filters are learnable polynomials. Instead of the hard "keep/discard" thresholding used in classical SVD-based methods, CANEs learn a smooth frequency response.
Datasets
The authors evaluated CANEs on seven real-world datasets across two domains:
- Brain MRI (Regression): Predicting a patient's age from morphological features (thickness and volume) across 60+ brain regions. Datasets: ADNI1, ADNI2, ADNI3, and ABIDE.
- Motion Sensing (Classification): Identifying human activities (e.g., walking, climbing stairs) from sensors at different body locations. Datasets: MHEALTH, REALDISP, and DSADS.
Key Insights and Ideas
- Stability through Lipschitz Continuity: The authors prove mathematically that CANEs are more stable than CCA. While CCA's error is governed by the "singular-value gap" (which can be tiny and lead to huge errors), CANE's error depends on the Lipschitz constants of its filters. Because these are controllable during training, the model is much more robust to "finite-sample errors"—the noise that comes from estimating covariances from limited data.
- Parameter Efficiency: Because the filters are defined by a few polynomial coefficients rather than weights for every single input feature, CANEs are incredibly lean. In some tests, they used 8x fewer parameters than standard baselines while achieving better results.
- Principled Design: The paper shows that the classical CCA transform is just one specific point in the space of functions a CANE can learn. This means the model is "rooted" in solid statistics but has the flexibility of deep learning.
Key Results
- Top Performance: CANEs achieved the best performance on 6 out of 7 datasets, outperforming single-view models, concatenated multi-view models, and deep task-aware CCA variants.
- Robustness to Small Samples: In a "stress test," the researchers re-estimated the covariance matrices using only a fraction of the training data. While classical CCA-based methods saw their performance plummet, CANEs remained stable, confirming their theoretical robustness.
- Ablation Success: The authors showed that both the in-view and cross-view stages are necessary; removing either one led to a significant drop in accuracy.
Limitations and Future Work
While promising, the authors note a few areas for improvement:
- Two-View Limit: The current framework is designed for two views. Extending this to "multi-view" (3+ views) in a computationally efficient way is a non-trivial next step.
- High-Dimensional Bottleneck: Estimating covariance matrices for extremely high-dimensional data (e.g., raw images) is computationally expensive. Future work could combine CANEs with sparse or regularized covariance estimators.
- Transferability: While the model is equivariant to feature permutations, more work is needed to see how well these learned covariance-based rules transfer across entirely different domains.
For those interested in the math or implementation, the authors have made their code available on GitHub.
Summary updated