Canonical Correlation Neural Networks

Andrea Cavallo, Sundeep Prabhakar Chepuri, Charilaos Kanatsoulis, Elvin IsufiarXiv 2026 · arXiv

In the world of deep learning, "inductive biases"—like the grid structure in images for CNNs or the connectivity in graphs for GNNs—are the secret sauce that allows models to generalize from limited data. But what happens when your data doesn't have an obvious pre-defined structure?

A new paper from researchers at Delft University of Technology, the Indian Institute of Science, and Stanford University introduces CAnonical correlation NEural networks (CANEs). This architecture leverages data statistics—specifically covariance matrices—as a relational structure to process multi-view data more effectively and robustly than classical statistical methods or standard neural networks.

TL;DR

CANEs are a new class of neural networks that use in-view and cross-view covariance matrices as "relational inductive biases" to solve multi-view tasks. By replacing the rigid, often unstable operations of classical Canonical Correlation Analysis (CCA) with learnable polynomial filters, CANEs achieve state-of-the-art performance on brain imaging and motion sensing tasks while remaining remarkably robust to noisy data.

What Problem Does This Paper Address?

Many real-world problems involve "multi-view" data: observing the same phenomenon through different lenses. Think of a brain scan that provides both cortical thickness and volume measurements, or a person wearing both an accelerometer and a gyroscope.

To process this, researchers often turn to Canonical Correlation Analysis (CCA), a century-old statistical tool that finds a shared space where these views are most correlated. However, CCA has three major flaws:

  1. Task-Agnosticism: It maximizes correlation regardless of whether that correlation is actually useful for your final goal (like predicting a patient's age).
  2. Instability: It relies on matrix inverses and singular value decompositions (SVD), which can explode if your data is noisy or if you have few samples.
  3. Linearity: Standard CCA can't capture complex, non-linear relationships.

While deep learning versions of CCA exist, they often ignore the internal "relational" structure within each view or suffer from complex training dynamics.

Methods

CANEs solve these issues by embedding the logic of CCA directly into a neural network architecture. The model operates in two distinct stages:

  1. In-view Covariance Stage: Each view is processed independently using coVariance Filters (VFs). These are polynomial filters that operate in the eigenspace of the view's own covariance matrix. This stage generalizes the "whitening" step of CCA but allows the network to learn which parts of the internal signal to emphasize.
  2. Cross-view Covariance Stage: The model computes a cross-view covariance matrix to link the views. It then applies Cross-view coVariance Filters (CVFs) to integrate information across modalities.
Figure 1: Diagram of the CANE with 2 layers per stage (i.e., Lv = Lc = 2) and filterbanks of size 1 (i.e., Fℓ= 1 ∀ℓ). The input is first processed by the in-view covariance filters independently (in-view covariance stage). Then, the cross-view covariance matrix ˆA12 is computed to merge the two views, and the cross-view covariance filters further process the data via cross-view correlations, generating one output for each view (cross-view covariance stage). At test time, the matrix ˆA12 is computed from the training data.

The "magic" here is that these filters are learnable polynomials. Instead of the hard "keep/discard" thresholding used in classical SVD-based methods, CANEs learn a smooth frequency response.

Figure 2: CCA performs two consecutive filtering operations: first, the filter hw(ˆλ) = ˆλ−1/2 acts on the covariance eigenvalues of each view to perform whitening (left); then, the high-pass filter hcca(ˆσ2) acts on the cross-view covariance singular values to select the top-D directions (right). CAFs replace these operations with learnable polynomial filters: the VF hv(ˆλ) in (3) and the CVF hc(ˆσ2) in (4).

Datasets

The authors evaluated CANEs on seven real-world datasets across two domains:

  • Brain MRI (Regression): Predicting a patient's age from morphological features (thickness and volume) across 60+ brain regions. Datasets: ADNI1, ADNI2, ADNI3, and ABIDE.
  • Motion Sensing (Classification): Identifying human activities (e.g., walking, climbing stairs) from sensors at different body locations. Datasets: MHEALTH, REALDISP, and DSADS.

Key Insights and Ideas

  • Stability through Lipschitz Continuity: The authors prove mathematically that CANEs are more stable than CCA. While CCA's error is governed by the "singular-value gap" (which can be tiny and lead to huge errors), CANE's error depends on the Lipschitz constants of its filters. Because these are controllable during training, the model is much more robust to "finite-sample errors"—the noise that comes from estimating covariances from limited data.
  • Parameter Efficiency: Because the filters are defined by a few polynomial coefficients rather than weights for every single input feature, CANEs are incredibly lean. In some tests, they used 8x fewer parameters than standard baselines while achieving better results.
  • Principled Design: The paper shows that the classical CCA transform is just one specific point in the space of functions a CANE can learn. This means the model is "rooted" in solid statistics but has the flexibility of deep learning.

Key Results

  • Top Performance: CANEs achieved the best performance on 6 out of 7 datasets, outperforming single-view models, concatenated multi-view models, and deep task-aware CCA variants.
  • Robustness to Small Samples: In a "stress test," the researchers re-estimated the covariance matrices using only a fraction of the training data. While classical CCA-based methods saw their performance plummet, CANEs remained stable, confirming their theoretical robustness.
  • Ablation Success: The authors showed that both the in-view and cross-view stages are necessary; removing either one led to a significant drop in accuracy.

Limitations and Future Work

While promising, the authors note a few areas for improvement:

  • Two-View Limit: The current framework is designed for two views. Extending this to "multi-view" (3+ views) in a computationally efficient way is a non-trivial next step.
  • High-Dimensional Bottleneck: Estimating covariance matrices for extremely high-dimensional data (e.g., raw images) is computationally expensive. Future work could combine CANEs with sparse or regularized covariance estimators.
  • Transferability: While the model is equivariant to feature permutations, more work is needed to see how well these learned covariance-based rules transfer across entirely different domains.

For those interested in the math or implementation, the authors have made their code available on GitHub.

Summary updated

Abstract

Relational inductive biases such as grids or graphs allow neural networks with fewer trainable parameters and improved generalization, but require a known structure beforehand. When no structure is available, the covariance matrix provides a principled substitute, and coVariance Neural Networks (VNNs) have shown that using it yields models that are theoretically grounded in Principal Component Analysis, stable to finite-sample estimation errors and parameter-efficient. This framework, however, is limited to single-view data, whereas many applications observe the same phenomenon through multiple views, each with its own in-view covariance and a cross-view covariance linking them. To incorporate these matrices as inductive biases in a principled way, we introduce CAnonical correlation NEural networks (CANEs), architectures that use in-view and cross-view covariances as distinct relational biases to solve downstream tasks. Via spectral analysis, CANEs draw analogies with Canonical Correlation Analysis (CCA), a classical technique to project two-view data onto a correlated subspace. However, while CCA's objective is to compute shared view representations, CANEs are trained to solve a downstream task directly. We bound their output error under finite-sample covariance perturbations and show that it depends on controllable Lipschitz constants of the learned filters in place of the singular-value gap that governs the stability of CCA, thus guaranteeing better robustness. Experiments on seven datasets from brain imaging and motion sensing show that CANEs match or outperform single-view, two-view and CCA-based baselines, often with fewer parameters, and retain their performance when covariances are re-estimated from fewer samples.
BibTeX
@misc{cavallo2026canonical,
  author = {Andrea Cavallo and Sundeep Prabhakar Chepuri and Charilaos Kanatsoulis and Elvin Isufi},
  title = {{Canonical Correlation Neural Networks}},
  year = {2026},
  eprint = {2611.07212},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07212},
}
Figure