Canonical Correlation Guided Deep Neural Network

Zhiwen Chen, Siwen Mo, Haobin Ke, Steven X. Ding, Zhaohui Jiang, Chunhua Yang, Weihua GuiarXiv 2024 · OpenReview

Abstract

Learning representations of two views of data such that the resulting representations are highly linearly correlated is a fundamental challenge in multi-view learning. This paper introduces the Canonical Correlation Guided Deep Neural Network (CCDNN), a framework that merges multivariate analysis (MVA) with deep learning. Unlike traditional methods such as Deep CCA (DCCA), which maximize correlation as a primary objective, CCDNN treats canonical correlation as a constraint. This approach preserves the ability to learn correlated representations while focusing on specific engineering tasks like reconstruction, classification, and prediction. A key innovation is the "redundancy filter," designed to reduce information overlap between views. Experimental results on MNIST, industrial fault diagnosis, and remaining useful life (RUL) prediction datasets demonstrate that CCDNN outperforms existing methods in both accuracy and denoising performance.

Introduction

In multi-view learning, the goal is to extract meaningful features from multiple data sources—such as different sensor types on a single machine. The standard tool for this has been Canonical Correlation Analysis (CCA). However, traditional CCA and its deep variants (DCCA) often suffer from task misalignment and redundancy. By focusing solely on maximizing correlation, these models may ignore unique, task-relevant signals in favor of shared noise. CCDNN addresses these issues by reimagining correlation as a guide rather than the end goal.

Comparison of traditional CCA variants (Kernel CCA, Deep CCA) and the proposed CC Guided Deep Neural Network architecture.

Methodology

The CCDNN architecture is built on three pillars:

  1. Deep Feature Extractors: Parallel neural networks (e.g., CNNs or GRUs) map raw multi-view inputs into a latent space ($z_1, z_2$).
  2. CCA Constraint Layer: Instead of a correlation-based loss, CCDNN solves the CCA problem as a mathematical constraint during the forward pass, calculating linear mappings ($J, L$) and correlation coefficients without adding trainable parameters.
  3. Redundancy Filter (RF): This layer uses the CCA parameters to subtract the redundant, correlated components from the signals. The resulting residuals ($r_1, r_2$) contain the unique information from each view, which is then concatenated for downstream tasks.

Task-Specific Implementations

The framework is highly flexible and can be adapted for various engineering objectives. The authors demonstrate its utility across three primary tasks: image reconstruction, fault diagnosis (classification), and RUL prediction (regression).

The proposed CCDNN framework adapted for three distinct tasks: Reconstruction, Fault Diagnosis, and Remaining Useful Life (RUL) prediction.

1. Reconstruction Task

Using a modified MNIST dataset with noisy and rotated views, CCDNN employs decoders to reconstruct the original images from the filtered latent representations. By removing redundant noise through the RF layer, CCDNN achieved a Mean Squared Error (MSE) of 0.10, significantly outperforming DCCA (0.53) and DCCAE (0.19).

2. Fault Diagnosis

Tested on the SEU Bearing dataset, the model uses vibration and torque data to classify gearbox faults. The CCDNN architecture reached a diagnostic accuracy of 97.79%, showing particular strength in scenarios with limited training data compared to standard CNNs.

3. Remaining Useful Life (RUL)

For the NASA C-MAPSS turbofan engine dataset, CCDNN was integrated with Gated Recurrent Units (GRUs). The CCA-guided features allowed the model to predict the remaining lifespan of engines with lower error rates than standalone temporal models, proving that the correlation constraint helps isolate critical degradation signals.

Conclusion

CCDNN represents a significant step in integrating classical multivariate analysis with modern deep learning. By treating correlation as a constraint and introducing a redundancy filter, the model effectively balances shared information with task-specific features. While the current implementation relies on a linear redundancy filter, the framework's modularity allows for future extensions into fully nonlinear filtering and deeper residual architectures.

Summary updated

Abstract

Learning representations of two views of data such that the resulting representations are highly linearly correlated is appealing in machine learning. In this paper, we present a canonical correlation guided learning framework, which allows to be realized by deep neural networks (CCDNN), to learn such a correlated representation. It is also a novel merging of multivariate analysis (MVA) and machine learning, which can be viewed as transforming MVA into end-to-end architectures with the aid of neural networks. Unlike the linear canonical correlation analysis (CCA), kernel CCA and deep CCA, in the proposed method, the optimization formulation is not restricted to maximize correlation, instead we make canonical correlation as a constraint, which preserves the correlated representation learning ability and focuses more on the engineering tasks endowed by optimization formulation, such as reconstruction, classification and prediction. Furthermore, to reduce the redundancy induced by correlation, a redundancy filter is designed. We illustrate the performance of CCDNN on various tasks. In experiments on MNIST dataset, the results show that CCDNN has better reconstruction performance in terms of mean squared error and mean absolute error than DCCA and DCCAE. Also, we present the application of the proposed network to industrial fault diagnosis and remaining useful life cases for the classification and prediction tasks accordingly. The proposed method demonstrates superior performance in both tasks when compared to existing methods. Extension of CCDNN to much more deeper with the aid of residual connection is also presented in appendix.
BibTeX
@misc{chen2024canonical,
  author = {Zhiwen Chen and Siwen Mo and Haobin Ke and Steven X. Ding and Zhaohui Jiang and Chunhua Yang and Weihua Gui},
  title = {{Canonical Correlation Guided Deep Neural Network}},
  year = {2024},
  eprint = {2409.19396},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.19396},
}

Loading related content...

Figure