Deep Co-Attention Network for Multi-View Subspace Learning

Lecheng Zheng, Yu Cheng, Hongxia Yang, Nan Cao, Jingrui He

Abstract

Multi-view learning is essential for processing data from heterogeneous sources, such as medical imaging or multi-modal social media content. However, effectively balancing the "consensus" (shared information) with "complementary" (view-specific) details remains a significant challenge. We introduce the Adversarial Co-atTention Network for multi-view Subspace learning (ANTS), a framework that integrates adversarial training with a co-attention mechanism. ANTS disentangles shared and specific representations while providing interpretability through feature weighting. Experimental results across five diverse datasets demonstrate that ANTS consistently outperforms state-of-the-art methods in both accuracy and robustness.

Introduction

In real-world applications, data often originates from multiple views. For instance, a medical diagnosis may integrate X-rays, blood tests, and patient history. Multi-view learning aims to leverage these diverse sources to improve predictive performance. Traditional approaches like Canonical Correlation Analysis (CCA) focus primarily on finding a common subspace, often discarding unique, view-specific information that could be critical for classification. Furthermore, many deep learning models operate as "black boxes," offering little insight into which features drive their predictions. ANTS addresses these issues by explicitly modeling both shared and specific information and incorporating an interpretable co-attention mechanism.

The ANTS Framework

The ANTS architecture is designed to extract robust, unified representations while preserving the unique nuances of each data source.

Overview of the ANTS architecture showing the flow from input views to final classification.

The framework consists of several key components:

  1. Co-attention Encoder: This module computes an affinity matrix between views, allowing the model to weight features based on their cross-view correlation. This acts as a "heat map" of importance, enhancing interpretability and filtering out noise.
  2. Adversarial Subspace Learning: To ensure a truly unified shared representation, ANTS employs a GAN-style discriminator. While encoders strive to create a shared latent space, the discriminator attempts to identify the source view of a given representation. This adversarial game forces the encoders to produce view-invariant features.
  3. Cross-Reconstruction and Residuals: A unique "cross-reconstruction" loss is used, where the model attempts to reconstruct View A from View B's representation. The difference between the original input and this reconstruction—the residual—is captured as view-specific information.
  4. Label Guidance: Unlike purely unsupervised subspace methods, ANTS integrates the classification task directly into the training process, ensuring that the learned features are optimized for the end goal.

Experimental Evaluation

The performance of ANTS was evaluated against several baselines, including Linear CCA, Deep CCA (DCCA), and Deep Information Bottleneck (Deep IB), across datasets such as Noisy MNIST, XRMB, CelebA, and CUB-200-2011 Birds.

Accuracy and F1 Score comparison across four major benchmarks, highlighting ANTS's superior performance.

As shown in the results, ANTS consistently achieves the highest scores across all metrics. On the Noisy MNIST dataset, it reached an accuracy of 98.05%, surpassing the next best baseline by nearly 2%. Its robustness is particularly evident in the high-dimensional CUB Birds dataset, where it outperformed the second-best algorithm by over 8%. Qualitative analysis of the co-attention maps confirmed that the model focuses on relevant physical features, such as beaks and wings, rather than background noise. On the WebKB dataset, ANTS achieved a leading accuracy of 0.708 and an F1 score of 0.686.

Conclusion

ANTS provides a powerful solution for multi-view learning by effectively balancing consensus and complementarity. Its use of adversarial regularization and co-attention not only improves performance but also offers much-needed interpretability. While the current model shows great promise, future work will focus on improving the scalability of the co-attention mechanism for high-view counts (via "Centroid-ANTS") and reducing the sensitivity to hyperparameter tuning.

Summary updated

Abstract

Many real-world applications involve data from multiple modalities and thus exhibit the view heterogeneity. For example, user modeling on social media might leverage both the topology of the underlying social network and the content of the users' posts; in the medical domain, multiple views could be X-ray images taken at different poses. To date, various techniques have been proposed to achieve promising results, such as canonical correlation analysis based methods, etc. In the meanwhile, it is critical for decision-makers to be able to understand the prediction results from these methods. For example, given the diagnostic result that a model provided based on the X-ray images of a patient at different poses, the doctor needs to know why the model made such a prediction. However, state-of-the-art techniques usually suffer from the inability to utilize the complementary information of each view and to explain the predictions in an interpretable manner. To address these issues, in this paper, we propose a deep co-attention network for multi-view subspace learning, which aims to extract both the common information and the complementary information in an adversarial setting and provide robust interpretations behind the prediction to the end-users via the co-attention mechanism. In particular, it uses a novel cross reconstruction loss and leverages the label information to guide the construction of the latent representation by incorporating the classifier into our model. This improves the quality of latent representation and accelerates the convergence speed. Finally, we develop an efficient iterative algorithm to find the optimal encoders and discriminator, which are evaluated extensively on synthetic and real-world data sets. We also conduct a case study to demonstrate how the proposed method robustly interprets the predictions on an image data set.
BibTeX
@inproceedings{zheng2021deep,
  author = {Lecheng Zheng and Yu Cheng and Hongxia Yang and Nan Cao and Jingrui He},
  title = {{Deep Co-Attention Network for Multi-View Subspace Learning}},
  booktitle = {The Web Conference},
  year = {2021},
  doi = {10.1145/3442381.3449801},
}
Figure