Communication-Efficient Distributed Linear and Deep Generalized Canonical Correlation Analysis

Sagar Shrestha, Xiao FuarXiv 2021 · arXiv

Introduction

Generalized Canonical Correlation Analysis (GCCA) is a fundamental technique in multiview learning, aimed at finding a shared, low-dimensional representation for data entities observed through multiple modalities—such as images, audio, and text. In the era of big data and privacy-preserving computing, these "views" are frequently distributed across disparate edge devices or organizations. This distributed nature necessitates federated learning approaches where raw data remains local, and only model parameters or updates are exchanged.

However, standard distributed GCCA algorithms face a significant hurdle: the communication bottleneck. Exchanging high-precision matrices between nodes and a central server in every iteration is bandwidth-intensive. Moreover, GCCA presents a unique mathematical challenge as a manifold-constrained, multi-block nonconvex optimization problem, making standard federated optimization techniques difficult to apply with convergence guarantees.

The CuteMaxVar Framework

To address these challenges, we introduce CuteMaxVar (Communication-quantized MAX-VAR), a framework designed for large-scale federated GCCA. CuteMaxVar optimizes the "Maximum Variance" (MAX-VAR) formulation of GCCA, supporting both linear transformations and deep neural network-based feature extraction.

Federated learning architecture for multiview GCCA.

The framework relies on three core technical pillars:

  1. Bi-directional Compression: Unlike many existing methods that only compress the uplink, CuteMaxVar employs $\delta$-compressors for both the uplink (device-to-server) and the downlink (server-to-device), significantly reducing total network traffic.
  2. Error Feedback (EF): To mitigate the potential for quantization noise to hinder convergence, CuteMaxVar incorporates an error-feedback mechanism. By tracking and re-incorporating compression errors from previous iterations, the algorithm ensures that the effective noise vanishes as the model approaches an optimum.
  3. Proximal Alternating Optimization: The server utilizes a proximal term when updating the global representation, regularizing the process against the quantized updates received from the distributed nodes.

Bi-directional Compression and Quantization

The efficiency of CuteMaxVar stems from its ability to operate with extremely low-precision updates. By normalizing values into discrete bins, the system can reduce communication overhead by over 90%.

Illustration of value normalization and quantization.

Experimental Evaluation

The performance of CuteMaxVar was validated across three diverse datasets:

  • Synthetic Data: Scalability tests with up to 100 views and 50,000 samples.
  • Multilingual Data (Europarl): Sentence alignment across 10 languages (160,000 samples).
  • Medical Data (EHR): Learning diagnosis representations from inpatient, outpatient, and carrier claim views.

Key Results

  • Communication Efficiency: The framework achieved a Compression Ratio (CR) of approximately 0.906, utilizing less than 10% of the bandwidth required by full-precision baselines.
  • Accuracy and Generalization: In the multilingual task, CuteMaxVar matched the accuracy of uncompressed models. Notably, in the EHR diagnosis task, the quantized model achieved 82% accuracy, slightly outperforming the 80% accuracy of the full-precision version. This suggests that quantization noise may serve as a beneficial regularizer.
  • Theoretical Convergence: Rigorous proofs demonstrate that CuteMaxVar converges to a KKT point at a sublinear rate for general cases and reaches the global optimum at a geometric rate for linear GCCA.

Conclusion

CuteMaxVar demonstrates that high-precision communication is not a prerequisite for finding optimal subspaces in multiview learning. By combining aggressive quantization with error feedback, the framework enables efficient, large-scale federated GCCA without sacrificing accuracy. Future research will explore extreme 1-bit compression and the impact of high data heterogeneity (Non-IID) on convergence stability.

Summary updated

Abstract

Classic and deep generalized canonical correlation analysis (GCCA) algorithms seek low-dimensional common representations of data entities from multiple ``views'' (e.g., audio and image) using linear transformations and neural networks, respectively. When the views are acquired and stored at different computing agents (e.g., organizations and edge devices) and data sharing is undesired due to privacy or communication cost considerations, federated learning-based GCCA is well-motivated. In federated learning, the views are kept locally at the agents and only derived, limited information exchange with a central server is allowed. However, applying existing GCCA algorithms onto such federated learning settings may incur prohibitively high communication overhead. This work puts forth a communication-efficient federated learning framework for both linear and deep GCCA under the maximum variance (MAX-VAR) formulation. The overhead issue is addressed by aggressively compressing (via quantization) the exchanging information between the computing agents and a central controller. Compared to the unquantized version, our empirical study shows that the proposed algorithm enjoys a substantial reduction of communication overheads with virtually no loss in accuracy and convergence speed. Rigorous convergence analyses are also presented, which is a nontrivial effort. Generic federated optimization results do not cover the special problem structure of GCCA. Our result shows that the proposed algorithms for both linear and deep GCCA converge to critical points at a sublinear rate, even under heavy quantization and stochastic approximations. In addition, in the linear MAX-VAR case, the quantized algorithm approaches a global optimum in a geometric rate under reasonable conditions. Synthetic and real-data experiments are used to showcase the effectiveness of the proposed approach.
BibTeX
@misc{shrestha2021communication,
  author = {Sagar Shrestha and Xiao Fu},
  title = {{Communication-Efficient Distributed Linear and Deep Generalized Canonical Correlation Analysis}},
  year = {2021},
  eprint = {2109.12400},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.12400},
}
Figure