Communication-Efficient Federated Linear and Deep Generalized Canonical Correlation Analysis

Sagar Shrestha, Xiao FuIEEE TSP 2023 · OpenReview

TL;DR

This paper introduces a novel federated learning framework for Generalized Canonical Correlation Analysis (GCCA) that significantly reduces communication overhead by aggressively quantizing information exchanged between devices. This allows for efficient learning of shared representations from multiple data views in privacy-sensitive, distributed settings without sacrificing accuracy or convergence speed.

What Problem Does This Paper Address?

Modern machine learning often deals with data that comes in multiple "views" (e.g., images and audio of the same object) and is distributed across different devices or organizations. Generalized Canonical Correlation Analysis (GCCA) is a powerful technique for finding common, low-dimensional representations from these multiple views. However, traditional GCCA algorithms require significant data exchange, which is problematic in federated learning scenarios where data privacy is paramount and communication bandwidth is limited. Existing federated GCCA approaches still suffer from high communication costs, hindering their practical application in areas like edge computing and IoT. This paper tackles the challenge of making federated GCCA computationally efficient by minimizing the amount of data transmitted between devices and a central server.

Methods

The core contribution of this work is a communication-efficient federated learning framework for both linear and deep GCCA, specifically targeting the Maximum Variance (MAX-VAR) formulation. The key innovation lies in aggressively compressing the information exchanged between computing agents and the central controller using quantization. This is achieved through a novel algorithm that incorporates an "error feedback" (EF) mechanism. EF helps to mitigate the noise introduced by quantization, ensuring that the compressed information still allows for accurate updates. The framework is designed to handle the specific multi-block, manifold-constrained, non-convex optimization problem structure of GCCA, which is not covered by standard federated optimization results. The authors provide rigorous convergence analyses for both linear and deep GCCA under heavy quantization.

Datasets

The paper evaluates its proposed method on both synthetic and real-world datasets.

  • Synthetic Data: Generated to mimic scenarios with multiple views and varying complexities, used for testing both linear and deep GCCA.
  • Real Data:
    • Europarl Corpus: Used for a cross-language sentence alignment task, treating sentences from different languages as distinct views.
  • EHR Data (Centers for Medicare and Medicaid Services - CMS): Utilized for learning representations of diagnoses from multiple views of electronic health records, highlighting the privacy-sensitive nature of such data.

Key Insights and Ideas

  • Quantization is Key for Federated GCCA: The paper demonstrates that aggressive quantization of exchanged information, when combined with error feedback, can drastically reduce communication overhead in federated GCCA without significant loss in performance.
  • Error Feedback is Crucial: The error feedback mechanism is vital for overcoming the challenges posed by quantization noise, enabling convergence even with heavy compression.
  • Tailored Analysis for GCCA: Standard federated learning convergence analyses are insufficient for GCCA due to its unique manifold-constrained, multi-block, non-convex structure. The authors developed custom analyses to address this.
  • Deep GCCA Benefits from Compression: The proposed method shows that deep learning-based GCCA can also be made communication-efficient, maintaining high accuracy and convergence speed.
  • Geometric Convergence for Linear Case: Under reasonable conditions, the quantized algorithm for linear MAX-VAR GCCA can achieve global optimality at a geometric rate, a significant theoretical result.

Key Results

  • Substantial Communication Reduction: Experiments show a reduction of communication overhead by up to 90% compared to unquantized versions, with compression ratios (CR) as high as 0.906.
  • No Loss in Accuracy or Speed: The proposed method achieves virtually no loss in accuracy or convergence speed compared to its uncompressed counterparts. For instance, in deep GCCA experiments, classification accuracy improved from 42.5% (raw data) to 80.0% (AltMaxVar-Deep) and 82.0% (CuteMaxVar-Deep), with CuteMaxVar-Deep achieving a CR of 0.906.
  • Rigorous Convergence Guarantees: The paper provides theoretical guarantees for convergence to critical points (sublinear rate) for both linear and deep GCCA, and geometric convergence to a neighborhood of the global optimum for the linear case.
  • Scalability: The method demonstrates robustness to an increasing number of views, performing well even with up to 100 views.

Limitations and Future Work

  • Initialization Sensitivity: The paper notes that proper initialization is crucial for establishing convergence, and suggests using full-precision communication for this initial step.
  • Compressor Limitations: The chosen compressor (d-compressor) has limitations, such as not supporting extremely heavy compression (e.g., 1 bit per scalar).
  • Future Work: The authors suggest that their algorithm design principles could be extended to other multimodal models beyond GCCA. Further exploration of different compressor types and their impact on convergence and communication efficiency could also be a direction. The paper also mentions that the proof of convergence for the deep GCCA case is asymptotic, and stronger convergence results might be achievable with more stringent step size specifications.

Summary updated

Abstract

Classic and deep learning-based generalized canonical correlation analysis (GCCA) algorithms seek low-dimensional common representations of data entities from multiple “views” (e.g., audio and image) using linear transformations and neural networks, respectively. When the views are acquired and stored at different computing agents (e.g., organizations and edge devices) and data sharing is undesired due to privacy or communication cost considerations, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">federated learning</i> -based GCCA is well-motivated. In federated learning, the views are kept locally at the agents and only derived, limited information exchange with a central server is allowed. However, applying existing GCCA algorithms onto such setting may incur prohibitively high communication overhead. This work puts forth a communication-efficient federated learning framework for both linear and deep GCCA under the maximum variance (MAX-VAR) formulation. The overhead issue is addressed by aggressively compressing (via quantization) the exchanging information between the computing agents and a central controller. Our synthetic and real-data experiments shows that the proposed algorithm enjoys a substantial reduction of communication overheads with virtually no loss in accuracy and convergence speed compared to the unquantized version. Rigorous convergence analyses are also presented, which is a nontrivial effort since generic federated optimization results do not cover the special problem structure of GCCA. Our result shows that the proposed algorithms for both linear and deep GCCA converge to critical points at a sublinear rate, even under heavy quantization and stochastic approximations. In addition, in the linear MAX-VAR case, the quantized algorithm approaches a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">global optimum</i> in a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">geometric</i> rate under reasonable conditions.
BibTeX
@article{shrestha2023communication,
  author = {Sagar Shrestha and Xiao Fu},
  title = {{Communication-Efficient Federated Linear and Deep Generalized Canonical Correlation Analysis}},
  journal = {IEEE Trans. Signal Process.},
  year = {2023},
  url = {https://openreview.net/forum?id=hvKZPnKtXP},
}
Figure