Efficient Language Model Architectures for Differentially Private Federated Learning

Jae Hun Ro, Srinadh Bhojanapalli, Zheng Xu, Yanxiang Zhang, Ananda Theertha Suresh

TL;DR

This paper proposes modifying language model architectures to be "scale-invariant," allowing them to be trained efficiently using simple SGD on resource-constrained edge devices. By replacing standard activation functions with scale-invariant versions, the authors achieve faster convergence and better performance in cross-device federated learning (FL), even outperforming standard models that require memory-intensive adaptive optimizers.

What Problem Does This Paper Address?

Cross-device federated learning (FL) involves training models on millions of edge devices (like smartphones) without their data ever leaving the device. In this setting, memory is a precious resource. While modern language models (LSTMs and Transformers) typically require adaptive optimizers like Adam to perform well, Adam is memory-hungry because it must store first and second moments of gradients for every parameter.

Consequently, FL practitioners usually fall back on Stochastic Gradient Descent (SGD) for client-side training. However, standard architectures often struggle with SGD, leading to slower convergence or training instability. The authors ask a fundamental question: Instead of trying to fit a heavy optimizer onto a device, can we change the model architecture so it trains effectively with a lightweight one?

Methods

The core idea is Scale Invariance. A function $f$ is scale-invariant if $f(ax) = f(x)$ for any scalar $a > 0$. The authors argue that standard activations like Sigmoid and Tanh are sensitive to input scale, which contributes to the optimization difficulties seen with SGD.

SI-CIFG (Scale-Invariant Coupled Input Forget Gate)

The authors focus on the CIFG recurrent network, a parameter-efficient variant of the LSTM. They introduce two new scale-invariant activation functions:

  1. SI-sigma: $\text{SI-}\sigma(x)_i = \frac{\text{ReLU}(x)_i}{\max_j(\text{ReLU}(x)_j)}$
  2. SI-tanh: $\text{SI-tanh}(x)_i = \frac{x_i}{\max_j(|x_j|)}$

By replacing standard activations with these and using Max-Normalization (MaxN)—which normalizes a vector by its maximum absolute value—they create a recurrent cell that is robust to the scale of its inputs and hidden states.

SI-Transformer

They also extend previous work on Scale-Invariant Transformers (which replaced Softmax with a normalized ReLU) to decoder-only models suitable for language modeling in FL.

Datasets

The researchers evaluated their approach across three distinct environments:

  • Stack Overflow: A public federated dataset containing forum questions and answers, used for large-scale simulations.
  • Live Production (Gboard): Experiments conducted on millions of real-world virtual keyboard devices.
  • C4 (Colossal Clean Crawled Corpus): Used for pre-training models before applying Differentially Private FL.

Key Insights and Ideas

  • Architecture as an Optimizer Substitute: The most striking insight is that architectural changes can compensate for the lack of an adaptive optimizer. Scale-invariant models trained with SGD on clients achieved utility comparable to or better than standard models.
  • Stability in the Wild: In live production experiments, the standard CIFG model actually diverged (failed to train) at one point. In contrast, the SI-CIFG model remained stable and converged smoothly.
  • Robustness to Hyperparameters: Scale-invariant models appeared more robust to higher learning rates, which is crucial in FL where hyperparameter tuning is difficult and expensive.
  • Synergy with Differential Privacy (DP): DP-FL requires clipping model updates and adding noise, which often degrades performance. Scale-invariant architectures seem naturally better suited to handle these perturbations, offering a better privacy-utility trade-off.

Key Results

The SI-architectures consistently outperformed their standard counterparts across all metrics (Perplexity and Accuracy).

Figure 1: Perplexity and accuracy on the Stack Overflow test dataset with shading indicating standard deviation over 5 random seeds.

In live production experiments (Figure 2), the SI-CIFG 9M model achieved an accuracy of 23.4%, significantly higher than the base CIFG 9M at 21.8% (which also suffered from divergence issues).

Live production experiments showing the stability and superior performance of SI-CIFG (orange) compared to the unstable base CIFG (blue).

When testing with Differential Privacy (Figure 3), the SI-CIFG model consistently maintained a lead in utility over the base model throughout the entire training process.

Figure 3: Smoothed perplexity and in-vocab-accuracy from DP live experiments on English virtual keyboard devices.

Limitations and Future Work

  • Hyperparameter Tuning: Due to the constraints of live production environments, the authors could not perform exhaustive hyperparameter sweeps for all models, potentially leaving some performance on the table.
  • Model Scaling: The experiments focused on relatively small models (up to 21M parameters) to fit edge device constraints. Future work could explore if these scale-invariant properties hold for much larger "edge-capable" models.
  • Theoretical Depth: While the empirical results are strong, a deeper theoretical understanding of why Max-Normalization specifically helps with SGD stability in recurrent networks remains an open area for exploration.

Summary updated

Abstract

Cross-device federated learning (FL) is a technique that trains a model on data distributed across typically millions of edge devices without data leaving the devices. SGD is the standard client optimizer for on device training in cross-device FL, favored for its memory and computational efficiency. However, in centralized training of neural language models, adaptive optimizers are preferred as they offer improved stability and performance. In light of this, we ask if language models can be modified such that they can be efficiently trained with SGD client optimizers and answer this affirmatively. We propose a scale-invariant Coupled Input Forget Gate (SI CIFG) recurrent network by modifying the sigmoid and tanh activations in the recurrent cell and show that this new model converges faster and achieves better utility than the standard CIFG recurrent model in cross-device FL in large scale experiments. We further show that the proposed scale invariant modification also helps in federated learning of larger transformer models. Finally, we demonstrate the scale invariant modification is also compatible with other non-adaptive algorithms. Particularly, our results suggest an improved privacy utility trade-off in federated learning with differential privacy.
BibTeX
@misc{ro2024efficient,
  author = {Jae Hun Ro and Srinadh Bhojanapalli and Zheng Xu and Yanxiang Zhang and Ananda Theertha Suresh},
  title = {{Efficient Language Model Architectures for Differentially Private Federated Learning}},
  year = {2024},
  eprint = {2403.08100},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.08100},
}
Figure