TL;DR
This paper proposes modifying language model architectures to be "scale-invariant," allowing them to be trained efficiently using simple SGD on resource-constrained edge devices. By replacing standard activation functions with scale-invariant versions, the authors achieve faster convergence and better performance in cross-device federated learning (FL), even outperforming standard models that require memory-intensive adaptive optimizers.
What Problem Does This Paper Address?
Cross-device federated learning (FL) involves training models on millions of edge devices (like smartphones) without their data ever leaving the device. In this setting, memory is a precious resource. While modern language models (LSTMs and Transformers) typically require adaptive optimizers like Adam to perform well, Adam is memory-hungry because it must store first and second moments of gradients for every parameter.
Consequently, FL practitioners usually fall back on Stochastic Gradient Descent (SGD) for client-side training. However, standard architectures often struggle with SGD, leading to slower convergence or training instability. The authors ask a fundamental question: Instead of trying to fit a heavy optimizer onto a device, can we change the model architecture so it trains effectively with a lightweight one?
Methods
The core idea is Scale Invariance. A function $f$ is scale-invariant if $f(ax) = f(x)$ for any scalar $a > 0$. The authors argue that standard activations like Sigmoid and Tanh are sensitive to input scale, which contributes to the optimization difficulties seen with SGD.
SI-CIFG (Scale-Invariant Coupled Input Forget Gate)
The authors focus on the CIFG recurrent network, a parameter-efficient variant of the LSTM. They introduce two new scale-invariant activation functions:
- SI-sigma: $\text{SI-}\sigma(x)_i = \frac{\text{ReLU}(x)_i}{\max_j(\text{ReLU}(x)_j)}$
- SI-tanh: $\text{SI-tanh}(x)_i = \frac{x_i}{\max_j(|x_j|)}$
By replacing standard activations with these and using Max-Normalization (MaxN)—which normalizes a vector by its maximum absolute value—they create a recurrent cell that is robust to the scale of its inputs and hidden states.
SI-Transformer
They also extend previous work on Scale-Invariant Transformers (which replaced Softmax with a normalized ReLU) to decoder-only models suitable for language modeling in FL.
Datasets
The researchers evaluated their approach across three distinct environments:
- Stack Overflow: A public federated dataset containing forum questions and answers, used for large-scale simulations.
- Live Production (Gboard): Experiments conducted on millions of real-world virtual keyboard devices.
- C4 (Colossal Clean Crawled Corpus): Used for pre-training models before applying Differentially Private FL.
Key Insights and Ideas
- Architecture as an Optimizer Substitute: The most striking insight is that architectural changes can compensate for the lack of an adaptive optimizer. Scale-invariant models trained with SGD on clients achieved utility comparable to or better than standard models.
- Stability in the Wild: In live production experiments, the standard CIFG model actually diverged (failed to train) at one point. In contrast, the SI-CIFG model remained stable and converged smoothly.
- Robustness to Hyperparameters: Scale-invariant models appeared more robust to higher learning rates, which is crucial in FL where hyperparameter tuning is difficult and expensive.
- Synergy with Differential Privacy (DP): DP-FL requires clipping model updates and adding noise, which often degrades performance. Scale-invariant architectures seem naturally better suited to handle these perturbations, offering a better privacy-utility trade-off.
Key Results
The SI-architectures consistently outperformed their standard counterparts across all metrics (Perplexity and Accuracy).
In live production experiments (Figure 2), the SI-CIFG 9M model achieved an accuracy of 23.4%, significantly higher than the base CIFG 9M at 21.8% (which also suffered from divergence issues).
When testing with Differential Privacy (Figure 3), the SI-CIFG model consistently maintained a lead in utility over the base model throughout the entire training process.
Limitations and Future Work
- Hyperparameter Tuning: Due to the constraints of live production environments, the authors could not perform exhaustive hyperparameter sweeps for all models, potentially leaving some performance on the table.
- Model Scaling: The experiments focused on relatively small models (up to 21M parameters) to fit edge device constraints. Future work could explore if these scale-invariant properties hold for much larger "edge-capable" models.
- Theoretical Depth: While the empirical results are strong, a deeper theoretical understanding of why Max-Normalization specifically helps with SGD stability in recurrent networks remains an open area for exploration.
Summary updated