Sign-SGD is the Golden Gate between Multi-Node to Single-Node Learning: Significant Boost via Parameter-Free Optimization

Daniil Medyakov, Sergey Stanko, Gleb Molodtsov, Philip Zmushko, Grigoriy Evseev, Egor Petrov, Aleksandr Beznosikov

Abstract

Training Large Language Models (LLMs) is notoriously resource-intensive, often requiring extensive hyperparameter tuning that consumes significant compute time. This paper introduces ALIAS (Automatic Local per-Iteration Approximation of the Stepsize), a parameter-free version of the SIGN-SGD optimizer that eliminates the need for manual stepsize tuning. By dynamically estimating problem-specific constants such as local smoothness and distance to the optimum during training, ALIAS matches the performance of grid-searched AdamW and SIGN-SGD. Crucially, it provides a ~1.5x end-to-end speedup by removing the overhead of hyperparameter searches.

Introduction

The training of Large Language Models (LLMs) faces two primary bottlenecks: memory consumption and communication overhead in distributed settings. SIGN-SGD has been proposed as a solution, as it only utilizes the sign of the gradient. This approach is memory-efficient, avoiding the storage of full second moments, and communication-efficient, requiring only 1-bit signs to be transmitted between nodes.

Despite these advantages, SIGN-SGD's performance is highly sensitive to the stepsize ($\gamma$). Theoretically optimal stepsizes depend on unknown quantities like the function's smoothness ($L_\infty$) and the distance to the optimal solution ($\Delta^*$). In practice, this necessitates expensive grid searches, which negate much of the efficiency gains. This paper addresses this "Achilles' heel" by proposing ALIAS, a tuning-free framework for sign-based optimization.

Methodology: The ALIAS Framework

The authors propose ALIAS to estimate necessary hyperparameters on the fly, transforming SIGN-SGD into a parameter-free optimizer. The core innovations include:

  1. Adaptive Smoothness Estimation: Rather than assuming a global smoothness constant, ALIAS approximates it at every iteration by observing gradient changes relative to the step taken.
  2. Distance Estimation: An iterative scheme estimates the distance to the optimum ($\Delta^*$) without requiring prior knowledge of the final loss value.
  3. Memory-Efficient Design: A variant of ALIAS maintains the low-memory promise of sign-based methods by storing only two scalars (max and min gradient coordinates) instead of full historical gradients.

For practical deep learning applications, the authors developed ALIAS-Adam, which integrates momentum and coordinate-wise normalization into the parameter-free framework.

Algorithm 1: ALIAS-Adam, the parameter-free variant of Sign-SGD with momentum and coordinate-wise normalization.

Experimental Evaluation

The researchers evaluated ALIAS across diverse scales and domains:

  • Language Modeling: Pre-training LLaMA architectures (130M and 350M parameters) on the C4 dataset.
  • Computer Vision: Fine-tuning a Swin Transformer (28M parameters) on Tiny ImageNet.
  • Benchmarks: MRI reconstruction (fastMRI) and molecular property prediction (OGBG) via the AlgoPerf benchmark.
  • Theoretical Validation: Logistic regression on standard LIBSVM datasets.

Key Insights and Results

The empirical results demonstrate that ALIAS is not only a theoretical contribution but a practical tool for efficient training.

Performance and Efficiency

On LLaMA pre-training, ALIAS matched or slightly outperformed tuned SIGN-SGD and AdamW with cosine schedules. In Swin Transformer fine-tuning, ALIAS-Adam achieved a final accuracy of 79.16%, surpassing tuned AdamW (77.61%) and the state-of-the-art parameter-free optimizer, Prodigy (77.94%).

From a resource perspective, the memory-efficient version of ALIAS used only 0.41 GB of memory per iteration, compared to 1.5 GB for AdamW and 3.5 GB for Prodigy, while maintaining comparable time-per-iteration.

The "Natural" Cosine Schedule

A striking finding is that ALIAS naturally produces a learning rate decay resembling the popular "cosine schedule" used in industry. Unlike traditional schedules, ALIAS does this automatically based on the loss landscape rather than a pre-defined timer.

Eliminating the "Tuning Tax"

The primary efficiency gain of ALIAS is the elimination of the "tuning tax." The 1.5x end-to-end speedup is achieved because training only needs to be run once, rather than across multiple hyperparameter configurations.

Conclusion and Future Work

ALIAS bridges the gap between the theoretical benefits of sign-based optimization and the practical requirements of deep learning. While the current theoretical convergence analysis relies on convexity assumptions, the empirical success on non-convex neural networks is compelling. Future work will explore more sophisticated momentum techniques and further optimize distributed implementations to fully leverage the 1-bit communication advantage of sign-based methods.

Summary updated

BibTeX
@inproceedings{medyakov2026sign,
  author = {Daniil Medyakov and Sergey Stanko and Gleb Molodtsov and Philip Zmushko and Grigoriy Evseev and Egor Petrov and Aleksandr Beznosikov},
  title = {{Sign-SGD is the Golden Gate between Multi-Node to Single-Node Learning: Significant Boost via Parameter-Free Optimization}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=yDLD3D95w3},
}
Figure