No article text was recorded for this version.
uniform learning rate limitation
Version 1 of 1155 papers183 conceptsfrozen 2026-10-04
Papers in this version
- Blog: What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysis
- Blog: One LR Doesn’t Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMs
- Blog: Reducing Underflow in Mixed Precision Training by Gradient Scaling
- Blog: When Can You Get Away with Low Memory Adam?
- Blog: Modality-specific Learning Rates for Effective Multimodal Additive Late-fusion
- Blog: Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning
- Blog: In Search of Adam’s Secret Sauce
- Blog: AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMs
- Blog: Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Training
- Blog: Softsign: Smooth Sign in Your Optimizer For Better Parameter Heterogeneity Handling
- Blog: New Hybrid Fine-Tuning Paradigm for LLMs: Algorithm Design and Convergence Analysis Framework
- Blog: Large Batch Optimization for Deep Learning Using New Complete Layer-Wise Adaptive Rate Scaling
- Blog: Channel-Wise Early Stopping without a Validation Set via NNK Polytope Interpolation
- Blog: Balancing Learning Rates Across Layers: Exact Two-Step Dynamics and Optimal Scaling in Linear Neural Networks
- Blog: Why Transformers Need Adam: A Hessian Perspective
- Blog: The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training
- Blog: Taming LLMs by Scaling Learning Rates with Gradient Grouping
- Blog: Train Faster, Perform Better: Modular Adaptive Training in Over-Parameterized Models
- Blog: GA-SAM: Gradient-Strength based Adaptive Sharpness-Aware Minimization for Improved Generalization
- Blog: Why Transformers Need Adam: A Hessian Perspective
- Blog: Coordination Algorithm in Hierarchical Structure of the Learning Process of Artificial Neural Network
- Blog: From Muon to Gluon: Bridging Theory and Practice of LMO-based Optimizers for LLMs
- Blog: Scaling SGD Batch Size to 32K for ImageNet Training
- Blog: Large Batch Training Does Not Need Warmup
- Blog: Bolstering stochastic gradient descent with model building
- Blog: SoftSignSGD(S3): An Enhanced Optimizer for Practical DNN Training and Loss Spikes Minimization Beyond Adam
- Blog: Momentum Stiefel Optimizer, with Applications to Suitably-Orthogonal Attention, and Optimal Transport
- Blog: Closed-Form Last Layer Optimization
- Blog: Taming LLMs with Gradient Grouping
- Blog: Exploiting Explainable Metrics for Augmented SGD
- Blog: Enabling Differentially Private Federated Learning for Speech Recognition: Benchmarks, Adaptive Optimizers, and Gradient Clipping
- Blog: Stochastic Gradient Descent: Going As Fast As Possible But Not Faster
- Blog: Neuromodulation by surprise: a biologically plausible model of the learning rate dynamics
- Blog: Scaling Laws for the Principled Design, Initialization, and Preconditioning of ReLU Networks
- Blog: Vision Transformer Finetuning Benefits from Non-Smooth Components
- Blog: Fast and Slow Gradient Approximation for Binary Neural Network Optimization
- Blog: Convergence Analysis for Learning Orthonormal Deep Linear Neural Networks
- Blog: Proportionate gradient updates with PercentDelta
- Blog: On the Weight Dynamics of Deep Normalized Networks
- Blog: Theoretical Insights into Fine-Tuning Attention Mechanism: Generalization and Optimization
- Blog: On Exploiting Layerwise Gradient Statistics for Effective Training of Deep Neural Networks
- Blog: HELENE: Hessian Layer-wise Clipping and Gradient Annealing for Accelerating Fine-tuning LLM with Zeroth-order Optimization
- Blog: Adaptive Weight Decay for Deep Neural Networks
- Blog: AutoLR: Layer-wise Pruning and Auto-tuning of Learning Rates in Fine-tuning of Deep Networks
- Blog: Relational Neural Markov Random Fields
- Blog: Data-dependent initializations of Convolutional Neural Networks Philipp Kraehenbuehl
- Blog: Efficient Learning of Entity and Predicate Embeddings for Link Prediction in Knowledge Graphs
- Blog: Adam with model exponential moving average is effective for nonconvex optimization
- Blog: Weighted Neural Tangent Kernel: A Generalized and Improved Network-Induced Kernel
- Blog: A Layer-Wise Natural Gradient Optimizer for Training Deep Neural Networks
- Blog: Fed-LAMB: Layerwise and Dimensionwise Locally Adaptive Optimization Algorithm
- Blog: Adaptive Federated Learning with Auto-Tuned Clients
- Blog: CoMMIT: Coordinated Instruction Tuning for Multimodal Large Language Models
- Blog: Understanding and Improving Early Stopping for Learning with Noisy Labels
- Blog: Large Batch Training of Convolutional Networks
- Blog: Det-CGD: Compressed Gradient Descent with Matrix Stepsizes for Non-Convex Optimization
- Blog: Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedback
- Blog: Layer-Specific Adaptive Learning Rates for Deep Networks
- Blog: Enabling Few-Shot Learning with PID Control: A Layer Adaptive Optimizer
- Blog: Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
Showing 60 of 155 papers.
History & citation
Current · Permanent link
- Layer-wise Adaptive Optimization for Heterogeneous Architectures merged into this direction (66% overlap)
- Decoupled Learning Rates for Heterogeneous Model Components merged into this direction (56% overlap)
Cite this direction
Latest version (a living page):
@misc{lacuna_uniform_learning_rate_limitation_85a4c0f7,
title = {uniform learning rate limitation},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/uniform-learning-rate-limitation/rdr_85a4c0f76d6247668548b7d3b8fd186c},
note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}This exact version (version 1, frozen text):
@misc{lacuna_uniform_learning_rate_limitation_85a4c0f7_v1,
title = {uniform learning rate limitation},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/uniform-learning-rate-limitation/rdr_85a4c0f76d6247668548b7d3b8fd186c/v/1},
note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}