Gompertz Linear Units: Leveraging Asymmetry for Enhanced Learning Dynamics

Indrashis Das, Mahmoud Safari, Steven Adriaensen, Frank Hutter

TL;DR

The Gompertz Linear Unit (GoLU) is a novel self-gated activation function that leverages right-skewed asymmetry to improve neural network training. By utilizing the Gompertz function as a gating mechanism, GoLU naturally compresses latent representations and smooths the loss landscape. This simple drop-in replacement consistently outperforms standard activations like ReLU and GELU across a wide array of vision and language tasks without sacrificing computational speed.

What Problem Does This Paper Address?

Choosing the right activation function is crucial because it dictates how gradients flow through a network during training. While the classic Rectified Linear Unit (ReLU) is computationally efficient, it suffers from the "dying ReLU" problem, where neurons permanently deactivate if they output negative values.

To fix this, researchers introduced smooth, self-gated activation functions like Swish and GELU. These functions use symmetric gating mechanisms (based on the Sigmoid function or the Gaussian CDF) to allow small negative gradients to pass through. However, almost all popular self-gated activations rely on symmetric or left-skewed distributions. This paper investigates a neglected dimension of activation function design: Can right-skewed asymmetry be leveraged to improve learning dynamics?

Methods

The authors introduce the Gompertz Linear Unit (GoLU), defined mathematically as:

$$\text{GoLU}(x) = x \cdot \text{Gompertz}(x)$$

where the gating mechanism is the Gompertz function:

$$\text{Gompertz}(x) = e^{-e^{-x}}$$

The Gompertz function acts as the Cumulative Distribution Function (CDF) of the standard Gumbel distribution, which is inherently right-skewed.

Figure 1: Activation functions (Left) and their corresponding gate functions (Right). GoLU and its gate, the Gompertz function, are highlighted in red. Note the slight rightward shift of the Gompertz gate.

Because of this rightward shift, GoLU has a smaller slope near the origin and across a large portion of the negative input domain compared to GELU or Swish. To ensure that GoLU doesn't slow down training, the authors implemented a custom, highly optimized CUDA kernel for PyTorch.

To isolate the effects of right-skewed asymmetry, the authors also engineered a control activation called Flipped Mish (FMish) by reflecting the left-skewed Mish distribution across the vertical axis.

Datasets

The authors evaluated GoLU across a diverse and rigorous suite of benchmarks:

  • Image Classification: ImageNet-1k and CIFAR-10
  • Language Modeling: TinyStories (babyGPT) and OpenWebText (GPT2-Small)
  • Downstream Vision Tasks (on MS-COCO): Semantic Segmentation (DeepLabV3), Object Detection (Faster R-CNN, RetinaNet), and Instance Segmentation (Mask R-CNN)
  • Generative Modeling: Denoising Diffusion Probabilistic Models (DDPM) on CelebA
  • Machine Translation: WMT14 English–German

Key Insights and Ideas

  • The "Squeezing Effect" (Variance Reduction): Because GoLU's slope is shallower around the origin, it acts as a variance reducer. It compresses the output distribution into a tighter range, effectively filtering out fine-grained noise and forcing the network to focus on core, robust features.
Figure 3: Image created by Dall-E 3 (Left) and kernel den- sity estimation curves for distributions of activation outputs for the image (Right). GoLU reduces variance most com- pared to baseline activations.
  • Flatter Loss Landscapes: Lower activation variance naturally leads to less noisy, more consistent gradients. This results in a significantly smoother loss landscape, helping optimizers converge to flatter minima that generalize better to unseen data.
  • Weight Diversity as Compensation: Interestingly, while GoLU reduces variance in activations, it induces a wider, more spread-out distribution of learned weights. The network compensates for the compressed activation space by learning a more diverse set of weight transformations.
  • Asymmetry is the Secret Sauce: The "Flipped Mish" experiment confirmed their hypothesis: giving Mish a right-leaning asymmetry caused it to outperform standard symmetric activations, proving that right-skewed gating is fundamentally beneficial for variance control.

Key Results

GoLU achieved state-of-the-art performance across nearly every task evaluated:

  • ImageNet-1k: On ResNet-50, GoLU achieved 76.63% top-1 accuracy, outperforming ReLU (75.44%) and GELU (76.07%). On Vision Transformers (ViT-B/16), it reached 80.72% (vs. 79.39% for GELU).
  • Language Modeling: For GPT2-Small on OpenWebText, GoLU achieved a lower perplexity score of 17.297 compared to GELU's 17.525.
  • Object Detection (MS-COCO): Faster R-CNN with a GoLU-backed ResNet-50 achieved a Box mAP of 38.31, outperforming GELU (38.16) and ReLU (37.44).
  • Efficiency: Thanks to the custom CUDA kernel, GoLU adds virtually zero computational overhead. Across all tasks, it averaged 1.01x the training time and 1.00x the inference time of default activations.

Limitations and Future Work

While GoLU is highly effective, the authors noted that its performance is tightly linked to hyperparameter tuning—specifically the learning rate. Because GoLU alters gradient dynamics, standard default learning rates optimized for ReLU or GELU can sometimes be suboptimal for GoLU. For instance, in the diffusion and segmentation tasks, GoLU only unlocked its superior performance after the learning rate was adjusted upward.

Additionally, architectures that rely heavily on internal non-linearities outside of standard activations (like the squeeze-and-excitation blocks in EfficientNet) showed less pronounced benefits. Future work could explore dynamically adjusting or learning the degree of asymmetry during training to make the activation more robust out-of-the-box.

Summary updated

Abstract

Activation functions are fundamental elements of deep learning architectures as they significantly influence training dynamics. ReLU, while widely used, is prone to the dying neuron problem, which has been mitigated by variants such as LeakyReLU, PReLU, and ELU that better handle negative neuron outputs. Recently, self-gated activations like GELU and Swish have emerged as state-of-the-art alternatives, leveraging their smoothness to ensure stable gradient flow and prevent neuron inactivity. In this work, we introduce the Gompertz Linear Unit (GoLU), a novel self-gated activation function defined as $\mathrm{GoLU}(x) = x \, \mathrm{Gompertz}(x)$, where $\mathrm{Gompertz}(x) = e^{-e^{-x}}$. The GoLU activation leverages the right-skewed asymmetry in the Gompertz function to reduce variance in the latent space more effectively compared to GELU and Swish, while preserving robust gradient flow. Extensive experiments across diverse tasks, including Image Classification, Language Modeling, Semantic Segmentation, Object Detection, Instance Segmentation, and Diffusion, highlight GoLU's superior performance relative to state-of-the-art activation functions, establishing GoLU as a robust alternative to existing activation functions.
BibTeX
@inproceedings{das2025gompertz,
  author = {Indrashis Das and Mahmoud Safari and Steven Adriaensen and Frank Hutter},
  title = {{Gompertz Linear Units: Leveraging Asymmetry for Enhanced Learning Dynamics}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=d8yAQOSLwv},
}
Figure