TL;DR
The Gompertz Linear Unit (GoLU) is a novel self-gated activation function that leverages right-skewed asymmetry to improve neural network training. By utilizing the Gompertz function as a gating mechanism, GoLU naturally compresses latent representations and smooths the loss landscape. This simple drop-in replacement consistently outperforms standard activations like ReLU and GELU across a wide array of vision and language tasks without sacrificing computational speed.
What Problem Does This Paper Address?
Choosing the right activation function is crucial because it dictates how gradients flow through a network during training. While the classic Rectified Linear Unit (ReLU) is computationally efficient, it suffers from the "dying ReLU" problem, where neurons permanently deactivate if they output negative values.
To fix this, researchers introduced smooth, self-gated activation functions like Swish and GELU. These functions use symmetric gating mechanisms (based on the Sigmoid function or the Gaussian CDF) to allow small negative gradients to pass through. However, almost all popular self-gated activations rely on symmetric or left-skewed distributions. This paper investigates a neglected dimension of activation function design: Can right-skewed asymmetry be leveraged to improve learning dynamics?
Methods
The authors introduce the Gompertz Linear Unit (GoLU), defined mathematically as:
$$\text{GoLU}(x) = x \cdot \text{Gompertz}(x)$$
where the gating mechanism is the Gompertz function:
$$\text{Gompertz}(x) = e^{-e^{-x}}$$
The Gompertz function acts as the Cumulative Distribution Function (CDF) of the standard Gumbel distribution, which is inherently right-skewed.
Because of this rightward shift, GoLU has a smaller slope near the origin and across a large portion of the negative input domain compared to GELU or Swish. To ensure that GoLU doesn't slow down training, the authors implemented a custom, highly optimized CUDA kernel for PyTorch.
To isolate the effects of right-skewed asymmetry, the authors also engineered a control activation called Flipped Mish (FMish) by reflecting the left-skewed Mish distribution across the vertical axis.
Datasets
The authors evaluated GoLU across a diverse and rigorous suite of benchmarks:
- Image Classification: ImageNet-1k and CIFAR-10
- Language Modeling: TinyStories (babyGPT) and OpenWebText (GPT2-Small)
- Downstream Vision Tasks (on MS-COCO): Semantic Segmentation (DeepLabV3), Object Detection (Faster R-CNN, RetinaNet), and Instance Segmentation (Mask R-CNN)
- Generative Modeling: Denoising Diffusion Probabilistic Models (DDPM) on CelebA
- Machine Translation: WMT14 English–German
Key Insights and Ideas
- The "Squeezing Effect" (Variance Reduction): Because GoLU's slope is shallower around the origin, it acts as a variance reducer. It compresses the output distribution into a tighter range, effectively filtering out fine-grained noise and forcing the network to focus on core, robust features.
- Flatter Loss Landscapes: Lower activation variance naturally leads to less noisy, more consistent gradients. This results in a significantly smoother loss landscape, helping optimizers converge to flatter minima that generalize better to unseen data.
- Weight Diversity as Compensation: Interestingly, while GoLU reduces variance in activations, it induces a wider, more spread-out distribution of learned weights. The network compensates for the compressed activation space by learning a more diverse set of weight transformations.
- Asymmetry is the Secret Sauce: The "Flipped Mish" experiment confirmed their hypothesis: giving Mish a right-leaning asymmetry caused it to outperform standard symmetric activations, proving that right-skewed gating is fundamentally beneficial for variance control.
Key Results
GoLU achieved state-of-the-art performance across nearly every task evaluated:
- ImageNet-1k: On ResNet-50, GoLU achieved 76.63% top-1 accuracy, outperforming ReLU (75.44%) and GELU (76.07%). On Vision Transformers (ViT-B/16), it reached 80.72% (vs. 79.39% for GELU).
- Language Modeling: For GPT2-Small on OpenWebText, GoLU achieved a lower perplexity score of 17.297 compared to GELU's 17.525.
- Object Detection (MS-COCO): Faster R-CNN with a GoLU-backed ResNet-50 achieved a Box mAP of 38.31, outperforming GELU (38.16) and ReLU (37.44).
- Efficiency: Thanks to the custom CUDA kernel, GoLU adds virtually zero computational overhead. Across all tasks, it averaged 1.01x the training time and 1.00x the inference time of default activations.
Limitations and Future Work
While GoLU is highly effective, the authors noted that its performance is tightly linked to hyperparameter tuning—specifically the learning rate. Because GoLU alters gradient dynamics, standard default learning rates optimized for ReLU or GELU can sometimes be suboptimal for GoLU. For instance, in the diffusion and segmentation tasks, GoLU only unlocked its superior performance after the learning rate was adjusted upward.
Additionally, architectures that rely heavily on internal non-linearities outside of standard activations (like the squeeze-and-excitation blocks in EfficientNet) showed less pronounced benefits. Future work could explore dynamically adjusting or learning the degree of asymmetry during training to make the activation more robust out-of-the-box.
Summary updated