Abstract
In deep generative modeling, the standard softmax function is often used to parameterize discrete distributions. However, its dense nature poses computational challenges in large-scale settings, while existing sparse alternatives like sparsemax frequently suffer from mode collapse. We introduce ev-softmax, a computationally efficient sparse normalization function that preserves multimodality. By utilizing mean-based thresholding and a continuous training-time relaxation, ev-softmax enables sparse predictions without sacrificing the diversity of generated outcomes. Our experiments across MNIST, tinyImageNet, and machine translation benchmarks demonstrate that ev-softmax achieves superior accuracy and efficiency compared to traditional methods.
Introduction
Deep learning architectures typically employ the softmax function to transform neural network outputs into probability distributions. While mathematically convenient, softmax assigns non-zero probability to every possible category. In massive discrete spaces—such as those found in large-scale language modeling or high-resolution image generation—this "denseness" necessitates summing over thousands of possibilities for marginalization, creating a significant computational bottleneck.
Sparse alternatives, most notably sparsemax, address this by zeroing out low-probability candidates. However, these methods often exhibit "mode collapse," where the distribution focuses exclusively on the single most likely outcome, discarding other valid predictions. Furthermore, the presence of exact zeros makes training with standard log-likelihood losses impossible without significant modification.
The Ev-Softmax Method
The proposed ev-softmax (evidential softmax) is designed to provide sparsity while maintaining the ability to represent multiple modes. Unlike projection-based methods, ev-softmax uses the arithmetic mean of the input vector as a threshold.
Geometric Intuition and Formulation
For an input vector $\mathbf{v}$, ev-softmax zeros out any element $v_i$ that falls below the mean $\bar{v}$: $$EVSOFTMAX(\mathbf{v})_i \propto \mathbb{1}\{v_i \geq \bar{v}\}e^{v_i}$$
Training via $\epsilon$-Relaxation
To overcome the training difficulties associated with zero probabilities in log-likelihood losses, the authors introduce a continuous approximation during training: $$EVSOFTMAX_{train, \epsilon}(\mathbf{v})_i \propto (\mathbb{1}\{v_i \geq \bar{v}\} + \epsilon)e^{v_i}$$ By setting $\epsilon$ to a small value (e.g., $10^{-6}$), the distribution maintains full support during training, allowing for standard cross-entropy optimization while the gradients behave like the sparse version as $\epsilon \to 0$. This approach is computationally efficient, running in $O(K)$ time, matching the speed of standard softmax.
Experimental Evaluation
The effectiveness of ev-softmax was evaluated across several domains, focusing on its ability to balance sparsity with predictive diversity.
Multimodality in Generative Tasks
A key advantage of ev-softmax is its resistance to mode collapse. In experiments using Conditional VAEs on MNIST, ev-softmax successfully represented multiple likely digits, whereas sparsemax often collapsed to a single peak.
Performance Benchmarks
- MNIST: Ev-softmax achieved 97.30% accuracy in semi-supervised tasks, outperforming both dense softmax and other sparse methods like entmax.
- tinyImageNet: In VQ-VAE experiments, ev-softmax reduced the number of active latent classes by 85% compared to softmax while actually improving the top-5 accuracy of generated images.
- Machine Translation (IWSLT 2014): The method achieved a state-of-the-art BLEU score of 29.4. Qualitative analysis showed that ev-softmax attended to approximately 8 contextually relevant words, compared to the ~40 words typically attended to by dense softmax, successfully ignoring superfluous words without losing focus on relevant keywords.
Conclusion
Ev-softmax offers a compelling alternative to standard normalization functions in deep generative models. By combining $O(K)$ computational efficiency with a robust mechanism for preserving multimodality, it addresses the primary weaknesses of existing sparse methods. While future work is needed to explore its impact on "foundation-scale" models and potential data biases, ev-softmax represents a significant step toward efficient and diverse discrete representation learning.
Summary updated