Evidential Softmax for Sparse Multimodal Distributions in Deep Generative Models

Phil Chen, Masha Itkina, Ransalu Senanayake, Mykel Kochenderfer

Abstract

In deep generative modeling, the standard softmax function is often used to parameterize discrete distributions. However, its dense nature poses computational challenges in large-scale settings, while existing sparse alternatives like sparsemax frequently suffer from mode collapse. We introduce ev-softmax, a computationally efficient sparse normalization function that preserves multimodality. By utilizing mean-based thresholding and a continuous training-time relaxation, ev-softmax enables sparse predictions without sacrificing the diversity of generated outcomes. Our experiments across MNIST, tinyImageNet, and machine translation benchmarks demonstrate that ev-softmax achieves superior accuracy and efficiency compared to traditional methods.

Introduction

Deep learning architectures typically employ the softmax function to transform neural network outputs into probability distributions. While mathematically convenient, softmax assigns non-zero probability to every possible category. In massive discrete spaces—such as those found in large-scale language modeling or high-resolution image generation—this "denseness" necessitates summing over thousands of possibilities for marginalization, creating a significant computational bottleneck.

Sparse alternatives, most notably sparsemax, address this by zeroing out low-probability candidates. However, these methods often exhibit "mode collapse," where the distribution focuses exclusively on the single most likely outcome, discarding other valid predictions. Furthermore, the presence of exact zeros makes training with standard log-likelihood losses impossible without significant modification.

The Ev-Softmax Method

The proposed ev-softmax (evidential softmax) is designed to provide sparsity while maintaining the ability to represent multiple modes. Unlike projection-based methods, ev-softmax uses the arithmetic mean of the input vector as a threshold.

Geometric Intuition and Formulation

For an input vector $\mathbf{v}$, ev-softmax zeros out any element $v_i$ that falls below the mean $\bar{v}$: $$EVSOFTMAX(\mathbf{v})_i \propto \mathbb{1}\{v_i \geq \bar{v}\}e^{v_i}$$

Geometric comparison of normalization functions on a 3-class simplex. Sparsemax maps to the boundary (sparsity), while ev-softmax provides a different sparse mapping that avoids the "winner-take-all" behavior of projection-based methods.

Training via $\epsilon$-Relaxation

To overcome the training difficulties associated with zero probabilities in log-likelihood losses, the authors introduce a continuous approximation during training: $$EVSOFTMAX_{train, \epsilon}(\mathbf{v})_i \propto (\mathbb{1}\{v_i \geq \bar{v}\} + \epsilon)e^{v_i}$$ By setting $\epsilon$ to a small value (e.g., $10^{-6}$), the distribution maintains full support during training, allowing for standard cross-entropy optimization while the gradients behave like the sparse version as $\epsilon \to 0$. This approach is computationally efficient, running in $O(K)$ time, matching the speed of standard softmax.

Experimental Evaluation

The effectiveness of ev-softmax was evaluated across several domains, focusing on its ability to balance sparsity with predictive diversity.

Multimodality in Generative Tasks

A key advantage of ev-softmax is its resistance to mode collapse. In experiments using Conditional VAEs on MNIST, ev-softmax successfully represented multiple likely digits, whereas sparsemax often collapsed to a single peak.

Comparison of probability distributions and generated digits for Even/Odd tasks. Sparsemax (red) exhibits mode collapse, focusing on a single digit, while ev-softmax (green) preserves multiple valid modes.

Performance Benchmarks

  • MNIST: Ev-softmax achieved 97.30% accuracy in semi-supervised tasks, outperforming both dense softmax and other sparse methods like entmax.
  • tinyImageNet: In VQ-VAE experiments, ev-softmax reduced the number of active latent classes by 85% compared to softmax while actually improving the top-5 accuracy of generated images.
  • Machine Translation (IWSLT 2014): The method achieved a state-of-the-art BLEU score of 29.4. Qualitative analysis showed that ev-softmax attended to approximately 8 contextually relevant words, compared to the ~40 words typically attended to by dense softmax, successfully ignoring superfluous words without losing focus on relevant keywords.

Conclusion

Ev-softmax offers a compelling alternative to standard normalization functions in deep generative models. By combining $O(K)$ computational efficiency with a robust mechanism for preserving multimodality, it addresses the primary weaknesses of existing sparse methods. While future work is needed to explore its impact on "foundation-scale" models and potential data biases, ev-softmax represents a significant step toward efficient and diverse discrete representation learning.

Summary updated

Abstract

Many applications of generative models rely on the marginalization of their high-dimensional output probability distributions. Normalization functions that yield sparse probability distributions can make exact marginalization more computationally tractable. However, sparse normalization functions usually require alternative loss functions for training since the log-likelihood is undefined for sparse probability distributions. Furthermore, many sparse normalization functions often collapse the multimodality of distributions. In this work, we present ev-softmax, a sparse normalization function that preserves the multimodality of probability distributions. We derive its properties, including its gradient in closed-form, and introduce a continuous family of approximations to ev-softmax that have full support and can be trained with probabilistic loss functions such as negative log-likelihood and Kullback-Leibler divergence. We evaluate our method on a variety of generative models, including variational autoencoders and auto-regressive architectures. Our method outperforms existing dense and sparse normalization techniques in distributional accuracy. We demonstrate that ev-softmax successfully reduces the dimensionality of probability distributions while maintaining multimodality.
BibTeX
@inproceedings{chen2021evidential,
  author = {Phil Chen and Masha Itkina and Ransalu Senanayake and Mykel Kochenderfer},
  title = {{Evidential Softmax for Sparse Multimodal Distributions in Deep Generative Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=rqjfa49ODLE},
}
Figure