Evidential Softmax for Sparse Multimodal Distributions in Deep Generative Models

Phil Chen, Masha Itkina, Ransalu Senanayake, Mykel J. Kochenderfer

TL;DR

This paper introduces "evidential softmax" (ev-softmax), a novel normalization function for deep generative models that balances sparsity with multimodality in probability distributions. Unlike existing sparse methods that often collapse distributions, ev-softmax allows for more accurate modeling and training with standard probabilistic loss functions, leading to improved performance in tasks like image generation and machine translation.

What Problem Does This Paper Address?

Many powerful deep generative models, especially those dealing with discrete latent spaces (like in image generation or natural language processing), rely on probability distributions. A common challenge is that these distributions can be very high-dimensional and dense, making computations like marginalization (averaging over possibilities) computationally intractable. To address this, researchers have developed "sparse" normalization functions that assign zero probability to many possibilities, simplifying calculations. However, these sparse functions often force the distribution to be unimodal (having only one peak), which can lose crucial information about multiple plausible interpretations or states. This loss of multimodality can hurt performance in tasks where diverse outputs are important. Furthermore, many sparse methods require custom loss functions, making them harder to integrate into existing frameworks.

Methods

The core contribution is the evidential softmax (ev-softmax) function. It's designed to be a sparse normalization function that preserves multimodality. The authors achieve this by drawing inspiration from "evidential theory," which allows distinguishing between "conflicting evidence" (multimodality) and "lack of evidence" (sparsity).

Key aspects of their method include:

  • Evidential Sparsification: They build upon a post-hoc evidential sparsification technique, formalizing it into a direct transformation.
  • Closed-form Gradient: They derive the gradient of ev-softmax, which is crucial for efficient training.
  • Continuous Approximations: To enable training with standard probabilistic loss functions (like negative log-likelihood and KL divergence), they introduce continuous approximations of ev-softmax that have "full support" (no zero probabilities). This allows them to be used directly during training, unlike previous post-hoc methods.
  • Training-time Modification: For practical training, they propose a slight modification to ev-softmax that introduces a small epsilon to ensure full support and enable gradient flow, which converges to the desired sparse behavior as epsilon approaches zero.

Datasets

The paper evaluates ev-softmax on several standard datasets:

  • MNIST: Used for Conditional Variational Autoencoder (CVAE) experiments to generate digit images.
  • tinyImageNet: Used for Vector-Quantized Variational Autoencoder (VQ-VAE) experiments for image generation.
  • IWSLT 2014 (English-German): Used for machine translation tasks.
  • WMT 2016 (English-German): Used for pretraining transformer models for machine translation.

Key Insights and Ideas

  • Sparsity vs. Multimodality Trade-off: The paper highlights that achieving sparsity doesn't have to come at the cost of multimodality. Evidential theory provides a principled way to balance these two desirable properties.
  • Training with Probabilistic Losses: The ability to train ev-softmax directly with negative log-likelihood and KL divergence is a significant practical advantage, making it easier to integrate into existing deep learning pipelines compared to methods requiring custom loss functions.
  • Evidential Theory's Role: The connection to evidential theory is key, as it provides a framework to interpret weights as evidence for or against a class, naturally leading to both sparsity and the preservation of multiple plausible interpretations.
  • Improved Generative Model Performance: The results suggest that by better modeling the true underlying distributions (maintaining multimodality while achieving sparsity), generative models can produce more accurate and higher-quality outputs.

Key Results

  • Distributional Accuracy: Ev-softmax consistently outperforms dense (softmax) and other sparse normalization techniques (sparsemax, entmax-1.5, post-hoc evidential) in distributional accuracy across various tasks.
  • Image Generation (CVAE & VQ-VAE): Models trained with ev-softmax generated more sensible and accurate images, with better representation of latent classes compared to baselines. For instance, in the CVAE MNIST experiment, ev-softmax learned a distribution with five nonzero probability modes that captured the even/odd digit classes effectively.
  • Machine Translation: Ev-softmax achieved the highest BLEU, METEOR, and ROUGE scores on the IWSLT 2014 dataset, indicating improved translation quality. It also maintained a higher number of attended source words compared to other sparse methods, suggesting a more nuanced attention mechanism.
  • Reduced Latent Space Dimensionality: Ev-softmax effectively reduces the dimensionality of probability distributions while preserving multimodality, a key goal of sparse methods. For example, in the semi-supervised VAE task, it reduced the latent space by 84% on average while achieving the highest accuracy.

Limitations and Future Work

  • Theoretical Guarantees: The authors acknowledge that they do not provide explicit theoretical guarantees for when a class probability will be exactly zero.
  • Bias Amplification: Like any model that learns from data, the use of sparse probability distributions could potentially amplify inherent biases present in the data or modeling assumptions.
  • Future Work: The authors suggest future research could focus on training larger computer vision and machine translation models from scratch using ev-softmax and exploring its efficacy on more complex downstream tasks. They also hope their work inspires further research into improving generative modeling accuracy and reducing bias.

Summary updated

Abstract

Many applications of generative models rely on the marginalization of their high-dimensional output probability distributions. Normalization functions that yield sparse probability distributions can make exact marginalization more computationally tractable. However, sparse normalization functions usually require alternative loss functions for training since the log-likelihood is undefined for sparse probability distributions. Furthermore, many sparse normalization functions often collapse the multimodality of distributions. In this work, we present $\textit{ev-softmax}$, a sparse normalization function that preserves the multimodality of probability distributions. We derive its properties, including its gradient in closed-form, and introduce a continuous family of approximations to $\textit{ev-softmax}$ that have full support and can be trained with probabilistic loss functions such as negative log-likelihood and Kullback-Leibler divergence. We evaluate our method on a variety of generative models, including variational autoencoders and auto-regressive architectures. Our method outperforms existing dense and sparse normalization techniques in distributional accuracy. We demonstrate that $\textit{ev-softmax}$ successfully reduces the dimensionality of probability distributions while maintaining multimodality.
BibTeX
@inproceedings{chen2021evidential,
  author = {Phil Chen and Masha Itkina and Ransalu Senanayake and Mykel J. Kochenderfer},
  title = {{Evidential Softmax for Sparse Multimodal Distributions in Deep Generative Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=rqjfa49ODLE},
}
Figure