Utility-Probability Duality of Neural Networks

Bojun Huang, Fei YuanarXiv 2023 · OpenReview

TL;DR

This paper challenges the common understanding that neural networks learn probability distributions. Instead, it proposes that they learn "utility functions" that encode preferences, explaining why greedy decoding often outperforms probability-maximizing approaches in tasks like machine translation. This "utility-probability duality" leads to a new way of estimating probabilities that significantly improves performance.

What Problem Does This Paper Address?

The prevailing view in deep learning is that supervised training aims to fit the groundtruth probability distribution of the data. However, this explanation struggles to account for counter-intuitive observations in tasks like language generation. Specifically, methods that aim to maximize predicted probabilities (like MAP decoding or sampling) often perform poorly, while simpler "greedy" approaches (picking the most likely token at each step) yield much better results. This paradox suggests that the standard probabilistic interpretation might be incomplete or even incorrect.

Methods

The authors propose a shift in perspective: instead of viewing neural networks as probability models, they interpret them as ordinal utility functions. This means the network learns to assign scores (utilities) to different outputs, reflecting preferences rather than strict probabilities.

  • Utility-Based Interpretation: The core idea is that the training process, typically framed as Maximum Likelihood Estimation (MLE), can be re-interpreted as learning an optimal utility function.
  • Q-Learning Analogy: The paper connects the MLE objective to a Q-learning algorithm (MABE), where the network's output (logits) is treated as a Q-value representing utility.
  • Dual Probabilities: A key theoretical contribution is an equation that transforms these learned Q-values into a new type of probability estimation, termed "dual probabilities." This transformation aims to better reflect the underlying utility.

Datasets

The experiments were conducted on several standard benchmarks:

  • Machine Translation:
    • WMT'14 English→German (en2de)
    • WMT'17 Chinese→English (zh2en)
  • Document Summarization:
    • CNN/DailyMail

Key Insights and Ideas

  • The "Greedy" Paradox: The most striking observation is that greedy decoding (picking the locally most probable token) often outperforms sophisticated methods like MAP decoding or sampling, which are theoretically grounded in probability maximization. This paper provides a theoretical explanation for this phenomenon.
  • Utility over Probability: The authors argue that neural networks are better understood as learning utility functions that capture preferences, rather than precise probability distributions. This aligns better with empirical success of greedy methods.
  • MLE as Utility Learning: The standard MLE training process can be viewed as a form of Q-learning, optimizing for utility rather than just probability.
  • Dual Probabilities: The proposed "dual probabilities" offer a more accurate probabilistic representation derived from the learned utility, leading to significant performance improvements when used in standard decision rules.

Key Results

  • Improved Performance with Dual Probabilities: Using dual probabilities for sampling and MAP estimation led to substantial gains. For example, in WMT'14 en2de translation, sampling with dual probabilities achieved a BLEU score of 20.0, a +14.6 improvement over softmax probabilities. MAP estimation with dual probabilities also saw a significant boost (+17.3 BLEU).
  • Greedy Decoding Dominance: Across tasks, greedy decoding consistently performed well, often outperforming sampling and MAP estimation based on softmax probabilities.
  • MABE Variants: Experiments with different MABE(λ) variants (representing perturbations of the MLE gradient) showed that the unperturbed MABE(0) often performed as well as or slightly better than the MLE-equivalent MABE(1), supporting the utility-based interpretation.
  • Better Probability Calibration: Dual probabilities provided more reasonable probability estimates, correctly assigning near-zero probability to undesirable outputs like empty translations.

Limitations and Future Work

  • Theoretical Foundation: While the paper provides strong empirical evidence and a theoretical framework, it acknowledges that the theory is defended by a combination of mathematical and experimental evidence, rather than pure mathematics.
  • Algorithm Refinement: The MABE(λ) family of algorithms could be further developed and fine-tuned. The "dual probability" formula involves a heuristic clipping and normalization step that could be improved.
  • Search Algorithms: Exploring more advanced search algorithms (like Monte Carlo Tree Search) that can better leverage the dual probability model is suggested.
  • Near-Greedy Performance: The theory currently struggles to fully explain why near-greedy outputs (e.g., small beam sizes) can sometimes slightly outperform pure greedy outputs.
  • Generalizability: The authors suggest testing the utility-based theory in a wider range of machine learning tasks.

Summary updated

Abstract

It is typically understood that the training of modern neural networks is a process of fitting the probability distribution of desired output. However, recent paradoxical observations in a number of language generation tasks let one wonder if this canonical probability-based explanation can really account for the empirical success of deep learning. To resolve this issue, we propose an alternative utility-based explanation to the standard supervised learning procedure in deep learning. The basic idea is to interpret the learned neural network not as a probability model but as an ordinal utility function that encodes the preference revealed in training data. In this perspective, training of the neural network corresponds to a utility learning process. Specifically, we show that for all neural networks with softmax outputs, the SGD learning dynamic of maximum likelihood estimation (MLE) can be seen as an iteration process that optimizes the neural network toward an optimal utility function. This utility-based interpretation can explain several otherwise-paradoxical observations about the neural networks thus trained. Moreover, our utility-based theory also entails an equation that can transform the learned utility values back to a new kind of probability estimation with which probability-compatible decision rules enjoy dramatic (double-digits) performance improvements. These evidences collectively reveal a phenomenon of utility-probability duality in terms of what modern neural networks are (truly) modeling: We thought they are one thing (probabilities), until the unexplainable showed up; changing mindset and treating them as another thing (utility values) largely reconcile the theory, despite remaining subtleties regarding its original (probabilistic) identity.
BibTeX
@misc{huang2023utility,
  author = {Bojun Huang and Fei Yuan},
  title = {{Utility-Probability Duality of Neural Networks}},
  year = {2023},
  eprint = {2305.14859},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.14859},
}

Loading related content...

Figure