TL;DR
This paper challenges the common understanding that neural networks learn probability distributions. Instead, it proposes that they learn "utility functions" that encode preferences, explaining why greedy decoding often outperforms probability-maximizing approaches in tasks like machine translation. This "utility-probability duality" leads to a new way of estimating probabilities that significantly improves performance.
What Problem Does This Paper Address?
The prevailing view in deep learning is that supervised training aims to fit the groundtruth probability distribution of the data. However, this explanation struggles to account for counter-intuitive observations in tasks like language generation. Specifically, methods that aim to maximize predicted probabilities (like MAP decoding or sampling) often perform poorly, while simpler "greedy" approaches (picking the most likely token at each step) yield much better results. This paradox suggests that the standard probabilistic interpretation might be incomplete or even incorrect.
Methods
The authors propose a shift in perspective: instead of viewing neural networks as probability models, they interpret them as ordinal utility functions. This means the network learns to assign scores (utilities) to different outputs, reflecting preferences rather than strict probabilities.
- Utility-Based Interpretation: The core idea is that the training process, typically framed as Maximum Likelihood Estimation (MLE), can be re-interpreted as learning an optimal utility function.
- Q-Learning Analogy: The paper connects the MLE objective to a Q-learning algorithm (MABE), where the network's output (logits) is treated as a Q-value representing utility.
- Dual Probabilities: A key theoretical contribution is an equation that transforms these learned Q-values into a new type of probability estimation, termed "dual probabilities." This transformation aims to better reflect the underlying utility.
Datasets
The experiments were conducted on several standard benchmarks:
- Machine Translation:
- WMT'14 English→German (en2de)
- WMT'17 Chinese→English (zh2en)
- Document Summarization:
- CNN/DailyMail
Key Insights and Ideas
- The "Greedy" Paradox: The most striking observation is that greedy decoding (picking the locally most probable token) often outperforms sophisticated methods like MAP decoding or sampling, which are theoretically grounded in probability maximization. This paper provides a theoretical explanation for this phenomenon.
- Utility over Probability: The authors argue that neural networks are better understood as learning utility functions that capture preferences, rather than precise probability distributions. This aligns better with empirical success of greedy methods.
- MLE as Utility Learning: The standard MLE training process can be viewed as a form of Q-learning, optimizing for utility rather than just probability.
- Dual Probabilities: The proposed "dual probabilities" offer a more accurate probabilistic representation derived from the learned utility, leading to significant performance improvements when used in standard decision rules.
Key Results
- Improved Performance with Dual Probabilities: Using dual probabilities for sampling and MAP estimation led to substantial gains. For example, in WMT'14 en2de translation, sampling with dual probabilities achieved a BLEU score of 20.0, a +14.6 improvement over softmax probabilities. MAP estimation with dual probabilities also saw a significant boost (+17.3 BLEU).
- Greedy Decoding Dominance: Across tasks, greedy decoding consistently performed well, often outperforming sampling and MAP estimation based on softmax probabilities.
- MABE Variants: Experiments with different MABE(λ) variants (representing perturbations of the MLE gradient) showed that the unperturbed MABE(0) often performed as well as or slightly better than the MLE-equivalent MABE(1), supporting the utility-based interpretation.
- Better Probability Calibration: Dual probabilities provided more reasonable probability estimates, correctly assigning near-zero probability to undesirable outputs like empty translations.
Limitations and Future Work
- Theoretical Foundation: While the paper provides strong empirical evidence and a theoretical framework, it acknowledges that the theory is defended by a combination of mathematical and experimental evidence, rather than pure mathematics.
- Algorithm Refinement: The MABE(λ) family of algorithms could be further developed and fine-tuned. The "dual probability" formula involves a heuristic clipping and normalization step that could be improved.
- Search Algorithms: Exploring more advanced search algorithms (like Monte Carlo Tree Search) that can better leverage the dual probability model is suggested.
- Near-Greedy Performance: The theory currently struggles to fully explain why near-greedy outputs (e.g., small beam sizes) can sometimes slightly outperform pure greedy outputs.
- Generalizability: The authors suggest testing the utility-based theory in a wider range of machine learning tasks.
Summary updated