Jam or Cream First? Modeling Ambiguity in Neural Machine Translation with SCONES

Felix Stahlberg, Shankar KumarNAACL 2022 · DOI

Introduction

Standard Neural Machine Translation (NMT) models rely on a softmax layer, which operates under the assumption that there is only one "correct" next word at any given step. However, language is inherently ambiguous; a single source sentence can often be translated into multiple equally valid target sequences. This paper introduces SCONES (Single-label Contrastive Objective for Non-Exclusive Sequences), a method that replaces the traditional softmax with a multi-label classification approach. By allowing models to assign high probabilities to multiple valid translations simultaneously, SCONES creates a more robust translation framework that mitigates the "beam search curse" and achieves significant inference speedups.

The Problem of Intrinsic Uncertainty

In traditional NMT, the final layer is a softmax function that forces the model to distribute a total probability of 1.0 across the entire vocabulary. This creates a "zero-sum game": if a source sentence has two equally valid translations ($y_1$ and $y_2$), the model must split its confidence (e.g., 0.5 each).

This mathematical constraint fails to distinguish between model uncertainty (where the model is confused due to lack of training) and intrinsic uncertainty (where the language itself allows multiple correct answers). This limitation contributes to the "beam search curse," a phenomenon where increasing the search space (beam size) actually leads to worse translations. In these cases, the model begins to favor short, high-probability but inadequate sequences because the probability mass is spread too thin across valid alternatives.

Methodology: From Softmax to SCONES

The core technical shift in SCONES involves moving from a global softmax to independent sigmoid activations for every token in the vocabulary.

SCONES architecture replacing the standard softmax layer with independent sigmoid activations for multi-label classification.

This treats translation as a multi-label classification task, enabling the model to theoretically assign a probability of 1.0 to multiple different tokens at once. To train this architecture with standard datasets—which typically provide only one reference translation—the authors designed the SCONES loss function to balance two goals:

  1. Positive Component: Maximize the probability of the "gold" token from the training data.
  2. Negative Component: Minimize the probability of all other tokens in the vocabulary, weighted by a scaling factor $\alpha$.

Tuning the hyperparameter $\alpha$ allows the model to adjust its tolerance for alternative, non-gold tokens. A lower $\alpha$ makes the model more "tolerant" of ambiguity, which the authors found to be a key factor in stabilizing the probability landscape.

Evaluation and Key Results

The researchers evaluated SCONES on several WMT19 benchmarks, including German-English (de-en), Finnish-English (fi-en), and Lithuanian-English (lt-en). They also utilized synthetic data generated via an IBM Model 3 to isolate how well SCONES handles varying levels of controlled ambiguity.

Translation quality (BLEU) vs. inference speed (sentences per second) for English-German, comparing SCONES to standard Softmax.

The results demonstrate several key advantages:

  • Performance Gains: SCONES yielded consistent BLEU score improvements (1% to 9% relative) across all six translation directions.
  • Massive Speedup: Because SCONES is better calibrated toward adequate translations, its "greedy search" (picking the single best word at each step) often outperforms the complex beam search of standard softmax models. This allows for a 3.9x inference speedup while maintaining or exceeding the same BLEU score.
  • Solving the Beam Search Curse: Unlike softmax models, where BLEU scores often drop sharply as beam size increases beyond 5 or 10, SCONES models show remarkably flat curves. Even with a beam size of 1000, the translation quality remains high, proving the model's robustness to larger search spaces.

Conclusion and Future Work

SCONES demonstrates that modeling ambiguity directly leads to more robust and efficient NMT systems. By moving away from the "one right answer" constraint of softmax, the model avoids penalizing itself for recognizing valid linguistic alternatives. While the optimal scaling factor $\alpha$ varies by language pair, the method inherently handles overconfidence without requiring traditional techniques like label smoothing. Future work could extend SCONES to other highly ambiguous NLP tasks, such as open-ended dialogue generation or creative writing, where intrinsic uncertainty is even more prevalent than in translation.

Summary updated

BibTeX
@inproceedings{stahlberg2022jam,
  author = {Felix Stahlberg and Shankar Kumar},
  title = {{Jam or Cream First? Modeling Ambiguity in Neural Machine Translation with SCONES}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-main.365},
}
Figure