Sequence-to-Sequence (Seq2Seq) models have revolutionized natural language processing, but their standard training regimens often leave performance on the table. This paper introduces a sequence-level training objective based on Optimal Transport (OT). By moving beyond traditional word-level Maximum Likelihood Estimation (MLE), the authors provide a robust method to capture global semantic structures and alleviate common generation issues, leading to state-of-the-art performance in translation, summarization, and image captioning.
The Flaws of Standard MLE Training
Standard Seq2Seq models are typically trained using Maximum Likelihood Estimation (MLE) with "teacher forcing." The model learns to predict the next word given the ground-truth previous words. While straightforward, this approach has two major flaws:
- Exposure Bias: During inference, the model generates words based on its own previous predictions rather than ground truth. If the model makes an early mistake, errors accumulate, leading to "drifting" and nonsensical sequences.
- Loss-Evaluation Mismatch: MLE optimizes for exact word-level accuracy, but we evaluate models using sequence-level, semantic metrics like BLEU or ROUGE. A model might achieve a high MLE score by matching local syntax while completely failing to capture the overall meaning of the sentence.
While Reinforcement Learning (RL) and Generative Adversarial Networks (GANs) have been proposed to address these sequence-level issues, they are notoriously difficult to train due to high gradient variance and stability problems.
A New Approach: Optimal Transport
To bridge the gap between word-level training and sequence-level evaluation, the authors propose adding an Optimal Transport (OT) loss as a regularizer to the standard MLE objective.
OT as Sequence Matching
Optimal Transport (also known as the Earth Mover’s Distance) finds the "cheapest" way to transform one distribution of words into another. Unlike "hard matching" (which requires exact word overlaps and heavily penalizes paraphrasing), OT allows for "soft matching" based on the semantic similarity of word embeddings.
Because OT operates on continuous word embeddings, it recognizes that phrases like "do you want to have lunch" and "would you like to join us for lunch" are semantically similar, even if they lack exact word overlap. MLE would penalize the difference heavily; OT does not.
Differentiable Sequence Generation and IPOT
To integrate this into a neural network, gradients must flow from the OT loss back into the model. The authors use a soft-argmax operator to approximate the discrete word selection process with a continuous, differentiable one.
To make the OT computation efficient and stable, they employ the Inexact Proximal point method for Optimal Transport (IPOT). The authors found IPOT to be more numerically stable than the commonly used Sinkhorn algorithm. Furthermore, they introduce a "soft-copying" mechanism that matches source sequence embeddings to target sequence embeddings, encouraging the model to preserve core semantic content from the input.
Theoretically, this method acts as an approximate Wasserstein Gradient Flow, pushing the model's generated distribution toward the true data distribution in a geometrically meaningful way.
Experimental Results
The researchers evaluated their method across three major NLP tasks, demonstrating consistent improvements over pure MLE baselines:
- Neural Machine Translation (NMT): On the English-Vietnamese task (IWSLT 2015), the OT-augmented model improved BLEU scores from 26.1 to 26.9. On the larger English-German dataset (WMT 2017), it successfully outperformed a strong Transformer baseline.
- Abstractive Text Summarization: On the Gigaword dataset (3.8M samples), ROUGE-L scores improved from 32.4 to 34.0.
- Image Captioning: On the MS COCO dataset, the model achieved a CIDEr score of 102.9 (compared to 99.1 for the baseline), indicating that the generated captions were more semantically faithful to the images.
Qualitative examples further show that the OT model is significantly better at retaining key entities and words from the source text compared to vanilla models, which often ignore or misinterpret them.
Key Insights and Limitations
The OT loss provides a stable, end-to-end supervised signal. Unlike GANs, it doesn't require a delicate min-max game with a discriminator, and unlike RL, it avoids high-variance reward signals. Additionally, the OT loss helps refine the word embedding matrix itself by forcing it to better represent semantic relationships that minimize transport cost.
Limitations: * Computational Overhead: Calculating OT for every batch is more expensive than simple cross-entropy, though the authors note the increase in training time is relatively small (about 10%). * Operator Sensitivity: The choice of differentiable operator matters. The authors found that using Gumbel-Softmax led to unstable training compared to soft-argmax, suggesting the extra randomness interferes with the OT gradient.
Future Directions: The authors suggest applying this framework to other sequence tasks where semantic consistency is notoriously difficult to maintain, such as conversational response generation for chatbots.
Summary updated