Abstract
Sequence-to-sequence (Seq2Seq) learning often requires measuring the semantic similarity between generated and target sequences. Traditional Optimal Transport (OT) methods, like the Word Mover's Distance, enforce a total matching of all words, which can be problematic in tasks like style transfer or image captioning where only a subset of content is relevant. We introduce SEPAM (SEmantic PArtial Matching), a framework that utilizes Optimal Partial Transport (OPT) to align sequences by focusing on semantically important words identified through prior linguistic knowledge.
Introduction
In Seq2Seq modeling, measuring the distance between a generated sentence and a target is crucial for training. Optimal Transport (OT) has emerged as a powerful tool for this, treating sentences as distributions of words. However, classical OT requires all "mass" from the source to be moved to the target. This total matching constraint is often too restrictive for real-world applications:
- Style Transfer: Content words (e.g., "food") should be preserved, while style words (e.g., "bad" to "good") must be transformed.
- Non-Parallel Data: In tasks like image captioning, the source might be a set of object labels rather than a full sentence.
- Hallucinations: Models often generate extra information not present in the source; standard OT lacks a mechanism to penalize these unmatched tokens effectively.
Methodology: Optimal Partial Transport
SEPAM addresses these limitations by replacing classical OT with Optimal Partial Transport (OPT). Unlike standard OT, OPT allows the model to match only a sub-portion of the sequences, effectively ignoring irrelevant or noisy tokens.
Active Region Detection
The primary challenge in OPT is determining which parts of a sequence are worth matching. SEPAM leverages prior knowledge to define these "active regions":
- Text-to-Text: Part-of-Speech (POS) tags identify content words (nouns, verbs) for matching while allowing style-specific adjectives to remain unmatched.
- Image-to-Text: Detected object labels are matched to specific nouns in the generated caption.
- Table-to-Text: Structured data keys are matched with nouns in the text to ensure factual fidelity.
Implementation
To ensure efficiency and end-to-end trainability, SEPAM utilizes the Inexact Proximal Optimal Transport (IPOT) algorithm. A "soft-argmax" approximation is employed to maintain differentiability, allowing the OPT cost to serve as a regularizer alongside standard Maximum Likelihood Estimation (MLE) loss.
Results and Discussion
SEPAM was evaluated across three tasks: unsupervised text-style transfer (Yelp), image captioning (MSCOCO), and table-to-text generation (WikiPerson).
Key Insights
- Sparsity and Alignment: Visualizations of the matching matrices reveal that SEPAM produces significantly cleaner and sparser alignments compared to classical OT. By focusing on core semantic meaning, the model avoids the "noise" of function words and style markers.
- Hallucination Penalty: In table-to-text tasks, SEPAM acts as a "hallucination tax." If the model generates a noun that lacks a corresponding entry in the source table, the OPT cost increases, training the model to remain faithful to the provided data.
Performance Metrics
- Style Transfer: On the Yelp dataset, SEPAM achieved a ref-BLEU of 28.2, significantly outperforming the base language model's 25.7.
- Image Captioning: In unsupervised settings, SEPAM reached a 42.1 BLEU score, surpassing the previous state-of-the-art of 38.2.
- Table-to-Text: The framework improved the PARENT score (measuring factual accuracy) from 52.60 to 53.16.
Conclusion
SEPAM demonstrates that incorporating linguistic priors into an Optimal Partial Transport framework significantly improves content preservation and factual accuracy in Seq2Seq tasks. While currently dependent on the quality of pre-processors like POS taggers, future work may involve learning these active regions dynamically within the model architecture.
Summary updated