TL;DR
This paper introduces Imbalanced Preference Optimization (IPO), a novel framework for improving audio-based text generation (like speech translation and audio captioning) using limited human feedback. IPO tackles the scarcity of preference data by using a model-adversarial sampling approach and an imbalanced optimization strategy, leading to significant performance gains on existing models.
What Problem Does This Paper Address?
Generating high-quality text from audio inputs, such as in speech translation or audio captioning, is a challenging task. While Reinforcement Learning from Human Feedback (RLHF) has proven effective for aligning text-based models (like LLMs) with human preferences, its application to audio-based generation is hindered by a critical bottleneck: the lack of readily available, labeled preference data. Annotating audio is more labor-intensive than text or images, requiring listeners to repeatedly process audio segments. This scarcity makes it difficult to train models that can accurately understand and generate natural language from audio. Furthermore, existing audio-text models often struggle with the complex and variable nature of audio signals, leading to errors in transcription and subsequent text generation.
Methods
The core of this paper is the Imbalanced Preference Optimization (IPO) framework. IPO introduces two key innovations:
- Model Adversarial Sampling: Instead of relying solely on direct human comparisons of model outputs, IPO pits two audio-text models against each other. A human annotator acts as a "referee," deciding which model's output is superior for a given audio input. This win-loss signal is then used to generate preference data. Crucially, this signal is extended beyond the immediate output using a beam search strategy. This means that if Model A wins against Model B on one output, other hypotheses generated by Model A via beam search are also implicitly favored, effectively expanding the limited human annotations.
- Imbalanced Optimization: Recognizing that the win-loss data generated by adversarial sampling can be imbalanced (one model might win more often than the other), IPO employs an optimization strategy that can handle this imbalance and unpaired preference data. It doesn't require perfectly matched preferred and dispreferred pairs, making it more flexible. The optimization process iteratively trains the models to "win" against each other, effectively distilling knowledge from stronger models to weaker ones and improving both.
Datasets
The paper evaluates its approach on the following datasets:
- FLEURS: Used for Speech Translation (AST) experiments. This benchmark provides audio segments and their corresponding transcriptions across various languages.
- Clotho: Used for Audio Captioning (AAC) experiments. This dataset contains audio clips paired with descriptive captions.
- LibriTTS: Used for Text-to-Speech (TTS) synthesis experiments, demonstrating the extensibility of IPO to a reverse task.
The authors also mention using GPT-4 to simulate human annotations for AST, leveraging its multilingual capabilities.
Key Insights and Ideas
- Data Efficiency is Key: The research highlights that even a few hundred human annotations can significantly improve audio-based text generation models, demonstrating the data efficiency of IPO.
- Weak Models Benefit Most, Strong Models Improve: IPO not only boosts the performance of weaker baseline models but also pushes stronger models to achieve new state-of-the-art results.
- Cross-Modal Alignment is Achievable: The work shows that human preference alignment, a successful paradigm in text-only models, can be effectively extended to audio-text generation.
- Adversarial Learning for Knowledge Distillation: The adversarial sampling mechanism acts as a form of knowledge distillation, allowing models to learn from each other's strengths and weaknesses.
- Extensibility Beyond Generation: IPO's applicability extends to tasks like zero-shot TTS, showcasing its versatility.
Key Results
- Speech Translation (AST): With just 100 annotations, IPO achieved clear BLEU improvements on the FLEURS benchmark for both Whisper-large-v2 and SeamlessM4T-large-v2 models. For instance, SeamlessM4T's De→En translation saw a 1.1 BLEU improvement. The stronger SeamlessM4T model was pushed to new state-of-the-art results.
- Audio Captioning (AAC): Similar significant improvements were observed on the Clotho dataset for both Pengi and Qwen-Audio models, with CIDEr scores showing notable gains even with limited annotations.
- Text-to-Speech (TTS): IPO demonstrated improved zero-shot TTS capabilities, enhancing metrics like MOS, WER, and SIM for models like VC-330M.
- Subjective Evaluation: A/B tests with human evaluators consistently showed IPO-enhanced models outperforming their baselines, validating the effectiveness of the approach.
Limitations and Future Work
- LLM Annotation for AST: While GPT-4 was used to simulate annotations for AST, this method relies on ground-truth source text and cannot be directly applied to tasks without it.
- Iterative Optimization: The paper suggests that iterative finetuning can further improve performance, but notes that performance gains eventually plateau, approaching the upper bound of what's achievable with the given preference data.
- Generalizability: While IPO shows promise across AST, AAC, and TTS, further exploration across a wider range of audio-based generation tasks would be beneficial.
- Human Annotator Bias: The paper acknowledges that human annotators might prefer longer captions, which is addressed by a length penalty, but this highlights the potential for subjective biases to influence training.
The authors propose that LLMs can replace human annotators when ground truth is available, and that the adversarial optimization method enables efficient distillation, allowing weaker models to selectively absorb knowledge.
Summary updated