Abstract
Transformer models have become the state-of-the-art for sequence-to-sequence tasks, yet their integration with Reinforcement Learning (RL) remains computationally prohibitive due to the high cost of sampling. We propose an off-policy RL algorithm where a lightweight behavior policy (GRU-based) performs exploration, while a complex Meshed Transformer serves as the target policy. To stabilize training, we employ Truncated Relative Importance Sampling (TRIS) and KL-control. Our approach achieves state-of-the-art performance on visual paragraph generation and improves image captioning results while significantly reducing computational overhead.
Introduction: The Computational Hurdle of Paragraph Generation
Generating coherent 200-word paragraphs from images is significantly more complex than standard image captioning. While Maximum Likelihood Estimation (MLE) is the standard training objective, it suffers from exposure bias, where errors accumulate during inference because the model was only trained on ground-truth sequences.
Reinforcement Learning (RL), specifically Self-Critical Sequence Training (SCST), can mitigate this by optimizing non-differentiable metrics like CIDEr. However, on-policy RL requires the Transformer to sample from its own distribution, creating a massive memory bottleneck as gradient graphs are built for every word in a long paragraph.
Proposed Method: The "Scout" Strategy
To make RL feasible for Transformers, we shift from an on-policy to an off-policy framework. We introduce a "behavior policy"—a simpler, faster GRU-based image-guided auto-encoder—to act as a scout. This model explores the environment and generates word samples, which the expensive Transformer model then learns from.
To bridge the gap between the behavior policy and the target policy, we implement three critical components:
- Importance Sampling (IS): Corrects the mismatch between the two probability distributions.
- Truncated Relative Importance Sampling (TRIS): Bounds the IS ratios to prevent gradient explosion or vanishing.
- KL-Control: Adds a Kullback-Leibler divergence penalty to keep the Transformer from drifting too far from the behavior policy, theoretically reducing variance.
Taming Variance in the Action Space
In Natural Language Processing, the "action space" is the entire vocabulary, often spanning tens of thousands of words. This makes off-policy learning notoriously unstable. Vanilla Importance Sampling often results in ratios that explode, leading to divergent training.
As shown in our ablation studies, vanilla IS ratios can exceed 3000. By applying TRIS and KL-control, we successfully bound these ratios, ensuring stable and effective gradient updates. This stability is the key to making off-policy RL viable for large-scale language models.
Key Results
We evaluated our method on the Stanford Visual Paragraph Dataset and MS-COCO:
- State-of-the-Art Performance: Our model outperformed previous hierarchical and flat models across all major metrics (BLEU, METEOR, CIDEr) on the Stanford dataset.
- Surpassing Human Baselines: The model achieved a BLEU-4 score of 9.8, marginally exceeding the human annotation baseline of 9.7.
- Efficiency: By offloading the sampling process to a GRU, we made Transformer RL training feasible on standard hardware, removing the need for massive compute clusters.
- Generalization: The approach also improved CIDEr scores on the MS-COCO image captioning task from 88.1 to 90.0.
Conclusion
By decoupling exploration from the primary model, we have demonstrated that complex Transformers can be efficiently trained using Reinforcement Learning for long-form text generation. The combination of TRIS and KL-control is essential for managing the high variance of off-policy sampling in large discrete action spaces. Future work will explore reducing the overhead of maintaining two separate models while further refining sample efficiency.
Summary updated