Off-Policy Self-Critical Training for Transformer in Visual Paragraph Generation

Shiyang Yan, Yang Hua, Neil Martin RobertsonarXiv 2020 · OpenReview

Abstract

Transformer models have become the state-of-the-art for sequence-to-sequence tasks, yet their integration with Reinforcement Learning (RL) remains computationally prohibitive due to the high cost of sampling. We propose an off-policy RL algorithm where a lightweight behavior policy (GRU-based) performs exploration, while a complex Meshed Transformer serves as the target policy. To stabilize training, we employ Truncated Relative Importance Sampling (TRIS) and KL-control. Our approach achieves state-of-the-art performance on visual paragraph generation and improves image captioning results while significantly reducing computational overhead.

Introduction: The Computational Hurdle of Paragraph Generation

Generating coherent 200-word paragraphs from images is significantly more complex than standard image captioning. While Maximum Likelihood Estimation (MLE) is the standard training objective, it suffers from exposure bias, where errors accumulate during inference because the model was only trained on ground-truth sequences.

Reinforcement Learning (RL), specifically Self-Critical Sequence Training (SCST), can mitigate this by optimizing non-differentiable metrics like CIDEr. However, on-policy RL requires the Transformer to sample from its own distribution, creating a massive memory bottleneck as gradient graphs are built for every word in a long paragraph.

Proposed Method: The "Scout" Strategy

To make RL feasible for Transformers, we shift from an on-policy to an off-policy framework. We introduce a "behavior policy"—a simpler, faster GRU-based image-guided auto-encoder—to act as a scout. This model explores the environment and generates word samples, which the expensive Transformer model then learns from.

The proposed off-policy RL framework featuring a Meshed Transformer target policy and a GRU-based behavior policy.

To bridge the gap between the behavior policy and the target policy, we implement three critical components:

  1. Importance Sampling (IS): Corrects the mismatch between the two probability distributions.
  2. Truncated Relative Importance Sampling (TRIS): Bounds the IS ratios to prevent gradient explosion or vanishing.
  3. KL-Control: Adds a Kullback-Leibler divergence penalty to keep the Transformer from drifting too far from the behavior policy, theoretically reducing variance.

Taming Variance in the Action Space

In Natural Language Processing, the "action space" is the entire vocabulary, often spanning tens of thousands of words. This makes off-policy learning notoriously unstable. Vanilla Importance Sampling often results in ratios that explode, leading to divergent training.

Comparison of Importance Sampling (IS) ratios. Vanilla IS (a) shows extreme variance, while TRIS with KL penalty (d) stabilizes the ratio near 1.0.

As shown in our ablation studies, vanilla IS ratios can exceed 3000. By applying TRIS and KL-control, we successfully bound these ratios, ensuring stable and effective gradient updates. This stability is the key to making off-policy RL viable for large-scale language models.

Key Results

We evaluated our method on the Stanford Visual Paragraph Dataset and MS-COCO:

  • State-of-the-Art Performance: Our model outperformed previous hierarchical and flat models across all major metrics (BLEU, METEOR, CIDEr) on the Stanford dataset.
  • Surpassing Human Baselines: The model achieved a BLEU-4 score of 9.8, marginally exceeding the human annotation baseline of 9.7.
  • Efficiency: By offloading the sampling process to a GRU, we made Transformer RL training feasible on standard hardware, removing the need for massive compute clusters.
  • Generalization: The approach also improved CIDEr scores on the MS-COCO image captioning task from 88.1 to 90.0.

Conclusion

By decoupling exploration from the primary model, we have demonstrated that complex Transformers can be efficiently trained using Reinforcement Learning for long-form text generation. The combination of TRIS and KL-control is essential for managing the high variance of off-policy sampling in large discrete action spaces. Future work will explore reducing the overhead of maintaining two separate models while further refining sample efficiency.

Summary updated

Abstract

Recently, several approaches have been proposed to solve language generation problems. Transformer is currently state-of-the-art seq-to-seq model in language generation. Reinforcement Learning (RL) is useful in solving exposure bias and the optimisation on non-differentiable metrics in seq-to-seq language learning. However, Transformer is hard to combine with RL as the costly computing resource is required for sampling. We tackle this problem by proposing an off-policy RL learning algorithm where a behaviour policy represented by GRUs performs the sampling. We reduce the high variance of importance sampling (IS) by applying the truncated relative importance sampling (TRIS) technique and Kullback-Leibler (KL)-control concept. TRIS is a simple yet effective technique, and there is a theoretical proof that KL-control helps to reduce the variance of IS. We formulate this off-policy RL based on self-critical sequence training. Specifically, we use a Transformer-based captioning model as the target policy and use an image-guided language auto-encoder as the behaviour policy to explore the environment. The proposed algorithm achieves state-of-the-art performance on the visual paragraph generation and improved results on image captioning.
BibTeX
@misc{yan2020off,
  author = {Shiyang Yan and Yang Hua and Neil Martin Robertson},
  title = {{Off-Policy Self-Critical Training for Transformer in Visual Paragraph Generation}},
  year = {2020},
  eprint = {2006.11714},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.11714},
}
Figure