Curiosity-driven Reinforcement Learning for Diverse Visual Paragraph Generation

Yadan Luo, Zi Huang, Zheng Zhang, Jingjing Li, Yang Yang

Abstract

Visual paragraph generation aims to automatically describe a given image from different perspectives and organize sentences in a coherent way. In this paper, we address three critical challenges for this task in a reinforcement learning setting: the mode collapse, the delayed feedback, and the time-consuming warm-up for policy networks. Generally, we propose a novel Curiosity-driven Reinforcement Learning (CRL) framework to jointly enhance the diversity and accuracy of the generated paragraphs. First, by modeling the paragraph captioning as a long-term decision-making process and measuring the prediction uncertainty of state transitions as intrinsic rewards, the model is incentivized to memorize precise but rarely spotted descriptions to context, rather than being biased towards frequent fragments and generic patterns. Second, since the extrinsic reward from evaluation is only available until the complete paragraph is generated, we estimate its expected value at each time step with temporal-difference learning, by considering the correlations between successive actions. Then the estimated extrinsic rewards are complemented by dense intrinsic rewards produced from the derived curiosity module, in order to encourage the policy to fully explore action space and find a global optimum. Third, discounted imitation learning is integrated for learning from human demonstrations, without separately performing the time-consuming warm-up in advance. Extensive experiments conducted on the Standford image-paragraph dataset demonstrate the effectiveness and efficiency of the proposed method, improving the performance by 38.4% compared with state-of-the-art.

Introduction

While AI has made significant strides in single-sentence image captioning, generating a cohesive and diverse paragraph remains a formidable challenge. Standard models often struggle with three primary issues:

  1. Mode Collapse: Models tend to favor generic, "safe" phrases (e.g., "two men are standing") that appear frequently in training data, failing to capture the unique, specific details of an image.
  2. Delayed Feedback: In traditional Reinforcement Learning (RL), rewards are typically sparse, only provided after the entire paragraph is generated. This makes it difficult for the model to attribute success to specific word choices.
  3. Training Inefficiency: Most RL approaches require a separate, lengthy supervised "warm-up" phase before the reinforcement learning process can begin.

The CRL Framework

To overcome these hurdles, the researchers introduced the Curiosity-driven Reinforcement Learning (CRL) framework. This system treats paragraph generation as a long-term decision-making process, utilizing both intrinsic and extrinsic rewards to guide the model toward more accurate and diverse descriptions.

The CRL framework architecture, showing the interaction between the internal curiosity modules, the policy network, and the external environment.

The core of CRL is the Curiosity Module, which generates intrinsic rewards based on the model's own "surprise" or prediction uncertainty. By rewarding the model for exploring states that it cannot easily predict, the system is incentivized to move beyond generic patterns and use more specific, descriptive language.

Furthermore, the framework employs Temporal-Difference (TD) Learning to estimate the expected final reward at each step of the generation process. This provides the "dense" feedback necessary for the model to understand the impact of individual actions. Finally, Discounted Imitation Learning integrates human demonstrations directly into the training loop, eliminating the need for a separate pre-training phase and allowing for a smoother transition from imitation to exploration.

Experimental Results and Insights

The CRL framework was rigorously tested on the Stanford image-paragraph dataset. The results were striking: the model achieved a 38.4% improvement in the CIDEr score compared to previous state-of-the-art methods, jumping from 22.74 to 31.47.

Qualitative comparison between the Baseline model and the CRL framework. CRL generates more specific and diverse descriptions compared to the generic outputs of the baseline.

Qualitative analysis reveals that CRL effectively breaks the bias toward frequent n-grams. As shown in the comparison above, while a baseline model might repeat generic phrases like "the sky is blue," CRL captures specific actions and objects, such as a "catcher squatting" or "young boys wearing blue uniforms." This diversity is further supported by semantic network graphs, which indicate that CRL utilizes a much broader vocabulary and more complex object relationships than standard RL methods.

Conclusion

The CRL framework demonstrates that treating model uncertainty as a positive signal for exploration can significantly enhance the quality of complex visual descriptions. By combining dense intrinsic rewards with estimated extrinsic rewards and integrated imitation learning, the model achieves superior performance and diversity. Future work will likely focus on closing the remaining gap between synthetic paragraphs and natural human narrative flow, as well as exploring metrics that better capture logical coherence.

Summary updated

Abstract

Visual paragraph generation aims to automatically describe a given image from different perspectives and organize sentences in a coherent way. In this paper, we address three critical challenges for this task in a reinforcement learning setting: the mode collapse, the delayed feedback, and the time-consuming warm-up for policy networks. Generally, we propose a novel Curiosity-driven Reinforcement Learning (CRL) framework to jointly enhance the diversity and accuracy of the generated paragraphs. First, by modeling the paragraph captioning as a long-term decision-making process and measuring the prediction uncertainty of state transitions as intrinsic rewards, the model is incentivized to memorize precise but rarely spotted descriptions to context, rather than being biased towards frequent fragments and generic patterns. Second, since the extrinsic reward from evaluation is only available until the complete paragraph is generated, we estimate its expected value at each time step with temporal-difference learning, by considering the correlations between successive actions. Then the estimated extrinsic rewards are complemented by dense intrinsic rewards produced from the derived curiosity module, in order to encourage the policy to fully explore action space and find a global optimum. Third, discounted imitation learning is integrated for learning from human demonstrations, without separately performing the time-consuming warm-up in advance. Extensive experiments conducted on the Standford image-paragraph dataset demonstrate the effectiveness and efficiency of the proposed method, improving the performance by 38.4% compared with state-of-the-art.
BibTeX
@inproceedings{luo2019curiosity,
  author = {Yadan Luo and Zi Huang and Zheng Zhang and Jingjing Li and Yang Yang},
  title = {{Curiosity-driven Reinforcement Learning for Diverse Visual Paragraph Generation}},
  booktitle = {ACM Multimedia},
  year = {2019},
  doi = {10.1145/3343031.3350961},
}
Figure