Preference-Controlled Multi-Objective Reinforcement Learning for Conditional Text Generation

Wenqing Chen, Jidong Tian, Caoyun Fan, Yitian Li, Hao He, Yaohui JinAAAI 2023 · OpenReview

Abstract

Conditional text generation aims to produce text sequences based on linguistic or non-linguistic input. While existing deterministic models focus on improving fidelity, they often neglect diversity. Conditional Variational Auto-Encoders (CVAEs) improve diversity but treat it as an implicit objective, which is often sub-optimal. This paper addresses two key questions: Can diversity be improved with an explicit objective, and can we obtain various optimal solutions based on user preferences? We propose a preference-controlled multi-objective reinforcement learning (MORL) method using CIDEr and Self-CIDEr as explicit rewards. By tuning a preference variable, our model navigates the fidelity-diversity trade-off, outperforming both deterministic and CVAE-based baselines in paraphrasing and image captioning tasks.

Introduction

In tasks like image captioning and paraphrasing, models face a fundamental trade-off between fidelity (accuracy relative to the input) and diversity (the variety of generated outputs). Deterministic models often favor fidelity at the cost of producing generic, repetitive text. While CVAEs introduce diversity through latent space sampling, they lack explicit control over the degree of diversity.

To solve this, researchers have introduced PCMORL (Preference-Controlled Multi-Objective Reinforcement Learning). This framework provides a "control knob"—a preference variable $r$—that allows users to adjust the balance between being precise and being creative without needing to retrain the model for different preference levels.

Methodology: PACVAE and MORL

The PCMORL framework consists of two primary components: a Preference-Aware CVAE (PACVAE) and a Multi-Objective Reinforcement Learning (MORL) fine-tuning stage.

Preference-Aware CVAE (PACVAE)

The base model is a CVAE modified with a Monotonic Neural Network (MNN). The MNN takes the user preference variable $r \in [0, 1]$ and maps it to the variance of the latent distribution. By design, the MNN ensures that as $r$ increases, the variance of the latent space also increases, leading to more diverse samples during inference.

Architecture of the PACVAE showing the Posterior and Prior Networks with the Monotonic Neural Network.

Multi-Objective Reinforcement Learning

To move beyond implicit goals, the model is fine-tuned using RL with two explicit rewards:

  • CIDEr: A standard metric for fidelity (how well the text matches human references).
  • Self-CIDEr: A metric for diversity (how different the generated sentences in a group are from one another).

The preference variable $r$ weights these two rewards, allowing the model to learn a "Pareto frontier"—a set of optimal solutions where one objective cannot be improved without degrading the other.

Experimental Evaluation

The model was evaluated on two primary tasks across three scenarios:

  • Quora Question Pairs: Used for paraphrasing tasks.
  • MSCOCO (Microsoft Common Objects in Context): Used for both image captioning and paraphrasing.

PCMORL was compared against deterministic Transformers, standard CVAEs, and various RL-tuned baselines.

Results and Insights

PCMORL consistently outperforms baselines by occupying the "top-right" of the fidelity-diversity space. This means for any target level of accuracy, PCMORL provides higher diversity than its competitors.

Fidelity-Diversity trade-off curves (Pareto frontiers) for MSCOCO Image Captioning.

Key Findings:

  • Predictable Control: The MNN ensures that the preference variable $r$ acts as a reliable "knob." Increasing $r$ from 0.0 to 0.9 predictably shifts the model from high-fidelity, nearly identical outputs to a diverse set of creative descriptions.
  • Regularization Effect: Interestingly, optimizing for diversity (even at $r=0$) often improved fidelity scores over deterministic baselines. This suggests that the diversity objective acts as a form of regularization, preventing the model from over-focusing on specific n-grams.
  • Performance Gains: On the Quora paraphrasing dataset, PCMORL achieved a CIDEr score of 267.8, a 13.5 point jump over the deterministic baseline while maintaining significant diversity.

Conclusion and Future Work

PCMORL demonstrates that explicit multi-objective optimization is a powerful tool for controllable text generation. While the current approach is computationally intensive due to the need for Monte-Carlo sampling during the MORL stage, it sets a new standard for balancing accuracy and variety. Future research will explore scaling this preference-control mechanism to large Pre-trained Language Models (PLMs) using parameter-efficient fine-tuning techniques like adapters.

Summary updated

Abstract

Conditional text generation is to generate text sequences conditioning on linguistic or non-linguistic data. The main line of existing work proposed deterministic models to improve the fidelity of the generated text but often ignored the diversity. Another line relied on conditional variational auto-encoders (CVAEs), which increased the diversity over their deterministic backbones. However, CVAEs regard diversity as an implicit objective and may not be optimal. In this paper, we raise two questions: i) Can diversity be further improved with an explicit objective? ii) Since fidelity and diversity are two conflicting objectives, how can we obtain different multi-objective optimal solutions according to user preferences? To answer question i), we propose a multi-objective reinforcement learning (MORL) method which explicitly takes CIDEr and Self-CIDEr scores as the fidelity-oriented and diversity-oriented rewards respectively. To answer question ii), we propose a preference-controlled MORL method, which can obtain infinite multi-objective optimal solutions by tuning the preference variable. We conduct extensive experiments on paraphrasing and image captioning tasks, which show that in the fidelity-diversity trade-off space, our model outperforms both deterministic and CVAE-based baselines.
BibTeX
@inproceedings{chen2023preference,
  author = {Wenqing Chen and Jidong Tian and Caoyun Fan and Yitian Li and Hao He and Yaohui Jin},
  title = {{Preference-Controlled Multi-Objective Reinforcement Learning for Conditional Text Generation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=kRJZ9mZCDF},
}
Figure