Abstract
Nowadays, there exist powerful language models such as OpenAI's GPT-2 that can generate readable text and can be fine-tuned to generate text for a specific domain. Considering GPT-2, it cannot directly generate synthetic news with respect to a given topic and the output of the language model cannot be explicitly controlled. In this paper, we study the novel problem of topic-preserving synthetic news generation. We propose a novel deep reinforcement learning-based method to control the output of GPT-2 with respect to a given news topic. When generating text using GPT-2, by default, the most probable word is selected from the vocabulary. Instead of selecting the best word each time from GPT-2's output, an RL agent tries to select words that optimize the matching of a given topic. In addition, using a fake news detector as an adversary, we investigate generating realistic news using our proposed method. In this paper, we consider realistic news as news that cannot be easily detected by a fake news classifier. Experimental results demonstrate the effectiveness of the proposed framework on generating topic-preserving news content than state-of-the-art baselines.
Introduction
Large-scale language models like GPT-2 are capable of generating remarkably human-like text. However, they lack explicit control mechanisms, often leading to "topic drift" where the generated content deviates from the intended subject. Traditional methods like fine-tuning are computationally expensive and do not always ensure topic consistency. This research introduces RLTG (Reinforcement Learning-based Text Generator), a framework designed to steer language models to remain topic-focused while maintaining high fluency and realism.
The RLTG Framework
The core of the proposed approach is the RLTG framework, which frames text generation as a Reinforcement Learning (RL) task. Rather than allowing the language model (GPT-2) to autonomously select the most probable next word, a Deep Q-Network (DQN) agent acts as a controller to make the final selection.
Key Components:
- Action Space: To maintain grammatical coherence, the agent's choices are restricted to the top 50 words suggested by GPT-2.
- State Representation: The agent perceives the context through two autoencoders that process the language model's hidden states and the positions of the top-K words.
- Reward Function: The agent is optimized based on three signals: topic similarity (cosine similarity to the target topic), fluency (BLEU score against real news), and adversarial stealth (the ability to bypass a fake news detector).
Key Insights
The RLTG approach offers several advantages over end-to-end fine-tuning. By using RL as a "controller" on top of a frozen or lightly fine-tuned model, the system achieves fine-grained control without the need to retrain billions of parameters. The "Top-K" constraint ensures that the RL agent provides the direction while the language model provides the necessary linguistic structure. Furthermore, incorporating a fake news detector as an adversary creates a training environment where the generator learns to avoid common machine-generated patterns.
Key Results
The RLTG framework was evaluated using the FakeNewsNet dataset (GossipCop and Politifact). It demonstrated superior performance in several key areas:
- Topic Matching: RLTG achieved a topic similarity score of 0.342, outperforming Grover (0.313) and standard fine-tuned GPT-2 (0.241).
- Fluency: The model recorded the lowest Perplexity (14.8), indicating more coherent and less repetitive text.
- Human Evaluation: In blind tests, human evaluators rated RLTG-generated content significantly higher than Grover across style, content quality, and topic similarity.
Conclusion and Future Work
RLTG successfully demonstrates that adversarial reinforcement learning can effectively control large language models for topic-preserving news generation. While the current study focused on 50-word segments in gossip and political domains, future work will explore long-form generation and expansion into more technical domains where factual accuracy is paramount.
Summary updated