TL;DR
This paper introduces GTA, a novel framework that combines supervised learning (SFT) and reinforcement learning (RL) for text classification with large language models (LLMs). GTA significantly accelerates convergence and improves performance by having the LLM first make a supervised "guess," then "think" through reasoning, and finally produce a refined "answer," all optimized through a hybrid training approach.
What Problem Does This Paper Address?
Large language models (LLMs) have revolutionized natural language processing, but adapting them for specific tasks like text classification presents a challenge. Pure supervised fine-tuning (SFT) is efficient but has performance limitations. Reinforcement learning (RL) offers a path to higher performance by enabling models to learn reasoning strategies without explicit human annotation, but traditional RL methods often suffer from slow convergence and inefficient exploration. This paper aims to bridge this gap by developing a method that leverages the efficiency of SFT with the performance gains of RL, specifically for text classification.
Methods
The core innovation is the Guess-Think-Answer (GTA) framework. This framework structures the LLM's output into three distinct stages:
- Guess: The model first generates a preliminary answer based on intuition. This "guess" stage is trained using standard cross-entropy loss (SFT), which helps accelerate convergence.
- Think: The model then engages in an explicit reasoning process, reflecting on its initial guess and the input to improve its understanding and generalization.
- Answer: Finally, the model produces a refined answer, integrating insights from both the guess and think stages. The quality of the final answer and the overall output structure are optimized using RL rewards.
To manage the potentially conflicting signals from SFT and RL, the authors employ:
- Loss Masking: This ensures that the SFT loss only applies to the "guess" segment and the RL objective focuses on the "think" and "answer" segments, preventing interference.
- Gradient Conflict Detection and Resolution: They use cosine similarity to detect gradient conflicts between SFT and RL. If a conflict is detected, they prioritize the RL objective to mitigate interference.
Datasets
The experiments were conducted on four publicly available text classification datasets:
- SST-5: Stanford Sentiment Treebank (5-class sentiment classification).
- Amazon: A subset of the Massive Text Embedding Benchmark (MTEB) for intent recognition.
- Emotion: A dataset of English tweets annotated with six basic emotions.
- BBC News: A dataset for news topic classification.
These datasets vary in the number of classes and training samples, covering different domains and complexities.
Key Insights and Ideas
- Supervised Guess as a Warm-up: Using SFT for the initial "guess" acts as an effective warm-up, guiding the RL process and significantly accelerating convergence compared to pure RL.
- Reasoning Improves Robustness: Even when the initial guess is incorrect, the "think" stage allows the model to correct its mistakes and arrive at the right answer, demonstrating improved robustness over methods that only rely on a single prediction.
- Bridging the Efficiency-Capability Gap: GTA successfully combines the efficiency of SFT with the superior performance potential of RL, offering a practical solution for fine-tuning LLMs.
- RL for Reasoning Without Annotation: The framework allows LLMs to learn effective reasoning patterns through RL without requiring costly manual annotation of reasoning chains.
Key Results
- Performance Gains: GTA consistently outperformed both standard SFT and the GRPO (a strong RL baseline) across all four datasets and multiple model sizes (3B and 4B parameters). For instance, on the Emotion dataset, GTA achieved an F₁ score of 92.47% with Qwen2.5, surpassing other methods.
- Faster Convergence: GTA demonstrated significantly faster convergence than pure RL methods, as evidenced by accuracy reward trajectories. Even on the SST-5 dataset, GTA reached higher performance levels within 10,000 steps compared to GRPO.
- Zero-Shot Reasoning Benefits: Experiments showed that even without fine-tuning, incorporating explicit reasoning steps ("think" mode) improved classification accuracy across all tested datasets, highlighting the inherent benefit of structured reasoning.
Limitations and Future Work
- Model Scale: The experiments were limited to 3B and 4B parameter models due to resource constraints. The authors suggest that evaluating on larger-scale models could yield even greater benefits, as RL's performance is often tied to model capacity.
- Dataset Quality: The study did not perform extensive preprocessing or sample high-quality subsets from the publicly available datasets, which might contain noisy data.
- Future Directions: The authors plan to extend GTA to broader NLP tasks beyond text classification, leveraging its ability to combine supervised learning efficiency with RL-driven reasoning capabilities.
Summary updated