Abstract
Large language models (LLMs) have demonstrated exceptional performance across diverse NLP tasks, yet precise control over output length remains a significant challenge. While most existing research focuses on "equal to" constraints, real-world applications often require more flexible bounds, such as "less than," "more than," or "between." This paper introduces a prompt-based method to achieve high-accuracy length control across multiple constraint types. By integrating a Standard Prompt Extractor (SPE), rule-based reward models, and reinforcement learning (RL), the proposed framework enhances the model's ability to follow complex length instructions. Experiments on the CNNDM and NYT datasets demonstrate that this approach significantly reduces length errors and generalizes effectively to unseen prompt templates.
Introduction
The autoregressive nature of Large Language Models—generating text one token at a time—makes it inherently difficult for them to plan the final length of an output. For users, this often results in summaries that are either too verbose or too brief for specific constraints. Traditional methods for length control typically involve architectural modifications or are limited to simple "exact length" targets. However, practical utility requires a broader range of control types.
This research addresses the gap by proposing a system that handles diverse length constraints through natural language prompts. By treating length control as a reinforcement learning problem guided by objective, rule-based rewards, the system achieves precise adherence to user requirements without sacrificing text quality.
Methodology
The framework consists of three primary components designed to bridge the gap between messy user input and precise model generation.
1. Standard Prompt Extractor (SPE)
Users express length constraints in varied ways (e.g., "Keep it under 50 words" vs. "A short summary of max 50 tokens"). The SPE acts as a pre-processor that parses these diverse inputs into a Standard Control Prompt (SCP). The researchers evaluated both generative (GPT-style) and discriminative (BERT-style) extractors, finding that a BERT-based parser could achieve up to 99.9% accuracy in mapping user intent to standard constraints like less than 50.
2. Rule-Based Reward Model
Unlike Reinforcement Learning from Human Feedback (RLHF), which relies on subjective human preferences, length is an objective metric. The authors utilized a rule-based reward model that provides consistent, mathematical feedback. For instance, if a prompt specifies "less than 100 tokens," any output exceeding that limit receives a negative reward proportional to the violation, while outputs within the limit receive a neutral or positive reward. This eliminates the latency and inconsistency associated with neural reward models.
3. Reinforcement Learning and Sample Filtering
The model is fine-tuned using Proximal Policy Optimization (PPO). To ensure the model retains its linguistic capabilities while learning length control, the training objective includes a Supervised Fine-Tuning (SFT) loss term.
At the inference stage, the system employs Sample Filtering. The model generates multiple candidate outputs (using a beam size of 8), and the reward model selects the candidate that best satisfies the length constraint. This dual approach—training the model to be "length-aware" and then filtering its best efforts—yields the highest accuracy.
Experimental Results
The method was evaluated using GPT-2 (Small, Medium, and Large) on the CNN/Daily Mail (CNNDM) and New York Times (NYT) summarization datasets.
Performance Gains
The combination of RL and sample filtering showed dramatic improvements. On the CNNDM dataset, the length control error for GPT-Small was reduced from 18.16 to 7.06. Across all model sizes, the "Prompt+RL+filter" configuration consistently outperformed standard prompting or RL-only approaches.
Generalization and Control Types
The system proved robust across four control types: Equal, Less, More, and Between. While "More than" constraints proved the most difficult—likely due to the models' inherent bias toward conciseness in summarization—the "Between" constraint showed significant improvement over baseline models. Crucially, the SPE demonstrated strong generalization, successfully parsing prompt templates that were not present during the training phase.
Discussion and Limitations
A key insight from the study is the trade-off between control and quality. While aggressive RL can improve length accuracy, it can occasionally lead to a slight decrease in ROUGE scores. The inclusion of the SFT loss during training is vital to mitigating this degradation.
However, the method introduces certain overheads:
- Inference Latency: Sample filtering requires generating multiple candidates, which increases computational time.
- Complexity: The system relies on an external parser (SPE); if the parser fails, the subsequent control mechanism fails.
- Scalability: While effective on GPT-2, fine-tuning massive models (e.g., 70B+ parameters) via PPO is computationally expensive, suggesting a need for more efficient tuning methods like LoRA in future iterations.
Conclusion
This research provides a robust framework for multi-type length control in LLMs. By combining the flexibility of prompt engineering with the precision of rule-based reinforcement learning, the authors have created a system that respects user-defined constraints while maintaining high-quality text generation. Future work will focus on reducing inference latency and applying these techniques to larger, state-of-the-art models.
Summary updated