Abstract
Large language models (LLMs) have attracted great attention given their strong performance on a wide range of NLP tasks. In practice, users often expect generated texts to fall within a specific length range, making length controlled generation an important topic, especially for GPT-style models. Existing length control methods mostly focus on a simple control type of “equal to” a target length. Different from them, we propose a prompt-based method to achieve length controlled generation under different control types with high accuracy. In particular, we adopt reinforcement learning (RL) and sample filtering with the reward signal given by rule-based reward models, which enhances the length control ability of models by rewarding outputs that follow certain control instructions. In addition, we introduce a standard prompt extractor to parse arbitrary users’ input into standard control instructions. Experiments show that our method significantly improves the accuracy of prompt-based length control on popular summarization datasets like CNNDM and NYT under multiple control types. Moreover, both the standard prompt extractor and RL-tuned model show strong generalization to unseen control prompt templates.
Introduction
While Large Language Models (LLMs) demonstrate remarkable capabilities in text generation, they are notoriously poor at adhering to precise length constraints—a phenomenon often described as the model's inability to "count." In real-world scenarios, users frequently require outputs that meet specific criteria, such as "less than 50 words" or "between 100 and 150 tokens."
Most existing research into length control focuses on a single, rigid constraint: making the output "equal to" a target length. Furthermore, natural language instructions for these constraints are highly varied (e.g., "Give me a short blurb" vs. "Summarize in less than 30 words"). This paper addresses these challenges by proposing a framework that maps diverse natural language instructions to standardized constraints and utilizes reinforcement learning to optimize model performance across multiple control types.
Methodology
The proposed architecture consists of four primary components designed to decouple intent extraction from text generation and optimization.
1. Standard Prompt Extractor (SPE)
To handle the inherent variability of human language, the framework employs an independent model (such as BERT or a smaller GPT) to parse arbitrary user inputs into a Standard Control Prompt (SCP). For instance, an instruction like "Can you make this shorter than 100 words?" is converted into a structured format like less than 100 tokens. This ensures the generative model only needs to learn to follow a limited set of standardized instructions.
2. Rule-Based Reward Model
Unlike subjective qualities like "helpfulness," length compliance is a hard metric that can be evaluated mathematically. The authors designed rule-based functions to score how well a generated sequence fits a constraint. If a target is "less than 100" and the model produces 110, it receives a negative reward proportional to the deviance.
3. Reinforcement Learning (RL)
The framework utilizes the Proximal Policy Optimization (PPO) algorithm to fine-tune GPT models. During training, the model is rewarded for generating summaries that satisfy the SCPs parsed by the SPE. This process enhances the model's internal ability to regulate output length according to the provided prompt.
4. Inference-Time Sample Filtering
To further refine accuracy, the model generates multiple candidate summaries (e.g., 8 candidates) during inference. The reward model ranks these candidates based on their adherence to the length constraint, and the highest-scoring candidate is selected as the final output.
Experimental Results
The method was evaluated on the CNNDM (CNN/Daily Mail) and NYT (New York Times) summarization benchmarks. The combination of RL and sample filtering (Prompt+RL+filter) achieved significant improvements over baselines:
- Error Reduction: On the NYT dataset using a GPT-Small model, the average length error dropped from 17.62 tokens to just 3.89 tokens.
- Complex Constraints: The model showed exceptional performance on "Between" constraints, reducing the error on the CNNDM dataset to a mere 1.09 tokens.
- Generalization: The system demonstrated strong zero-shot generalization. When tested on prompt templates never seen during training, error rates remained nearly as low as in-sample tests, validating the robustness of the Standard Prompt Extractor.
- Quality Preservation: Importantly, reducing length error did not degrade summary quality. Metrics like ROUGE and BERTScore remained stable, suggesting that the filtering process often selects candidates that are both length-compliant and highly relevant.
Discussion and Limitations
While highly effective, the framework introduces certain trade-offs. The Inference-Time Sample Filtering stage requires generating multiple sequences, which increases computational latency. Additionally, there is a minor risk of "language modeling degradation," where the model might prioritize length over linguistic fluency; however, this was largely mitigated by incorporating a Supervised Fine-Tuning (SFT) loss term during RL. Future work may explore applying these techniques to larger models like Llama-3 or utilizing in-context learning to achieve similar control without full fine-tuning.
Summary updated