Prompt-Based Length Controlled Generation with Multiple Control Types

Renlong Jie, Xiaojun Meng, Lifeng Shang, Xin Jiang, Qun Liu

Abstract

Large language models (LLMs) have demonstrated exceptional performance across diverse NLP tasks, yet precise control over output length remains a significant challenge. While most existing research focuses on "equal to" constraints, real-world applications often require more flexible bounds, such as "less than," "more than," or "between." This paper introduces a prompt-based method to achieve high-accuracy length control across multiple constraint types. By integrating a Standard Prompt Extractor (SPE), rule-based reward models, and reinforcement learning (RL), the proposed framework enhances the model's ability to follow complex length instructions. Experiments on the CNNDM and NYT datasets demonstrate that this approach significantly reduces length errors and generalizes effectively to unseen prompt templates.

Introduction

The autoregressive nature of Large Language Models—generating text one token at a time—makes it inherently difficult for them to plan the final length of an output. For users, this often results in summaries that are either too verbose or too brief for specific constraints. Traditional methods for length control typically involve architectural modifications or are limited to simple "exact length" targets. However, practical utility requires a broader range of control types.

This research addresses the gap by proposing a system that handles diverse length constraints through natural language prompts. By treating length control as a reinforcement learning problem guided by objective, rule-based rewards, the system achieves precise adherence to user requirements without sacrificing text quality.

Methodology

The framework consists of three primary components designed to bridge the gap between messy user input and precise model generation.

1. Standard Prompt Extractor (SPE)

Users express length constraints in varied ways (e.g., "Keep it under 50 words" vs. "A short summary of max 50 tokens"). The SPE acts as a pre-processor that parses these diverse inputs into a Standard Control Prompt (SCP). The researchers evaluated both generative (GPT-style) and discriminative (BERT-style) extractors, finding that a BERT-based parser could achieve up to 99.9% accuracy in mapping user intent to standard constraints like less than 50.

2. Rule-Based Reward Model

Unlike Reinforcement Learning from Human Feedback (RLHF), which relies on subjective human preferences, length is an objective metric. The authors utilized a rule-based reward model that provides consistent, mathematical feedback. For instance, if a prompt specifies "less than 100 tokens," any output exceeding that limit receives a negative reward proportional to the violation, while outputs within the limit receive a neutral or positive reward. This eliminates the latency and inconsistency associated with neural reward models.

3. Reinforcement Learning and Sample Filtering

The model is fine-tuned using Proximal Policy Optimization (PPO). To ensure the model retains its linguistic capabilities while learning length control, the training objective includes a Supervised Fine-Tuning (SFT) loss term.

At the inference stage, the system employs Sample Filtering. The model generates multiple candidate outputs (using a beam size of 8), and the reward model selects the candidate that best satisfies the length constraint. This dual approach—training the model to be "length-aware" and then filtering its best efforts—yields the highest accuracy.

Experimental Results

The method was evaluated using GPT-2 (Small, Medium, and Large) on the CNN/Daily Mail (CNNDM) and New York Times (NYT) summarization datasets.

Performance Gains

The combination of RL and sample filtering showed dramatic improvements. On the CNNDM dataset, the length control error for GPT-Small was reduced from 18.16 to 7.06. Across all model sizes, the "Prompt+RL+filter" configuration consistently outperformed standard prompting or RL-only approaches.

Generalization and Control Types

The system proved robust across four control types: Equal, Less, More, and Between. While "More than" constraints proved the most difficult—likely due to the models' inherent bias toward conciseness in summarization—the "Between" constraint showed significant improvement over baseline models. Crucially, the SPE demonstrated strong generalization, successfully parsing prompt templates that were not present during the training phase.

Discussion and Limitations

A key insight from the study is the trade-off between control and quality. While aggressive RL can improve length accuracy, it can occasionally lead to a slight decrease in ROUGE scores. The inclusion of the SFT loss during training is vital to mitigating this degradation.

However, the method introduces certain overheads:

  • Inference Latency: Sample filtering requires generating multiple candidates, which increases computational time.
  • Complexity: The system relies on an external parser (SPE); if the parser fails, the subsequent control mechanism fails.
  • Scalability: While effective on GPT-2, fine-tuning massive models (e.g., 70B+ parameters) via PPO is computationally expensive, suggesting a need for more efficient tuning methods like LoRA in future iterations.

Conclusion

This research provides a robust framework for multi-type length control in LLMs. By combining the flexibility of prompt engineering with the precision of rule-based reinforcement learning, the authors have created a system that respects user-defined constraints while maintaining high-quality text generation. Future work will focus on reducing inference latency and applying these techniques to larger, state-of-the-art models.

Summary updated

Abstract

Large language models (LLMs) have attracted great attention given their strong performance on a wide range of NLP tasks. In practice, users often expect generated texts to fall within a specific length range, making length controlled generation an important topic, especially for GPT-style models. Existing length control methods mostly focus on a simple control type of "equal to" a target length. Different from them, we propose a prompt-based method to achieve length controlled generation under different control types with high accuracy. In particular, we adopt reinforcement learning (RL) and sample filtering with the reward signal given by rule-based reward models, which enhances the length control ability of models by rewarding outputs that follow certain control instructions. In addition, we introduce a standard prompt extractor to parse arbitrary users' input into standard control instructions. Experiments show that our method significantly improves the accuracy of prompt-based length control on popular summarization datasets like CNNDM and NYT under multiple control types. Moreover, both the standard prompt extractor and RL-tuned model show strong generalization to unseen control prompt templates.
BibTeX
@inproceedings{jie2024prompt,
  author = {Renlong Jie and Xiaojun Meng and Lifeng Shang and Xin Jiang and Qun Liu},
  title = {{Prompt-Based Length Controlled Generation with Multiple Control Types}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=8R4SOcRTcx},
}
Figure