Suri: Multi-constraint Instruction Following in Long-form Text Generation

Chau Pham, Simeng Sun, Mohit IyyerEMNLP 2024 · DOI

Introduction

Modern Large Language Models (LLMs) have demonstrated remarkable proficiency in following instructions for short-form tasks. However, as the required output length increases and the complexity of constraints grows, these models often struggle to maintain coherence and satisfy all user requirements. Researchers from UMass Amherst have introduced Suri, a comprehensive framework comprising a novel dataset and an alignment method designed to bridge this gap. Suri enables models to generate high-quality, long-form text (exceeding 5,000 tokens) while adhering to approximately ten simultaneous semantic and stylistic constraints.

The Challenge of Long-Form Alignment

The development of long-form instruction-following models faces two primary obstacles: data scarcity and the evaluation bottleneck. Most existing instruction-tuning datasets are dominated by short prompts and brief responses, which fail to provide the necessary signal for long-context generation. Furthermore, standard alignment techniques like Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO) rely on human or model-based rankings of different outputs. For a 5,000-word text, verifying the satisfaction of ten distinct constraints is prohibitively expensive for humans and increasingly unreliable for automated judges.

Methodology: Instructional ORPO (I-ORPO)

To address these challenges without the need for expensive human labeling of long-form preferences, the authors developed a two-stage pipeline: Instruction Backtranslation and Instructional ORPO (I-ORPO).

The Suri training pipeline: Instruction backtranslation generates the correct instruction ($x_w$) from human-written text $y$, followed by constraint corruption to create the dispreferred instruction ($x_l$) for I-ORPO training.

Instruction Backtranslation

Rather than manually authoring complex prompts, the researchers utilized high-quality, human-written long texts from sources like fanfiction (ChapterBreak), books (Books3), and web content (RedPajama). They then employed GPT-4-turbo to "reverse-engineer" these texts, generating a "main goal" and a set of constraints that the text naturally satisfies. This results in a pair $(x_w, y)$, where $x_w$ is the "winning" or correct instruction for the response $y$.

Instructional ORPO

Standard preference optimization typically compares a preferred response to a dispreferred one for a single prompt. I-ORPO inverts this logic. By taking the correct instruction $x_w$ and "corrupting" its constraints (e.g., changing a stylistic requirement from "humorous" to "serious"), the researchers create a "losing" instruction $x_l$. The model is then trained to maximize the probability of the response $y$ given $x_w$ while minimizing it given $x_l$. This approach teaches the model to distinguish the specific nuances of constraint satisfaction directly from the instruction space.

The Suri Dataset

The resulting Suri dataset contains 20,000 single-turn examples. Unlike the popular Alpaca dataset, which features an average response length of 44 tokens, Suri's responses average approximately 4,371 tokens. This provides a rich training signal for long-form coherence and multi-constraint adherence.

Results and Evaluation

Fine-tuning on Suri led to a massive increase in output length. While the base Mistral-7B-Instruct-v0.2 model averaged roughly 1,000 tokens per response, the Suri-aligned models consistently generated outputs near the 5,000-token mark.

Training dynamics showing the model's increasing ability to distinguish between correct ($x_w$) and corrupted ($x_l$) instructions over training epochs.

Empirical results show that Suri-I-ORPO outperforms standard Supervised Fine-Tuning (SFT) baselines. In human evaluations, Suri-I-ORPO was preferred 67-73% of the time across metrics including coherence, informativeness, and overall enjoyability. Furthermore, the model showed a 10% improvement in ranking accuracy—the ability to correctly identify which instruction set a given text actually followed.

The Unreliability of LLM Judges

A critical finding of the study is that even state-of-the-art models like GPT-4o and Claude 3.5 Sonnet are poor judges of long-form constraint satisfaction. GPT-4o's evaluations aligned with human annotators only 39% of the time, underscoring the necessity of human-in-the-loop evaluation for long-context research.

Discussion and Limitations

While Suri represents a significant step forward, several areas for future work remain. The authors noted a trade-off between length and repetition; without careful epoch management, models can fall into repetitive n-gram loops at high token counts. Additionally, because the training data was truncated to 5,000 words for efficiency, SFT models occasionally produced abrupt or inconsistent endings. Future research will explore how these techniques scale to larger model architectures (70B+) and investigate the robustness of I-ORPO to varying prompting styles.

Summary updated

BibTeX
@inproceedings{pham2024suri,
  author = {Chau Pham and Simeng Sun and Mohit Iyyer},
  title = {{Suri: Multi-constraint Instruction Following in Long-form Text Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.94},
}
Figure