Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment

Geyang Guo, Ranchi Zhao, Tianyi Tang, Wayne Xin Zhao, Ji-Rong Wen

Abstract

Large Language Model (LLM) alignment typically relies on Supervised Fine-Tuning (SFT) or Reinforcement Learning from Human Feedback (RLHF). While SFT suffers from distribution shift and a lack of discriminative signal, RLHF is often unstable and computationally expensive. We present FIGA (Fine-grained Quality Signals for Alignment), a framework that bridges this gap by training models to contrast their own subpar outputs with revised, high-quality versions. By leveraging token-level edit distances, FIGA explicitly penalizes errors and reinforces corrections. Our experiments show that FIGA consistently outperforms SFT and RLHF baselines across diverse benchmarks, providing a more stable and efficient path to alignment.

Introduction

Aligning LLMs with human values—helpfulness, honesty, and harmlessness—is a critical challenge. Current paradigms like SFT act as imitation learning, where the model copies "gold" responses without understanding the specific attributes that make them superior. Conversely, RLHF explores the model's output space but is notoriously difficult to tune and prone to reward collapse.

The authors identify a "missing middle": a method that provides the stability of supervised learning while incorporating the discriminative power of RL. FIGA achieves this by showing the model exactly which tokens in its own generated text require correction.

The FIGA Framework

The FIGA framework consists of two primary stages: dataset curation and fine-grained tuning.

SubPar Alignment (SPA) Dataset Construction

Rather than relying solely on external expert data, FIGA curates the SPA Dataset by identifying and correcting the model's own failures.

  1. Rollout: The initial model generates responses to a set of instructions.
  2. Filtering: A reward model identifies "subpar" responses—those significantly worse than a reference.
  3. Revision: A teacher model (e.g., ChatGPT) performs minimal revisions on these subpar outputs, transforming them into high-quality responses while maintaining the original style.
The FIGA pipeline: Rollout, Reward-based filtering, Revision by a teacher LLM, and Alignment using Levenshtein distance.

Fine-grained Quality-Aware Tuning

Alignment is treated as a correction task. Using Levenshtein distance, the framework categorizes tokens in the revised response:

  • Added/Substituted tokens: Identified as "good actions" to be reinforced.
  • Deleted/Substituted tokens: Identified as "bad actions" from the original output to be penalized.
  • Unchanged tokens: Neutral tokens that are largely ignored during the specific alignment loss calculation.

This weighted loss function mimics the "advantage function" in RL but operates within a stable supervised framework.

Experimental Results

The authors evaluated FIGA against several baselines, including SFT and PPO, using the Alpaca-7B model as a starting point.

Performance Benchmarks

FIGA achieved an aggregate score of 34.9 across benchmarks like MMLU, TruthfulQA, and Vicuna, surpassing SFT (31.1) and PPO (33.1). Notably, FIGA outperformed PPO even when the latter was trained on four times more data, highlighting its superior sample efficiency.

Comparative win rates of FIGA against various baselines. FIGA consistently shows high win rates and competitive performance against even proprietary models like ChatGPT.

Error Analysis

Human evaluation revealed that FIGA significantly improves response quality by reducing specific failure modes. Inaccuracies were reduced from 20% to 5%, and instances of "lack of detail" dropped from 35% to 10%. The "revision" step proved crucial; training on revisions yielded a 3.8-point improvement over training on standard reference responses.

Discussion and Limitations

FIGA demonstrates that contrasting "before and after" versions of a model's own output is a powerful signal for alignment. By reducing distribution shift—training on the model's own linguistic style—the resulting model is more robust during inference.

However, the method currently depends on high-quality teacher models for revisions and reliable reward models for filtering. Future work will explore scaling FIGA to larger model sizes (70B+) and investigating the sensitivity of the weighting parameters across different domains.

Conclusion

FIGA offers a compelling alternative to traditional alignment methods. By moving beyond simple imitation to explicit, fine-grained error correction, it achieves state-of-the-art performance with the simplicity and stability of supervised fine-tuning.

Summary updated

BibTeX
@inproceedings{guo2024beyond,
  author = {Geyang Guo and Ranchi Zhao and Tianyi Tang and Wayne Xin Zhao and Ji-Rong Wen},
  title = {{Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=LNLjU5C5dK},
}
Figure