Abstract
Large Language Model (LLM) alignment typically relies on Supervised Fine-Tuning (SFT) or Reinforcement Learning from Human Feedback (RLHF). While SFT suffers from distribution shift and a lack of discriminative signal, RLHF is often unstable and computationally expensive. We present FIGA (Fine-grained Quality Signals for Alignment), a framework that bridges this gap by training models to contrast their own subpar outputs with revised, high-quality versions. By leveraging token-level edit distances, FIGA explicitly penalizes errors and reinforces corrections. Our experiments show that FIGA consistently outperforms SFT and RLHF baselines across diverse benchmarks, providing a more stable and efficient path to alignment.
Introduction
Aligning LLMs with human values—helpfulness, honesty, and harmlessness—is a critical challenge. Current paradigms like SFT act as imitation learning, where the model copies "gold" responses without understanding the specific attributes that make them superior. Conversely, RLHF explores the model's output space but is notoriously difficult to tune and prone to reward collapse.
The authors identify a "missing middle": a method that provides the stability of supervised learning while incorporating the discriminative power of RL. FIGA achieves this by showing the model exactly which tokens in its own generated text require correction.
The FIGA Framework
The FIGA framework consists of two primary stages: dataset curation and fine-grained tuning.
SubPar Alignment (SPA) Dataset Construction
Rather than relying solely on external expert data, FIGA curates the SPA Dataset by identifying and correcting the model's own failures.
- Rollout: The initial model generates responses to a set of instructions.
- Filtering: A reward model identifies "subpar" responses—those significantly worse than a reference.
- Revision: A teacher model (e.g., ChatGPT) performs minimal revisions on these subpar outputs, transforming them into high-quality responses while maintaining the original style.
Fine-grained Quality-Aware Tuning
Alignment is treated as a correction task. Using Levenshtein distance, the framework categorizes tokens in the revised response:
- Added/Substituted tokens: Identified as "good actions" to be reinforced.
- Deleted/Substituted tokens: Identified as "bad actions" from the original output to be penalized.
- Unchanged tokens: Neutral tokens that are largely ignored during the specific alignment loss calculation.
This weighted loss function mimics the "advantage function" in RL but operates within a stable supervised framework.
Experimental Results
The authors evaluated FIGA against several baselines, including SFT and PPO, using the Alpaca-7B model as a starting point.
Performance Benchmarks
FIGA achieved an aggregate score of 34.9 across benchmarks like MMLU, TruthfulQA, and Vicuna, surpassing SFT (31.1) and PPO (33.1). Notably, FIGA outperformed PPO even when the latter was trained on four times more data, highlighting its superior sample efficiency.
Error Analysis
Human evaluation revealed that FIGA significantly improves response quality by reducing specific failure modes. Inaccuracies were reduced from 20% to 5%, and instances of "lack of detail" dropped from 35% to 10%. The "revision" step proved crucial; training on revisions yielded a 3.8-point improvement over training on standard reference responses.
Discussion and Limitations
FIGA demonstrates that contrasting "before and after" versions of a model's own output is a powerful signal for alignment. By reducing distribution shift—training on the model's own linguistic style—the resulting model is more robust during inference.
However, the method currently depends on high-quality teacher models for revisions and reliable reward models for filtering. Future work will explore scaling FIGA to larger model sizes (70B+) and investigating the sensitivity of the weighting parameters across different domains.
Conclusion
FIGA offers a compelling alternative to traditional alignment methods. By moving beyond simple imitation to explicit, fine-grained error correction, it achieves state-of-the-art performance with the simplicity and stability of supervised fine-tuning.
Summary updated