Learning to Paraphrase for Alignment with LLM Preference

Junbo Fu, Guoshuai Zhao, Yimin Deng, Yunqi Mi, Xueming QianEMNLP 2024 · DOI

Introduction: The Challenge of Paraphrase Divergence

Large Language Models (LLMs) have demonstrated remarkable capabilities in reasoning and knowledge retrieval. However, they remain surprisingly fragile to subtle changes in input phrasing. A model might correctly answer a question in one format but fail entirely if a few words are swapped or the sentence structure is altered, even when the underlying meaning remains identical. This phenomenon, known as paraphrase divergence, suggests that LLMs do not just learn raw knowledge; they learn specific "expression patterns" associated with that knowledge during their pre-training phase.

Illustrating paraphrase divergence: specific linguistic shifts like spelling corrections or reordering can flip an LLM's response from incorrect to correct.

Traditionally, addressing this fragility required "white-box" tuning—retraining the entire model to be more robust. As models grow to hundreds of billions of parameters and are increasingly accessed via proprietary APIs (e.g., GPT-4), retraining is often computationally or practically impossible. This paper introduces PEARL (Preference-drivEn pAraphRase Learning), a black-box alignment strategy that improves model performance without modifying the target LLM's internal weights.

The PEARL Framework: Alignment via Rewriting

PEARL shifts the focus from changing the model to changing the input. It utilizes a small auxiliary model—the PEARL Generator (based on FLAN-T5-L, 770M parameters)—to act as a specialized translator. Its role is to rewrite "non-preferred" user queries into the specific expressions that the target LLM is most likely to answer correctly.

Automated Training Pipeline

To train the PEARL Generator, the researchers developed an automated pipeline to identify the target LLM's internal preferences:

  1. Generation: An open-source text editor (COEDIT-XXL) generates diverse paraphrases of existing questions using specific "type instructions," such as changing the word order or fixing spelling.
  2. Filtering: The BLUERT metric is employed to ensure that these paraphrases maintain the original semantic meaning of the question.
  3. Labeling: These paraphrases are fed to the target LLM. If the LLM answers a paraphrase correctly but failed the original question, that paraphrase is labeled as "preferred."
  4. Seq2Seq Training: The PEARL Generator is then fine-tuned to map original "failed" questions to these successful, preferred paraphrases.
The PEARL training pipeline: an automated process for generating, filtering, and labeling paraphrases to train the auxiliary generator.

Experimental Evaluation

The researchers validated PEARL across six datasets spanning three reasoning tasks: Open-Domain QA (WebQSP, CWQ, CompQ), Commonsense Reasoning (CSQA, SiQA), and Math Word Problems (SVAMP). While MPT-instruct-7B served as the primary target LLM, the study also tested transferability across other models including TinyLlama-1.3B, RedPajama-3B, BLOOMz-7B, and MPT-30B.

Key Results

  • Performance Gains: PEARL achieved an average improvement of 5.23% to 5.32% across all datasets when integrated with other black-box methods like In-Context Learning.
  • Efficiency vs. White-Box Tuning: In several benchmarks (CompQ, CWQ, and SVAMP), the PEARL black-box approach outperformed LoRA (Low-Rank Adaptation), a popular white-box fine-tuning method, despite PEARL utilizing significantly fewer trainable parameters.
  • Impact on Smaller Models: The most significant boost was observed on BLOOMz-7B, where PEARL increased performance by 12.12%.

Discussion and Insights

Analysis of the "preferred" paraphrases revealed that LLMs generally favor questions that emphasize specific requirements, place essential information at the beginning of the sentence, and maintain meticulous spelling.

Furthermore, PEARL demonstrates strong composability. When used to rewrite questions before they are processed by prompt-retrieval systems (like UPRISE), the performance gains are synergistic. Interestingly, a PEARL Generator trained for one specific model often improves performance for others, suggesting that different LLMs share certain "universal preferences" for clarity and syntax.

Conclusion and Future Work

PEARL offers a lightweight, effective method for aligning human questions with the internal preferences of LLMs. While the current generator is limited by its own capacity to produce diverse and semantically accurate paraphrases, the results prove that black-box input optimization is a viable alternative to costly model retraining. Future research will focus on utilizing more powerful paraphrase models and strategies to better exploit the "potential accuracy" of multiple generated candidates.

Summary updated

BibTeX
@inproceedings{fu2024learning,
  author = {Junbo Fu and Guoshuai Zhao and Yimin Deng and Yunqi Mi and Xueming Qian},
  title = {{Learning to Paraphrase for Alignment with LLM Preference}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.134},
}
Figure