Ensuring Safe and High-Quality Outputs: A Guideline Library Approach for Language Models

Yi Luo, Zhenghao Lin, Yuhao Zhang, Jiashuo Sun, Chen Lin, Chengjin Xu, Xiangdong Su, Yelong Shen, Jian Guo, Yeyun Gong

Abstract

Large Language Model (LLM) alignment typically relies on broad, manually crafted safety principles, which often lack the specificity required for complex real-world prompts. This paper introduces Guide-Align, a framework that utilizes a "Guideline Library" approach to provide fine-grained instructions for diverse inputs. By leveraging a safety-trained teacher model to generate specific guidelines and a lightweight retrieval model to inject them at inference time, Guide-Align enables smaller models to achieve alignment performance comparable to or exceeding that of much larger models like GPT-4.

Introduction

Aligning LLMs with human values—specifically the criteria of being Helpful, Honest, and Harmless (HHH)—is a critical challenge in AI deployment. Current methods like Supervised Fine-Tuning (SFT) or Reinforcement Learning from Human Feedback (RLHF) often depend on generic rules. However, these rules frequently suffer from limited matching precision and insufficient risk perception, failing to address nuanced prompts or subtle "jailbreak" attempts. Guide-Align addresses these gaps by moving toward a dynamic, retrieval-based guideline system.

The Guide-Align Framework

The Guide-Align framework consists of three primary components: library construction, retrieval training, and optional model distillation.

  1. Guideline Library Construction: Using a teacher model (GPT-3.5-turbo), the authors analyzed a training set of 153k questions. For each input, the teacher performed safety detection. Risky inputs triggered the generation of safety guidelines, while safe inputs generated quality guidelines. This process created a library of approximately 767k guidelines, which was refined to 33k unique entries.
  2. Retrieval Model Training: A BERT-based model was trained to map user inputs to the most relevant guidelines. At inference time, the top $N$ guidelines are retrieved and prepended to the user's prompt.
  3. Fine-tuning (Optional): To internalize these guidelines into a standalone model, the authors generated a high-quality alignment dataset (28k instructions) using the Guide-Align process. This data was used to fine-tune LLaMA-2-13B, resulting in the Labrador model.
Overview of the Guide-Align framework, including library construction, retrieval, and inference.

Experimental Setup

The framework was evaluated using several benchmarks:

  • Training Data: Built using Self-instruct (52k questions) and a custom set of 100k security-related questions covering various insecure content and command attacks.
  • Evaluation Benchmarks: Included Do_Not_Answer (939 prompts for refusal), HHH_Alignment (evaluating helpfulness, honesty, and harmlessness), and the Vicuna_Benchmark (80 diverse queries for general quality).

Results and Discussion

Guide-Align demonstrates significant improvements in both safety and general response quality. On the Do_Not_Answer benchmark, Labrador achieved a harmlessness score of 98.1%, outperforming GPT-4's 97.6%. In head-to-head comparisons on HHH_Alignment, Labrador secured a 54.9% Net Win Rate over GPT-3.5-turbo.

The "plug-and-play" nature of the retrieval component also allowed base models like Vicuna to improve their harmlessness by 3.5%. Furthermore, the retrieval model showed a high risk-perception accuracy of 94.7%, significantly higher than the 39% accuracy of a zero-shot Vicuna model.

Comparison of risk occurrences across different models and risk areas with and without guidelines.

Conclusion

Guide-Align proves that fine-grained, context-specific instructions are more effective for alignment than broad mandates. By using retrieval as a "safety buffer," the system provides the risk perception of large models with minimal latency. Future work will focus on expanding the library to support multiple languages and improving factual grounding to mitigate misinformation harms.

Summary updated

BibTeX
@inproceedings{luo2024ensuring,
  author = {Yi Luo and Zhenghao Lin and Yuhao Zhang and Jiashuo Sun and Chen Lin and Chengjin Xu and Xiangdong Su and Yelong Shen and Jian Guo and Yeyun Gong},
  title = {{Ensuring Safe and High-Quality Outputs: A Guideline Library Approach for Language Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.65},
}
Figure