UNIWIZ: A Unified Large Language Model Orchestrated Wizard for Safe Knowledge Grounded Conversations

Souvik Das, Rohini K. SrihariACL 2024 · DOI

Introduction

Large Language Models (LLMs) often struggle to balance safety constraints with factual accuracy, particularly in multi-turn adversarial dialogues. This paper introduces UNIWIZ, a 2-step data orchestration framework that combines iterative safety reasoning with factual knowledge grounding. By training a 7B parameter model on the resulting high-quality synthetic dataset, the authors demonstrate that smaller models can outperform significantly larger counterparts in both safety and truthfulness.

The Safety-Knowledge Trade-off

Modern conversational AI faces a significant hurdle: excessive safety alignment. When models are tuned aggressively to avoid harm, they often default to "hard refusals" or, more insidiously, begin to hallucinate when a user persistently probes sensitive topics. Conversely, models optimized purely for knowledge retrieval often lack the ethical reasoning required to steer users away from harmful intentions.

Conversational failure modes where models provide unsafe advice (a) or struggle with sensitive topics (b).

As illustrated above, current models can fail to maintain a safe stance over multiple turns, sometimes even validating harmful user intentions after initial resistance. This "safety-knowledge" trade-off suggests that current datasets focus too narrowly on one pillar at the expense of the other.

The UNIWIZ Framework

To bridge this gap, the UNIWIZ framework employs a 2-step orchestration process to generate high-quality synthetic training data using a powerful teacher model (Claude-2.1).

The UNIWIZ dataset creation pipeline, detailing safety priming, knowledge retrieval, and grounded conversation extension.

1. Safety Priming

Instead of simple refusals (e.g., "I cannot help with that"), UNIWIZ utilizes Safety Priming. The framework simulates "self-chat" sessions where an AI agent is tasked with providing persistent, reasoned counter-arguments to a user with ill intentions (sourced from ProsocialDialog and Atomic10x). This teaches the model to engage in iterative reasoning, maintaining its safety posture even under pressure.

2. Knowledge Injection

To prevent the conversation from drifting into hallucinations, the framework identifies key entities within the safety-primed dialogue and retrieves factual snippets from a curated BM25 index of the Wizard of Wikipedia. The conversation is then extended, ensuring that later turns are grounded in verifiable facts rather than model-generated fabrications. This grounding acts as a "safety buffer," preventing the model from running out of logical arguments.

Training and Dataset

The resulting UNIWIZ Dataset consists of 17,638 quality-controlled, multi-turn conversations and 10,000 augmented preference pairs. The training pipeline involves:

  • Supervised Fine-Tuning (SFT): Mistral 7B was fine-tuned using QLoRA on the UNIWIZ dataset.
  • Preference Alignment: The model was further refined using Direct Preference Optimization (DPO) on a dataset where an LLM ranked multiple generated responses to identify the safest and most helpful options.

Key Results

The final model, UniWiz-7B-v0.2 (DPO), demonstrated superior performance across several dimensions:

  • Benchmark Superiority: It achieved an average score of 64.56 on major benchmarks (ARC, MMLU, Hellaswag), surpassing both the base Mistral 7B (60.87) and Zephyr-7b-beta (61.95).
  • Enhanced Factuality: On TruthfulQA, the model scored 59.91, a massive jump from the base Mistral's 42.15, validating the effectiveness of the knowledge injection step.
  • Safety Preference: In head-to-head evaluations judged by Claude-2.1, UniWiz's responses were preferred 57% of the time over models trained on traditional safety datasets like ProsocialDialog and Faithdial.

Conclusion

UNIWIZ proves that high-quality, orchestrated synthetic data can effectively resolve the safety-knowledge trade-off. By prioritizing reasoning over refusal and grounding conversations in factual knowledge, the framework enables smaller models to achieve state-of-the-art performance in safe, informative dialogue. Future work will look toward expanding the knowledge scope to real-time sources like live web searches.

Summary updated

BibTeX
@inproceedings{das2024uniwiz,
  author = {Souvik Das and Rohini K. Srihari},
  title = {{UNIWIZ: A Unified Large Language Model Orchestrated Wizard for Safe Knowledge Grounded Conversations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.102},
}
Figure