CONSCENDI: A Contrastive and Scenario-Guided Distillation Approach to Guardrail Models for Virtual Assistants

Albert Yu Sun, Varun Nair, Elliot Schumacher, Anitha KannanNAACL 2024 · DOI

TL;DR

This paper introduces CONSCENDI, a novel approach to training smaller, more efficient "guardrail" models for virtual assistants. By using a large language model (LLM) like GPT-4 to generate diverse, scenario-guided, and contrastive training data, CONSCENDI enables these smaller models to achieve higher accuracy in identifying rule violations than even GPT-4 itself, while being significantly faster and cheaper to run.

What Problem Does This Paper Address?

As virtual assistants powered by large language models (LLMs) like GPT-4 become more sophisticated, ensuring they operate within specific task constraints and adhere to desired behaviors is crucial. This is often handled by "guardrail" systems that act as a safety net, verifying the assistant's output. However, traditional guardrail methods, often relying on complex LLM prompts, are difficult to engineer comprehensively and can be costly and slow. The challenge is to create guardrail models that are both accurate and efficient, capable of distinguishing subtle rule violations from acceptable behavior.

Methods

CONSCENDI tackles this by employing a two-pronged data generation strategy to train smaller LLMs (distillation).

  1. Scenario-Guided Generation: Instead of just generating random conversations, CONSCENDI first prompts an LLM to create a diverse set of "scenarios" that describe high-level ways a rule could be violated. These scenarios provide a structured way to generate a broader range of potential rule-breaking examples, including less common "tail" scenarios.
  2. Contrastive Learning: For each generated rule-breaking conversation, CONSCENDI then creates a "contrastive" version. This involves taking the same conversation up to the point of violation and replacing the violating assistant turn with a non-violating one. This forces the model to learn fine-grained distinctions between acceptable and unacceptable responses.

These generated conversations (violations, contrastive non-violations, and general non-violations) are then used to fine-tune smaller LLMs (like GPT-3 variants), making them efficient guardrail models.

Datasets

The authors created datasets for three domains: flights, restaurants, and buses, inspired by the Schema-Guided Dialogue (SGD) dataset. They generated approximately 500 violations, 500 contrastive non-violations, and 200 non-contrastive non-violations per domain. The data was split into training, in-distribution (ID) test, and out-of-distribution (OOD) test sets, where OOD examples were generated from scenarios not seen during training.

Key Insights and Ideas

  • Scenarios are Key for Diversity: Generating rule-breaking scenarios first, rather than just random conversations, leads to a more diverse and comprehensive training dataset, crucial for robust guardrails.
  • Contrastive Examples Teach Nuance: The ability to distinguish between very similar conversations (one violating a rule, one not) is critical. Contrastive examples are vital for teaching models these subtle differences, significantly improving accuracy.
  • Distillation Empowers Smaller Models: Fine-tuning smaller LLMs with this carefully curated data allows them to surpass the performance of larger, prompt-based models, including GPT-4, on the guardrail task.
  • OOD Performance is Crucial: The real test of a guardrail is its ability to handle unseen situations. CONSCENDI's approach significantly improves performance on OOD examples, demonstrating strong generalization.

Key Results

CONSCENDI, when applied to fine-tune GPT-3 models, consistently outperformed baseline approaches, including prompt-based GPT-3 and even GPT-4, on both in-distribution (ID) and out-of-distribution (OOD) scenarios.

  • Accuracy Gains: Fine-tuned GPT-3 models achieved accuracy improvements of up to 15-35% compared to baselines. For instance, the smallest GPT-3 model (Ada) often doubled its accuracy.
  • OOD Superiority: The most significant gains were observed in OOD performance, highlighting CONSCENDI's ability to generalize to unseen scenarios.
  • Efficiency: The fine-tuned models offer substantial benefits in terms of latency and cost, being up to 2-4x faster and 200x cheaper than GPT-4.
  • GPT-4's Weakness: Interestingly, GPT-4 itself struggled with contrastive examples, often misclassifying non-violations as violations, an area where CONSCENDI excelled.

Limitations and Future Work

  • API Reliance: The study relies on OpenAI's API for data generation and model fine-tuning, which can be subject to changes and replication challenges.
  • Dataset Size: While experiments were conducted across domains, only a subset of the SGD dataset was used, and only a single run per model was performed due to cost constraints.
  • Multi-Label Violations: The current approach focuses on single-label violations. Future work could explore handling scenarios with multiple rule violations.
  • Evaluation Strategies: Developing better evaluation strategies for generated conversational data is important to ensure real-world applicability.
  • Domain Expertise: The authors emphasize the need for extensive QA guided by domain experts for real-world applications.
  • External Knowledge: The current rules are designed to be verifiable within the assistant's messages, not relying on external APIs or databases, which is an area for future exploration.

Summary updated

BibTeX
@inproceedings{sun2024conscendi,
  author = {Albert Yu Sun and Varun Nair and Elliot Schumacher and Anitha Kannan},
  title = {{CONSCENDI: A Contrastive and Scenario-Guided Distillation Approach to Guardrail Models for Virtual Assistants}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.223},
}
Figure