RETAIN: Interactive Tool for Regression Testing Guided LLM Migration

Tanay Dixit, Daniel Lee, Sally Fang, Sai Sree Harsha, Anirudh Sureshan, Akash Maharaj, Yunyao Li

Introduction

The rapid evolution of the Large Language Model (LLM) landscape frequently necessitates migrating applications to newer, more cost-effective, or higher-performing models. However, this transition is often fraught with difficulty because prompts are frequently "overfitted" to the specific nuances of a source model. A prompt optimized for GPT-3.5 may yield inconsistent or buggy results when deployed on Llama-3, creating a significant regression testing challenge.

Currently, developers often resort to manual inspection of hundreds of outputs to identify behavioral shifts. While aggregate metrics like BERTScore provide a high-level view of performance, they fail to explain why a model is underperforming or how to remediate the prompt. RETAIN (REgression Testing guided LLM migrAtIoN) addresses this gap by providing an interactive environment that combines automated error discovery with specialized debugging tools.

The RETAIN System Architecture

RETAIN is designed around three functional pillars that facilitate a structured migration workflow:

  1. The Interactive Interface: A comprehensive three-panel dashboard that includes a Metric Panel for tracking aggregate performance, a Data Panel for side-by-side instance comparisons and distribution visualizations, and an Error Analysis Panel.
  2. Goal-Driven Error Discovery: Instead of simple diffing, RETAIN utilizes a high-level model (e.g., GPT-4) to analyze clusters of outputs and generate natural language descriptions of behavioral regressions, such as "Model B uses more informal language than Model A."
  3. The Selector Module: Once an error type is identified, RETAIN employs a classifier to scan the dataset and "slice" it, isolating the specific inputs that triggered the regression.
The RETAIN interactive interface, featuring metric tracking, side-by-side data comparison, and automated error discovery.

To manage the inherent non-determinism of LLMs, the tool incorporates a "Metric Tolerance" threshold. This allows developers to filter out minor score fluctuations and focus their attention on statistically significant regressions.

Methodology and Evaluation

The researchers evaluated RETAIN through a synthetic evaluation and a user study involving 12 ML engineers. The synthetic dataset consisted of 100 points with manually injected stylistic and stance-based differences to test the accuracy of the Error Discovery module. For the user study, participants performed a summarization task, migrating from gpt-3.5-turbo-16k to llama-3-8b.

One of the tool's core strengths is its ability to close the loop between discovery and testing. Users can transform a discovered error (e.g., "too wordy") into a permanent "LLM Assertion," effectively building project-specific unit tests during the migration process. This prevents "whack-a-mole" prompt engineering, where fixing one edge case inadvertently breaks others.

Key Results

The user study demonstrated that RETAIN significantly outperforms traditional manual workflows (such as using Excel or Jupyter notebooks):

  • Enhanced Detection: Users identified twice as many behavioral errors compared to the manual group.
  • Increased Iteration Speed: Participants were able to experiment with 75% more prompt versions (4.55 vs 2.6 edits) within the same 15-minute timeframe.
  • Superior Performance: RETAIN users achieved 12% higher BERTScore results than those in the manual group.
Performance comparison showing BERTScore improvement over time for users using RETAIN versus manual prompt engineering.

User sentiment was overwhelmingly positive, with 76% of participants praising the tool's ability to reinforce observations and quickly identify common error patterns across the dataset.

Conclusion and Future Work

RETAIN demonstrates that natural language is a powerful debugging tool for LLMs. By providing actionable descriptions of failures rather than just abstract scores, it enables developers to migrate models with greater confidence and efficiency. Future work will explore scaling the Error Discovery module for massive datasets and potentially automating the prompt-editing process by suggesting specific fixes based on identified regressions.

Summary updated

BibTeX
@inproceedings{dixit2024retain,
  author = {Tanay Dixit and Daniel Lee and Sally Fang and Sai Sree Harsha and Anirudh Sureshan and Akash Maharaj and Yunyao Li},
  title = {{RETAIN: Interactive Tool for Regression Testing Guided LLM Migration}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
}
Figure