Abstract
Fine-tuning is widely used to adapt language models for specific goals, often leveraging real-world data such as patient records, customer-service interactions, or web content in languages not covered in pre-training. These datasets are typically massive, noisy, and often confidential, making their direct inspection challenging. However, understanding them is essential for guiding model deployment and informing decisions about data cleaning or suppressing any harmful behaviors learned during fine-tuning. In this study, we introduce the task of novelty discovery through generation, which aims to identify novel properties of a fine-tuning dataset by generating examples that illustrate these properties. Our approach, Contrastive Generative Exploration (CGE), assumes no direct access to the data but instead relies on a pre-trained model and the same model after fine-tuning. By contrasting the predictions of these two models, CGE can generate examples that highlight novel characteristics of the fine-tuning data. However, this simple approach may produce examples that are too similar to one another, failing to capture the full range of novel phenomena present in the dataset. We address this by introducing an iterative version of CGE, where the previously generated examples are used to update the pre-trained model, and this updated model is then contrasted with the fully fine-tuned model to generate the next example, promoting diversity in the generated outputs. Our experiments demonstrate the effectiveness of CGE in detecting novel content, such as toxic language, as well as new natural and programming languages. Furthermore, we show that CGE remains effective even when models are fine-tuned using differential privacy techniques.
Introduction
Fine-tuning Large Language Models (LLMs) on domain-specific data is standard practice for creating specialized assistants. However, the underlying datasets—often containing millions of documents or sensitive information—are frequently treated as "black boxes." This lack of transparency poses significant risks: a model might inadvertently learn toxic behaviors, memorize private data, or adopt biases present in the fine-tuning set.
Traditional Out-of-Distribution (OOD) detection methods typically require direct access to the training data or a predefined set of labels to search for. In contrast, we propose Contrastive Generative Exploration (CGE), a "discovery through generation" framework. CGE allows developers to audit what a model has learned by letting the model itself generate representative examples of the novel content it encountered during fine-tuning, without ever needing to inspect the raw training data.
Methodology: Contrastive Generative Exploration
The core intuition behind CGE is that the difference in predictions between a pre-trained model ($\theta_{pt}$) and its fine-tuned counterpart ($\theta_{ft}$) isolates the "new" knowledge acquired during the second training phase.
Static CGE
Static CGE calculates a contrastive score by subtracting the log probabilities of the pre-trained model from those of the fine-tuned model. This score acts as a "tug-of-war": the fine-tuned model pushes toward tokens seen in its recent training, while the pre-trained model pulls back toward its original distribution. To ensure the generated text remains coherent, an Adaptive Plausibility Constraint is applied, requiring that generated tokens maintain a minimum probability threshold in the fine-tuned model.
Iterative CGE
A limitation of the static approach is "mode collapse," where the model repeatedly generates the most prominent novelty (e.g., only Japanese text if the dataset contained multiple new languages). To address this, we introduce an iterative version. After each generation, the pre-trained model is briefly updated on the newly generated example. This process effectively "masks" that specific novelty for the next generation, forcing the contrastive score to highlight different, previously uncovered features of the fine-tuning data.
Experimental Setup
We evaluated CGE using OpenLLaMA-3B and Falcon-RW-1B. Synthetic fine-tuning datasets were constructed with 90% "in-distribution" English text and 10% "novel" data across three categories:
- Non-English Text: Wikipedia articles in 10 diverse languages.
- Toxic Text: Machine-generated hate speech (ToxiGen).
- Source Code: 10 different programming languages from GitHub.
Results and Key Insights
CGE demonstrates exceptional performance in both identifying and exploring novel data distributions.
Superior Detection and Coverage
In extraction tasks, CGE achieved AUROC scores of 0.98 to 0.99 for non-English text, significantly outperforming standard baselines like Maximum Softmax Probability (MSP). In generative tasks, Iterative CGE proved highly effective at uncovering the full breadth of the data. For instance, it reached a 90% coverage rate for source code, successfully identifying nearly all programming languages hidden in the fine-tuning set.
Privacy and Robustness
Notably, CGE remains effective even when models are fine-tuned using Differential Privacy (DP-Adam). While DP aims to prevent the memorization of specific training examples, CGE successfully extracts the general distributional shifts (e.g., new domains or styles) the model has learned. Furthermore, the method is robust enough to surface novelties even when they constitute as little as 0.01% of the fine-tuning data.
Discussion and Limitations
While CGE is a powerful tool for model auditing, it faces the "Empty Well" problem: once all novel domains are exhausted in the iterative process, the model reverts to generating standard in-distribution text. This requires users to monitor outputs for a drop in novelty. Additionally, while effective on distinct categories like languages or code, future work is needed to assess CGE's performance on more subtle shifts, such as changes in writing style or factual nuances.
Despite these challenges, CGE provides a vital, privacy-preserving mechanism for developers to understand and audit the "black box" of fine-tuning data, ensuring safer and more transparent model deployment.
Summary updated