What's New in My Data? Novelty Exploration via Contrastive Generation

Masaru Isonuma, Ivan Titov

TL;DR

Fine-tuning datasets are often massive, confidential, or too messy to inspect directly, making it difficult to know exactly what new behaviors or domains a model has learned. This paper introduces Contrastive Generative Exploration (CGE), a technique that uncovers hidden domains—like new languages or toxic content—by contrasting a fine-tuned model's outputs with its pre-trained base model. By iteratively updating the base model, CGE can generate a diverse portfolio of the dataset's novelties without ever requiring direct access to the underlying training data.

The Challenge of Black-Box Fine-Tuning

When developers fine-tune Large Language Models (LLMs) on domain-specific data (like medical records, customer service logs, or internal codebases), the models acquire new capabilities. However, they might also pick up undesirable traits, like toxic language or biased behaviors.

Ideally, developers would just inspect the fine-tuning dataset to see what's in it. In reality, these datasets are often overwhelmingly large, noisy, or strictly confidential due to privacy regulations. If a third-party contractor fine-tunes a model for you, you might only receive the final weights, not the data. Existing Out-of-Distribution (OOD) detection methods require direct access to the training data to flag novelties. This paper asks: If you only have the pre-trained base model and the fine-tuned model, can you figure out what novel information the fine-tuning dataset contained just by analyzing the models' outputs?

Uncovering Novelty with Contrastive Generative Exploration (CGE)

The authors introduce the task of novelty discovery through generation. To solve it, they propose Contrastive Generative Exploration (CGE).

Illustration of Static and Iterative CGE
  • Static CGE: This approach uses contrastive decoding, a technique usually used to suppress bad model behaviors. Here, it's used for exploration. CGE generates text token-by-token by calculating a score based on the difference in log probabilities between the fine-tuned model and the pre-trained model. It essentially plays a game of "tug-of-war," rewarding tokens that the fine-tuned model strongly prefers but the pre-trained model doesn't recognize. This forces the model to output text representative of the new data it saw during fine-tuning.
  • Iterative CGE: A major flaw with Static CGE is mode collapse—if the novel data contains Spanish and Python, the model might just generate Spanish over and over. To fix this, Iterative CGE takes the newly generated novel text and fine-tunes the pre-trained model on it. In the next generation step, the pre-trained model now "knows" Spanish, forcing the contrastive decoder to look for a different novelty (like Python).
  • Dataset Distillation Connection: The authors show mathematically that CGE acts as a computationally cheap, first-order approximation of dataset distillation, searching for text whose gradients match the parameter changes caused by the original fine-tuning.

Experimental Setup

Because the authors needed ground truth to evaluate their method, they constructed controlled fine-tuning datasets where the vast majority of data was "in-distribution" (matching the pre-training data), but a tiny fraction (e.g., 1%) was "novel." * Pre-training Data: RedPajama (for OpenLLaMA-3B) and RefinedWeb (for Falcon-RW-1B). * Novel Fine-tuning Data: The 1% novel injections consisted of 10 non-English languages (from Wikipedia), 10 programming languages (from GitHub), or toxic text (from the ToxiGen dataset). * Evaluation: They used LLaMA 3 (70B) as an automated judge to classify the generated texts and measure how many novel domains were successfully uncovered.

Key Insights

  • Flipping Contrastive Decoding: Contrastive decoding is typically used defensively (e.g., contrasting a standard model with a "toxic" model to generate safe text). Using it offensively to map the delta between two model checkpoints is a clever way to audit black-box datasets.
  • The Iterative Diversity Hack: Updating the base model with its own generated outputs is a brilliant way to "cross off" discovered domains. It forces the search algorithm out of local optima and into new latent spaces, ensuring a wider variety of novelties are found.
  • Robustness to Privacy: Differential Privacy (DP) techniques are explicitly designed to prevent models from memorizing specific training examples. Surprisingly, CGE still works on DP-fine-tuned models. It proves that CGE isn't just extracting memorized strings; it's successfully identifying generalized novel domains.

Results: The Detection vs. Coverage Trade-off

The authors measured two main metrics: Detection Rate (what percentage of the generated text actually belongs to a novel domain) and Coverage Rate (how many of the 10 hidden sub-domains, like the 10 different languages, were found).

Detection and Coverage Rates for CGE
  • Random Sampling Fails: Simply asking the fine-tuned model to generate text rarely reveals novelties. For Falcon-RW on non-English text, random sampling had a detection rate of just 1%.
  • Static CGE is highly accurate but narrow: Static CGE achieved massive detection rates (e.g., 99% for OpenLLaMA on non-English text), but coverage was mediocre (~55%), meaning it missed almost half the languages.
  • Iterative CGE trades accuracy for diversity: By updating the base model, Iterative CGE boosted coverage significantly (up to 97% for OpenLLaMA on source code). However, the detection rate dropped (down to 28%), meaning the model generated more "normal" text alongside the novelties.
  • DP Robustness: When fine-tuning with DP-Adam at a high noise multiplier (0.8), the coverage rate only dropped slightly, remaining above 60% for both models.

Limitations and Future Work

The most prominent limitation is the strict trade-off between the quantity of novelties generated (detection rate) and their diversity (coverage rate). With Iterative CGE, a human analyst would have to sift through a fair amount of standard, in-distribution text to find all the novel domains.

For future work, the authors suggest developing algorithms that can resolve this trade-off more effectively. They also plan to test CGE on actual, messy real-world datasets rather than synthetically mixed benchmarks to prove its viability in industry settings.

Summary updated

Abstract

Fine-tuning is widely used to adapt language models for specific goals, often leveraging real-world data such as patient records, customer-service interactions, or web content in languages not covered in pre-training. These datasets are typically massive, noisy, and often confidential, making their direct inspection challenging. However, understanding them is essential for guiding model deployment and informing decisions about data cleaning or suppressing any harmful behaviors learned during fine-tuning. In this study, we introduce the task of novelty discovery through generation, which aims to identify novel domains of a fine-tuning dataset by generating examples that illustrate these properties. Our approach - Contrastive Generative Exploration (CGE) - assumes no direct access to the data but instead relies on a pre-trained model and the same model after fine-tuning. By contrasting the predictions of these two models, CGE can generate examples that highlight novel domains of the fine-tuning data. However, this simple approach may produce examples that are too similar to one another, failing to capture the full range of novel domains present in the dataset. We address this by introducing an iterative version of CGE, where the previously generated examples are used to update the pre-trained model, and this updated model is then contrasted with the fully fine-tuned model to generate the next example, promoting diversity in the generated outputs. Our experiments demonstrate the effectiveness of CGE in detecting novel domains, such as toxic language, as well as new natural and programming languages. Furthermore, we show that CGE remains effective even when models are fine-tuned using differential privacy techniques.
BibTeX
@inproceedings{isonuma2025what,
  author = {Masaru Isonuma and Ivan Titov},
  title = {{What's New in My Data? Novelty Exploration via Contrastive Generation}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=IZDiRbVSVN},
}
Figure