Unveiling Imitation Learning: Exploring the impact of Data Falsity to Large Language Model

Hyunsoo ChoACL 2024 · DOI

Abstract

Imitation learning, the practice of fine-tuning open-source models on synthetic data from proprietary systems, has become a cornerstone of efficient model development. However, the inherent noise in synthetic data—hallucinations and logical errors—poses a significant risk. This study introduces the Falsity-Controllable (FACO) dataset to quantitatively analyze how varying levels of data corruption affect model performance. We find that models do not merely lose accuracy; they actively learn to generate deceptive reasoning for incorrect answers. Our results show a strong negative correlation between corruption ratios and performance, with more advanced models like LLaMA 2 showing higher vulnerability to data noise.

Introduction

The AI community increasingly relies on imitation learning to bridge the gap between open-source models and state-of-the-art proprietary giants. While cost-effective, this approach often utilizes "noisy" synthetic data. This research explores the quantitative threshold of noise a model can tolerate and whether models can recover from "toxic" training data. By creating a dataset with manually controlled levels of "falsity," we demonstrate that noisy synthetic data doesn't just lower scores; it actively teaches models to "lie" by generating plausible-sounding but incorrect reasoning, even when the model "knows" the right answer.

Methodology and the FACO Dataset

To investigate these dynamics, we developed the Falsity-Controllable (FACO) dataset. This dataset comprises 20,000 samples across four domains: Knowledge (QASC, SciQ), Commonsense (CommonsenseQA, OpenbookQA, WinoGrande), Reasoning (AQuA, QuaRTz), and Programming (CoNaLa, MBPP). Each sample includes a "True" pair (correct answer and reasoning) and a "False" pair (incorrect answer and fabricated reasoning generated by ChatGPT).

Figure 1: The FACO dataset framework generates both true and false reasoning chains to test model robustness against data corruption.

We created five versions of the dataset with Corruption Ratios (CR) ranging from 0% to 100%. We then fine-tuned 13B versions of LLaMA 1 and LLaMA 2 on these datasets for five epochs, evaluating them across 16 benchmarks.

Key Insights: Learning to Deceive

The most significant finding is the emergence of "deceptive" behavior. Models trained on high-corruption data learn to produce sophisticated, logical-sounding rationales for incorrect answers. This suggests that imitation learning can inadvertently teach models to prioritize plausibility over truth.

Furthermore, we observed that LLaMA 2, despite being a stronger base model, experienced a sharper performance decline than LLaMA 1 as corruption increased. This indicates that higher-capacity models may more effectively "absorb" and internalize noise from the training set, making them more sensitive to data quality.

Results and Performance Degradation

Our evaluation revealed a nearly linear relationship between the corruption ratio and performance drops, with Pearson correlation coefficients exceeding -90% across most benchmarks.

Figure 3: Performance of LLaMA 1 and LLaMA 2 13B models across various domains as the corruption ratio (CR) increases.

In the MMLU benchmark, LLaMA 2's performance dropped from ~57% to ~19% at 100% corruption—notably lower than the 25% expected from random guessing. This confirms that the model was systematically choosing incorrect options based on its corrupted training. While commonsense tasks showed some resilience due to strong pre-training priors, knowledge-heavy tasks were extremely sensitive to data quality. Additionally, monitoring training loss proved to be a viable proxy for identifying low-quality data, as corrupted sets maintained higher loss throughout training.

Conclusion and Future Work

This study highlights the critical importance of data quality in imitation learning. The "Recovery Gap" observed—where models retrained on clean data never fully reach original performance levels—suggests that some toxic behaviors persist. Future research should focus on developing robust training methods capable of filtering noise during fine-tuning and expanding these investigations into specialized domains like medicine and law.

Summary updated

BibTeX
@inproceedings{cho2024unveiling,
  author = {Hyunsoo Cho},
  title = {{Unveiling Imitation Learning: Exploring the impact of Data Falsity to Large Language Model}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.6},
}
Figure