Unraveling the Influence of Training Data and Internal Structures in Large Language Models for Enhanced Explainability (Student Abstract)

Lingfang Li, Procheta SenAAAI 2025 · DOI

TL;DR

This paper proposes a more efficient way to understand which specific data points in a massive training set most influence a Large Language Model's (LLM) predictions. By using techniques like FAISS for faster data point identification and then analyzing the model's internal workings, the authors aim to pinpoint how training data and model architecture contribute to LLM behavior, particularly in addressing issues like hallucinations and bias.

What Problem Does This Paper Address?

Large Language Models (LLMs) are becoming incredibly powerful and are being applied in critical domains like medicine and finance. However, their "black box" nature makes it difficult to trust their outputs, especially when they produce incorrect information (hallucinations) or exhibit biases. Understanding why an LLM makes a certain prediction is crucial for debugging, improving reliability, and fostering user trust. Existing methods for explaining LLMs often focus on token importance or attention mechanisms, but there's a significant gap in understanding how the vast training data and the model's internal structures interact to produce these behaviors. This paper specifically targets this gap, aiming to provide deep learning practitioners with actionable insights into LLM decision-making.

Methods

The authors propose a multi-pronged approach to tackle their research questions:

  • Influence Functions for Data Point Importance (RQ1 & RQ2): They leverage influence functions, a technique that estimates the impact of individual training data points on a model's predictions.
  • Efficient Data Subsetting with FAISS (RQ1): To overcome the computational burden of traditional influence function calculations on massive datasets, they propose using Facebook AI Similarity Search (FAISS). FAISS efficiently identifies training samples that are semantically similar to test samples, allowing influence functions to be computed on a much smaller, relevant subset of the data.
  • Approximating LLMs with Smaller Models: For further computational efficiency, they suggest training smaller LLMs on the identified influential subsets. These smaller models are then used to test if they can approximate the behavior of larger LLMs, allowing for more feasible analysis of data point influence across different training stages.
  • Analyzing Performance Drop (RQ2): To understand how training data impacts performance across different LLM usage modes (pre-training, fine-tuning, in-context learning), they plan to measure performance drops after removing the most influential training samples.
  • Mechanistic Interpretability (RQ3): For understanding internal structures, they will employ mechanistic interpretability techniques, specifically focusing on "edge attribution," to identify specific neurons, layers, or attention heads responsible for particular functions and behaviors.

Datasets

The paper mentions using the "Movie Review" dataset for their experiments with GPT-2 Small. No new datasets are introduced.

Key Insights and Ideas

  • FAISS is a Game-Changer for Efficiency: The core idea is that you don't need to analyze every single data point to understand influence. FAISS can quickly find the most relevant ones, making influence function analysis feasible for large-scale LLMs.
  • Semantically Relevant Data is Key: The preliminary results suggest that LLMs are heavily influenced by training data points that are semantically similar to the test data. This is a crucial insight for understanding how LLMs generalize and where issues like bias might creep in.
  • Targeted Analysis for Targeted Solutions: By identifying influential data points and internal components, the goal is to enable the design of LLMs that are less prone to hallucinations and biases for specific tasks.

Key Results

The paper presents preliminary results from an experiment using GPT-2 Small on the Movie Review dataset, comparing different methods for identifying influential training points:

  • Random Removal: Removing 1% of randomly selected training data points increased accuracy by 2%, suggesting the removal of noisy or uninformative samples.
  • TF-IDF Removal: Removing samples identified by TF-IDF (word-level similarity) had no measurable effect on accuracy.
  • FAISS Removal: Removing samples identified by FAISS (semantic similarity) led to a 2% decrease in accuracy. This highlights the importance of semantically relevant training data and FAISS's effectiveness in identifying it.

Limitations and Future Work

  • Limitations: The current study uses a smaller model (GPT-2 Small) and a specific dataset (Movie Review). The generalizability of these findings to larger, more complex LLMs and diverse datasets remains to be seen. The computational cost, while reduced by FAISS, is still a factor.
  • Future Work: The authors plan to extend their methods to more complex models and larger datasets. They also aim to explore mechanistic interpretability further to identify key internal components responsible for tasks like machine translation and emotion recognition. The ultimate goal is to use these insights to build more interpretable and reliable LLMs.

Summary updated

BibTeX
@inproceedings{li2025unraveling,
  author = {Lingfang Li and Procheta Sen},
  title = {{Unraveling the Influence of Training Data and Internal Structures in Large Language Models for Enhanced Explainability (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i28.35268},
}
Figure