When training a deep neural network, we typically focus on the final destination: the model's accuracy or loss at the very last epoch. However, the trajectory a model takes to reach that destination contains a wealth of information about the underlying dataset.
By observing how a model's predictions for individual data points evolve during the training process—a concept known as training dynamics—we can build a diagnostic map of our data. This paradigm, often called dataset cartography, allows us to identify which samples drive generalization, which are redundant, and which are likely corrupted or mislabeled.
The Coordinate Space: Confidence and Variability
To map a dataset, we need a way to quantify how a model "wrestles" with each example over time. Swabha Swayamdipta et al. established a framework for this in their seminal paper, Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics. They proposed tracking two simple statistics for each training instance across multiple training epochs:
- Confidence: The mean probability assigned by the model to the correct ground-truth label across training epochs.
- Variability: The standard deviation of that probability across the same epochs.
When we plot every training instance along these two dimensions, the dataset naturally segregates into three distinct, intuitive regions:
Confidence
^
| [ Easy-to-Learn ]
| (High Conf, Low Var)
| [ Ambiguous ]
| (Mid Conf, High Var)
| [ Hard-to-Learn ]
| (Low Conf, Low Var)
+--------------------------------------------> Variability
Why This Separation Happens
- Easy-to-learn instances (High Confidence, Low Variability): These are the straightforward, prototypical examples of a class. Early in training, the model learns to classify them correctly and its confidence remains high and stable. While necessary for establishing baseline representations, the authors found these samples are largely redundant for advanced learning; they can often be pruned with minimal impact on final performance.
- Ambiguous instances (High Variability): These are the most informative samples in the dataset. Because they exhibit high variability, the model's predictions fluctuate wildly from epoch to epoch. This fluctuation occurs because these instances sit directly on or near the shifting decision boundary. Small updates to the model's weights push these samples back and forth between correct and incorrect classifications.
- Hard-to-learn instances (Low Confidence, Low Variability): These are samples the model consistently fails to grasp. The low variability indicates that the model is not undecided; rather, it is consistently confident that the ground-truth label is wrong. This pattern is a strong indicator of an annotation error or a severe outlier.
Alternative Windows into the Learning Journey
While confidence and variability are powerful coordinates, training dynamics can be measured through other behavioral proxies that capture similar underlying phenomena.
Example Forgetting
Instead of tracking continuous probabilities, we can track discrete transitions. Mariya Toneva et al. introduced the concept of "forgetting events" in An Empirical Study of Example Forgetting during Deep Neural Network Learning. A forgetting event occurs when a model correctly classifies a training sample at epoch $t$, but misclassifies it at epoch $t+1$.
The intuition aligns perfectly with dataset cartography: "unforgettable" examples are learned early and never missed again (easy-to-learn), while highly volatile examples undergo frequent forgetting events because they reside near complex decision boundaries. This approach has proven highly transferable; for instance, Ryan Benkert et al. utilized forgetting events in Example Forgetting: A Novel Approach to Explain and Interpret Deep Neural Networks in Seismic Interpretation to successfully isolate geologically complex regions in seismic data.
Prediction Margins
Another granular metric is the margin—the difference between the model's confidence in the target class versus its next closest competitor. In DynClean: Training Dynamics-based Label Cleaning for Distantly-Supervised Named Entity Recognition, Qi Zhang et al. demonstrated that tracking how this margin fluctuates during the early stages of training provides a highly sensitive indicator for separating true linguistic signals from distant-supervision noise. Similarly, Haizhong Zheng et al. leveraged margin-based trajectories over time in ELFS: Label-Free Coreset Selection with Proxy Training Dynamics to categorize data difficulty without relying on human annotations.
Putting the Map to Work
Once we understand how to read a dataset's map, we can move away from treating data as a monolithic block and instead manipulate specific regions to improve data efficiency, robustness, and quality.
1. Automated Data Cleaning
Because hard-to-learn samples represent instances where the model's internal logic strongly contradicts the provided label, they serve as an excellent filter for finding errors. Jan-Christoph Klie et al. performed a large-scale benchmark in Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future and confirmed that tracking these training dynamics consistently surfaces genuine annotation errors more effectively than static, post-hoc loss heuristics. In a standalone analysis of data corruption, Nabeel Seedat showed in Dissecting Sample Hardness: A Fine-Grained Analysis of Hardness Characterization Methods for Data-Centric AI that tracking confidence and learning dynamics over time provides a robust, general-purpose mechanism for identifying various types of data corruption and distribution shifts.
2. Informed Data Augmentation
Standard data augmentation techniques like MixUp combine random pairs of samples, which can inadvertently create nonsensical training data. Training dynamics allow for a more surgical approach. In A Data Cartography based MixUp for Pre-trained Language Models, Cornelia Caragea et al. introduced TD-MixUp, a strategy that explicitly pairs stable, "easy-to-learn" anchor samples with highly informative "ambiguous" samples. This targeted interpolation anchors the model's representations while smoothing out complex decision boundaries, leading to superior model calibration and generalization. A similar principle was applied by Shangeth Rajaa et al. in Improving Spoken Language Identification with Map-Mix to resolve ambiguities between highly similar spoken dialects.
3. Smarter Active Learning
Active learning aims to minimize labeling costs by querying a human to label only the most informative data points. However, traditional active learning algorithms often fail because they accidentally target unlearnable outliers or corrupted data that maximize model uncertainty.
Ranjay Krishna et al. diagnosed this failure mode in Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering, showing how dataset cartography can isolate these "collective outliers" before they waste labeling budgets. By filtering out the unlearnable region and focusing queries strictly on the ambiguous region—where the model is uncertain but capable of learning—frameworks like the one proposed by Mike Zhang et al. in Cartography Active Learning dramatically reduce the volume of data required to achieve peak performance.
Key Foundational Work
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics (2020) — contributed 2 concept elements to this direction.
An Empirical Study of Example Forgetting during Deep Neural Network Learning (2019) — contributed 1 concept elements to this direction.
Confidence-Aware Learning for Deep Neural Networks (2020) — contributed 1 concept elements to this direction.