The Dipping Phenomenon

Marco Loog, Robert P. W. Duin

Abstract

In machine learning, the monotonicity assumption suggests that increasing training data leads to improved or stable model performance. This paper challenges this notion by introducing "dipping," a phenomenon where a classifier's error rate reaches a global minimum at a finite training set size and subsequently increases as more data is added. This behavior occurs when there is a fundamental mismatch between the classifier's inductive bias and the true underlying data distribution.

Introduction

The prevailing wisdom in the machine learning community is that "more data is better." Practitioners often assume that the asymptotic limit—training on an infinite number of samples—yields the best possible version of a model. While "peaking" (a temporary increase in error) has been documented, it was generally believed that error rates eventually converge to a minimum.

Loog and Duin demonstrate that this is not always the case. They identify the "dipping phenomenon," where the best performance occurs at a small, finite sample size. This finding has significant implications for model deployment, suggesting that in certain mismatched scenarios, collecting more data can actually degrade system performance.

Experimental Setup and Model Mismatch

The authors isolate dipping using synthetic datasets designed to create "model-mismatched" scenarios. In these cases, the "true" optimal decision boundary cannot be represented by the chosen classifier's hypothesis space.

One primary example is a 1D Gaussian Mixture problem. As shown in the distribution plot below, Class 1 is represented by a single Gaussian, while Class 2 is a mixture of two Gaussians placed symmetrically around the first.

The 1D Gaussian Mixture distribution used to demonstrate model mismatch. Class 1 (dashed) is centered between the two lobes of Class 2 (solid).

When a linear classifier like the Nearest Mean Classifier (NMC) is applied to this data, it attempts to find the means of both classes. With infinite data, the estimated means for both classes converge to the same central point, leading to an error rate of 50% (random guessing). However, with very few samples, the high variance in mean estimation often results in an "accidental" separation that allows the model to correctly classify one of the lobes, achieving a significantly lower error rate.

The Dipping Phenomenon

The core of the paper's contribution is the visualization of non-monotonic learning curves. By adjusting the geometry of synthetic data, such as the radius of a 2D noisy ring, the authors can "tune" the point at which the dip occurs.

A learning curve exhibiting the dipping phenomenon. The error rate reaches a global minimum at a small training size before steadily increasing.

As illustrated above, the error rate initially decreases as the model learns from the first few samples. However, it hits a global minimum at a specific training size (e.g., $N \approx 15$) and then begins to rise as the model converges toward an inferior asymptotic solution dictated by its structural mismatch with the data.

Key Results across Classifiers

The authors tested several standard classifiers, including:

  • Nearest Mean Classifier (NMC)
  • Fisher Discriminant Analysis
  • Logistic Regression
  • Quadratic Discriminant Analysis (QDA)

Even relatively flexible models like QDA were found to be susceptible to dipping when faced with specific multi-modal distributions. For linear classifiers on the 1D mixture problem, the error rate was observed to climb steadily from ~0.3 toward 0.5 as training size increased, confirming that the model's "best self" is often its "small-data self" in these contexts.

Conclusion and Implications

The dipping phenomenon serves as a theoretical proof of concept that challenges fundamental assumptions about data scaling. While demonstrated here on synthetic data, it raises critical questions for real-world applications:

  1. Training Size as a Hyperparameter: If dipping occurs in real-world high-dimensional data, practitioners may need to treat the amount of training data as a hyperparameter to be optimized via cross-validation.
  2. Model Selection: Dipping is a direct consequence of model mismatch. It highlights the importance of choosing models whose hypothesis space can actually encompass the true data distribution.

By identifying dipping, Loog and Duin provide a cautionary tale: more data is only beneficial if your model is capable of using it correctly.

Summary updated

Abstract

One typically expects classifiers to demonstrate improved performance with increasing training set sizes or at least to obtain their best performance in case one has an infinite number of training samples at ones’s disposal. We demonstrate, however, that there are classification problems on which particular classifiers attain their optimum performance at a training set size which is finite. Whether or not this phenomenon, which we term dipping, can be observed depends on the choice of classifier in relation to the underlying class distributions. We give some simple examples, for a few classifiers, that illustrate how the dipping phenomenon can occur. Additionally, we speculate about what generally is needed for dipping to emerge. What is clear is that this kind of learning curve behavior does not emerge due to mere chance and that the pattern recognition practitioner ought to take note of it.
BibTeX
@inproceedings{loog2012dipping,
  author = {Marco Loog and Robert P. W. Duin},
  title = {{The Dipping Phenomenon}},
  booktitle = {SSPR/SPR},
  year = {2012},
  doi = {10.1007/978-3-642-34166-3\_34},
}
Figure