Abstract
Although the popular MNIST dataset [LeCun et al., 1994] is derived from the NIST database [Grother and Hanaoka, 1995], the precise processing steps for this derivation have been lost to time. We propose a reconstruction that is accurate enough to serve as a replacement for the MNIST dataset, with insignificant changes in accuracy. We trace each MNIST digit to its NIST source and its rich metadata such as writer identifier, partition identifier, etc. We also reconstruct the complete MNIST test set with 60,000 samples instead of the usual 10,000. Since the balance 50,000 were never distributed, they enable us to investigate the impact of twenty-five years of MNIST experiments on the reported testing performances. Our results unambiguously confirm the trends observed by Recht et al. [2018, 2019]: although the misclassification rates are slightly off, classifier ordering and model selection remain broadly reliable. We attribute this phenomenon to the pairing benefits of comparing classifiers on the same digits.
Introduction
MNIST is the "Hello World" of machine learning. Since the mid-90s, thousands of papers have used its 10,000-image test set to prove the superiority of new algorithms. However, this ubiquity created a looming crisis of confidence. In statistics, if you "test" on the same data too many times, you eventually overfit to that specific set—a phenomenon known as "test set rot."
Furthermore, the original code used to transform the raw NIST source images into the normalized MNIST format was lost to time. We knew what MNIST was, but we didn't exactly know how it was made. This paper aims to solve both: reconstruct the lost preprocessing pipeline and use it to generate a "fresh" test set to see if 25 years of ML research has been one giant exercise in overfitting.
Digital Archaeology: Reconstructing the Pipeline
The authors engaged in what can only be described as "digital archaeology." To recreate the dataset, they iteratively adjusted cropping and centering algorithms until the resulting pixels matched the official MNIST digits.
A key discovery was that the original MNIST didn't use standard bilinear or bicubic interpolation. Instead, the researchers found a "mysterious" algorithm in the ancient Lush codebase that calculates the exact overlap of input and output pixel areas. To ensure a perfect match, they used the Hungarian assignment algorithm to pair each MNIST digit with its most likely NIST source based on $L_2$ distance.
The QMNIST Dataset
The result of this effort is QMNIST, which includes QMNIST10K (matching the original test set) and QMNIST50K (the "lost" digits that were never distributed). By tracing digits back to NIST, the authors recovered lost metadata, such as the specific writer ID for every single digit. This revealed that the first 5,001 test images were written by high school students, while the remaining 4,999 were written by Census Bureau employees—a distribution shift hidden in plain sight for decades.
Key Results: Fidelity and Generalization
The reconstruction accuracy is remarkably high. QMNIST images are nearly identical to MNIST, with only about 0.25% of images showing a one-pixel shift due to rounding differences in the center-of-gravity calculation.
Testing a variety of models (KNN, SVM, MLP, CNNs) showed that performance on the "lost" 50k digits is highly correlated with the original 10k. For example, a LeNet-5 model that gets ~0.8% error on MNIST gets ~1.1% on the fresh QMNIST50K. While absolute error rates are slightly higher on the fresh data, the relative ranking of models stays the same. If Model A beat Model B on the old MNIST, it almost certainly beats it on the new QMNIST.
Discussion: The Power of Pairing
The authors attribute this stability to "pairing benefits." Because researchers compare different classifiers on the exact same digits, the relative differences remain statistically significant even if the absolute performance numbers drift. This suggests that the machine learning community's reliance on MNIST has not led to the catastrophic overfitting many feared.
Conclusion
"Test set rot" is a real but manageable problem. While absolute performance metrics may drift over decades of intensive benchmarking, model selection remains broadly reliable. This digital archaeology not only recovers a lost piece of computer science history but also reinforces the validity of the progress made in handwritten digit recognition over the last quarter-century.
Summary updated