Introduction
The MNIST dataset is widely considered the "Hello World" of machine learning. Despite its ubiquity, it harbors two major historical blind spots. First, the exact preprocessing steps used to convert the original NIST source images into the famous 28x28 MNIST digits were lost to time. Second, while the original NIST test set contained 60,000 samples, only 10,000 were ever distributed as the "official" MNIST test set.
This historical gap creates a modern scientific dilemma: after 25 years of researchers evaluating models on those exact same 10,000 images, have we collectively "overfit" the test set? Is the continuous progress reported in literature genuine, or is it merely "test set rot"? Without the remaining 50,000 samples to act as a fresh, unseen benchmark, this question remained unanswerable.
Digital Archaeology and Reconstruction
To solve this, researchers engaged in an iterative process of "digital archaeology" to create QMNIST (the "Q" standing for "reconstructed"). The goal was to trace every MNIST digit back to its original NIST source and perfectly replicate the lost preprocessing pipeline.
The reconstruction involved several key steps:
- Codebase Excavation: The team hunted down a mysterious resampling algorithm buried in the ancient C++ codebase of Lush, a predecessor to modern deep learning frameworks. Unlike standard bilinear interpolation, this specific code computed the exact overlap of input and output pixels.
- Metadata Matching: Using NIST metadata, such as writer IDs, they traced individual MNIST digits back to their original high-resolution source images.
- Iterative Refinement: The Hungarian assignment algorithm was used to match the newly reconstructed digits to the original MNIST samples. By examining the "worst matches," the researchers fine-tuned their cropping, centering, and anti-aliasing algorithms.
- The "Fudge Factor": A crucial breakthrough was discovering a 0.99 "fudge factor" in the original code's resampling logic, which was absolutely necessary to perfectly replicate the historical anti-aliasing patterns.
Key Findings and Insights
Reconstruction Accuracy and The Centering Error
The resulting QMNIST images are nearly identical to the original MNIST dataset. Approximately 99.75% of the images align perfectly. The remaining 0.25% differ by only a single pixel shift, which the authors traced to floating-point rounding errors at the boundary of the center-of-gravity calculation in the original pipeline.
Interestingly, this revealed that MNIST digits aren't perfectly centered. There is a systematic rounding error in the original code that shifts the center of mass by half a pixel. Paradoxically, training models on "perfectly" centered images actually leads to worse performance on the standard MNIST test set.
Test Set Robustness and Pairing Benefits
With the reconstruction complete, the authors recovered the 50,000 "lost" test images that were never previously distributed. Testing modern models on this fresh data yielded the paper's most significant finding: "test set rot" is far less severe than feared.
Modern models (like CNNs) do show a slight drop in absolute performance on the "lost" 50k samples compared to the standard 10k samples. For example, a model achieving a 0.82% error rate on MNIST might show a 1.08% error rate on the new 50k set. However, the relative ranking of models (e.g., SVM vs. KNN vs. ResNet) remains almost perfectly consistent across both the old and new test sets.
The authors attribute this robustness to "pairing benefits." When comparing two models on the exact same set of digits, the relative difference in their performance is much more stable than their absolute accuracy metrics.
Conclusion
While the QMNIST reconstruction is incredibly close, it is not a 100% bit-for-bit match due to the historical rounding anomalies in the centering step. Furthermore, while model rankings are stable, the absolute error rates reported in the literature over the last two decades are likely slightly optimistic due to test set reuse.
Ultimately, the authors suggest that the real danger to machine learning benchmarks isn't necessarily "overusing" a test set, but rather "distribution shift"—when the real-world data distribution changes over time, rendering the benchmark obsolete. For MNIST, however, the cold case is closed: the benchmark's relative rankings still hold up remarkably well.
Summary updated