A Model is Worth Tens of Thousands of Examples

Thomas Dagès, Laurent D. Cohen, Alfred M. Bruckstein

In the current landscape of artificial intelligence, the dominant paradigm is often "more is better"—more data, more parameters, and more compute. However, this approach often overlooks the efficiency of classical, model-based mathematical solutions. A recent study by Thomas Dagès and colleagues from Technion and Université Paris Dauphine provides a concrete quantification of this trade-off. Their research suggests that for well-defined tasks, a robust mathematical model is equivalent to approximately 10,000 to 100,000 training examples.

The Sample Complexity Gap

Deep neural networks (DNNs) have largely supplanted traditional signal processing methods. While DNNs are remarkably flexible, their "sample complexity"—the amount of data required to achieve a certain level of performance—is often high and difficult to predict. Engineers frequently rely on heuristics, but these fail to provide a direct comparison with classical benchmarks like the Wiener filter. The authors of this study sought to identify the exact point at which a purely data-driven approach begins to outperform a model-driven one.

Experimental Methodology

The researchers evaluated performance on two fundamental tasks where optimal or state-of-the-art classical solutions are well-established:

  1. 1D Signal Recovery: This involves deconvolving Gaussian signals that have been degraded by circulant blur and additive noise. The baseline for this task is the Wiener Filter, the provably optimal linear recovery method.
  2. 2D Geometric Estimation: This task requires estimating the radius and center coordinates of a disk within a noisy, blurred grayscale image. The baseline is Pointflow, a classical subpixel contour integrator that operates without any training.

For the neural approaches, the study utilized simple CNNs for the 1D task and established architectures like AlexNet, VGG11, and ResNet18 for the 2D task. Performance was tracked as the training set size ($N$) scaled from 10 to 100,000 samples.

Dataset Characteristics and Priors

To ensure a rigorous comparison, the study employed synthetic datasets with known ground truths. For the 1D task, signals were generated using specific autocorrelation matrices ($R_\varphi$) to define their statistical structure, representing the mathematical "prior" that a model-based approach uses for free.

The autocorrelation matrix $R_\varphi$ defines the statistical structure of the 1D signals, representing the "model" prior.

Key Insights: The 10,000 Sample Threshold

The most striking finding across both tasks was the consistency of the "data debt" incurred by neural networks. In both 1D and 2D scenarios, networks typically required at least 10,000 training examples to match the performance of the expert mathematical models.

Comparison of a 1D signal ($\varphi$), its noisy/blurred version ($H\varphi$), and the reconstruction ($\varphi^{data}$) by a neural network.

The Failure of Generic Features

A significant observation was the poor performance of Transfer Learning (TL) on the geometric estimation task. When using pre-trained ResNet or VGG models with frozen weights (trained on ImageNet), the networks failed to accurately estimate disk parameters. This suggests that features optimized for object classification (e.g., identifying a "dog") do not naturally generalize to basic Euclidean geometric tasks.

Asymptotic Convergence vs. Physical Priors

While neural networks do eventually converge toward optimal solutions as $N$ increases, they must "re-learn" the underlying physics or mathematics that a classical model incorporates by design. A mathematical model essentially acts as an infinite source of high-quality "virtual" data, providing a significant head start in low-data regimes.

Conclusion

For engineers and researchers, the takeaway is clear: if the data generation process is well-understood—such as in physics-based imaging or geometric tasks—leveraging a mathematical model is far more efficient than relying solely on a "black box" neural network. Before embarking on massive data collection and labeling efforts, it is worth investigating whether a classical solution can provide the necessary performance with zero training data.

Summary updated

Abstract

Traditional signal processing methods relying on mathematical data generation models have been cast aside in favour of deep neural networks, which require vast amounts of data. Since the theoretical sample complexity is nearly impossible to evaluate, these amounts of examples are usually estimated with crude rules of thumb. However, these rules only suggest when the networks should work, but do not relate to the traditional methods. In particular, an interesting question is: how much data is required for neural networks to be on par or outperform, if possible, the traditional model-based methods? In this work, we empirically investigate this question in two simple examples, where the data is generated according to precisely defined mathematical models, and where well-understood optimal or state-of-the-art mathematical data-agnostic solutions are known. A first problem is deconvolving one-dimensional Gaussian signals and a second one is estimating a circle's radius and location in random grayscale images of disks. By training various networks, either naive custom designed or well-established ones, with various amounts of training data, we find that networks require tens of thousands of examples in comparison to the traditional methods, whether the networks are trained from scratch or even with transfer-learning or finetuning.
BibTeX
@inproceedings{dages2023model,
  author = {Thomas Dagès and Laurent D. Cohen and Alfred M. Bruckstein},
  title = {{A Model is Worth Tens of Thousands of Examples}},
  booktitle = {SSVM},
  year = {2023},
  doi = {10.1007/978-3-031-31975-4\_17},
}
Figure