Needles in Haystacks: On Classifying Tiny Objects in Large Images

Nick Pawlowski, Suvrat Bhooshan, Nicolas Ballas, Francesco Ciompi, Ben Glocker, Michal DrozdzalarXiv 2019 · arXiv

In standard computer vision benchmarks like ImageNet, the objects of interest typically occupy a significant portion of the image frame. However, real-world applications—particularly in medical imaging and satellite surveillance—often require identifying "needles in haystacks": tiny objects buried within massive high-resolution backgrounds. Researchers from Facebook AI Research and Imperial College London have investigated the limits of Convolutional Neural Networks (CNNs) in these extreme scenarios.

The Challenge of the O2I Ratio

The core metric introduced in this study is the Object-to-Image (O2I) ratio, which measures the area of the target object relative to the total image area. While objects in ImageNet typically have an O2I ratio of at least 1%, medical histopathology slides can feature cancerous lesions with O2I ratios as low as $10^{-6}$. This discrepancy suggests that standard CNN architectures, optimized for large-object datasets, may not be inherently suited for detecting microscopic signals.

Comparison of O2I ratios across standard and medical datasets, highlighting the extreme scale difference.

Methodology: Probing the Limits

To systematically test how CNNs handle decreasing O2I ratios, the researchers utilized a classification framework based on BagNets. This architecture allows for explicit control over the receptive field (RF)—the specific input region that influences a single feature in the network. The pipeline consists of a topological embedding extractor (CNN), a global pooling operation (testing Max, Average, and Attention mechanisms), and a binary classifier.

Experimental Datasets

Because existing benchmarks do not sufficiently stress-test low O2I ratios, the authors developed two specialized datasets:

  1. nMNIST (needle MNIST): A synthetic dataset where 28x28 MNIST digits are placed randomly on canvases up to 1024x1024, allowing for precise O2I control from 19.1% down to 0.075%.
  2. nCAMELYON (needle CAMELYON): A real-world dataset derived from histopathology, where images are cropped around cancerous lesions to simulate O2I ratios from 100% down to 0.1%.
Examples of varying O2I ratios in the synthetic nMNIST (a, b) and real-world nCAMELYON (c, d) datasets.

Key Insights and Findings

1. The Data-Signal Tradeoff

The study found that as the O2I ratio decreases, the volume of training data required for the model to generalize increases exponentially. Specifically, the data requirement scales with the inverse of the O2I ratio. Finding a "needle" that is half as large requires significantly more than twice the original training data.

2. Receptive Field Optimization

The "sweet spot" for the receptive field depends heavily on the nature of the data. For synthetic digits (nMNIST), a receptive field larger than the object was optimal. Conversely, for biological data (nCAMELYON), a smaller receptive field performed better, suggesting that the network relies on local textures rather than global morphology to identify cancerous cells.

3. The Superiority of Max-Pooling

In scenarios with extremely low signal-to-noise ratios, Max-pooling proved to be the most robust optimization strategy. While average pooling tends to "wash out" tiny signals during the learning process, Max-pooling effectively preserves the strongest local features, allowing the model to converge even when the O2I ratio is minimal.

Conclusion

Standard CNNs can indeed classify tiny objects without pixel-level annotations, but they face a clear generalization limit. On nMNIST, models struggled below a 0.3% O2I ratio, while on nCAMELYON, the limit was approximately 1%. To push past these boundaries, future research must focus on developing stronger inductive biases that allow networks to efficiently ignore background noise and focus on sparse, high-importance signals.

Summary updated

Abstract

In some important computer vision domains, such as medical or hyperspectral imaging, we care about the classification of tiny objects in large images. However, most Convolutional Neural Networks (CNNs) for image classification were developed using biased datasets that contain large objects, in mostly central image positions. To assess whether classical CNN architectures work well for tiny object classification we build a comprehensive testbed containing two datasets: one derived from MNIST digits and one from histopathology images. This testbed allows controlled experiments to stress-test CNN architectures with a broad spectrum of signal-to-noise ratios. Our observations indicate that: (1) There exists a limit to signal-to-noise below which CNNs fail to generalize and that this limit is affected by dataset size - more data leading to better performances; however, the amount of training data required for the model to generalize scales rapidly with the inverse of the object-to-image ratio (2) in general, higher capacity models exhibit better generalization; (3) when knowing the approximate object sizes, adapting receptive field is beneficial; and (4) for very small signal-to-noise ratio the choice of global pooling operation affects optimization, whereas for relatively large signal-to-noise values, all tested global pooling operations exhibit similar performance.
BibTeX
@misc{pawlowski2019needles,
  author = {Nick Pawlowski and Suvrat Bhooshan and Nicolas Ballas and Francesco Ciompi and Ben Glocker and Michal Drozdzal},
  title = {{Needles in Haystacks: On Classifying Tiny Objects in Large Images}},
  year = {2019},
  eprint = {1908.06037},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.06037},
}
Figure