Improved Robustness from Biologically Inspired Sparse Contrast Representations

Lorena Stracke, Lia Nimmermann, Shashank Agnihotri, Bhaskar Choubey, Margret Keuper, Volker BlanzarXiv 2025 · arXiv

TL;DR

Researchers have developed a model-agnostic preprocessing module inspired by the human retina that significantly improves neural network robustness to adverse lighting and weather. By transforming RGB images into sparse, contrast-centered representations, vision models can maintain high accuracy in challenging nighttime and foggy conditions while discarding up to 70% of the input data.

What Problem Does This Paper Address?

While deep neural networks often outperform humans on standard vision benchmarks, they are notoriously fragile when faced with "distribution shifts"—changes in the environment that differ from the training data. A model trained on sunny daytime street scenes often fails spectacularly at night or in heavy fog.

The common "brute force" solutions involve collecting massive amounts of specialized nighttime data, using heavy data augmentation, or designing complex test-time adaptation schemes. This paper takes a different, more elegant route: changing the input representation itself.

The authors argue that standard RGB values are inherently unstable because they represent absolute light intensity, which fluctuates wildly with the time of day or weather. In contrast, the human visual system (HVS) focuses on relative differences (contrast) and opponent color signals, which remain much more stable across different lighting conditions.

Methods

The authors propose a fixed, lightweight preprocessing front-end that mimics early biological vision. It consists of three computational stages:

  1. Color Remapping: The RGB input is converted into one of three variants: a single luminance channel, a three-channel "color-opponency" space (Black-White, Red-Green, Blue-Yellow), or a standard identity mapping as a control.
  2. Spatial Contrast Extraction: Inspired by the center-surround receptive fields of retinal ganglion cells, the module computes a local contrast field. This is done by approximating a Difference of Gaussians (DoG)—essentially subtracting a blurred version of the image from itself. This process strips away absolute illumination and leaves behind structural cues like edges and boundaries.
  3. Sparsification: Since natural images are mostly smooth, the contrast-domain representation is naturally sparse. The authors apply a magnitude threshold, setting the weakest 40% to 90% of contrast responses to zero.
Figure 2. Overview of the four evaluated pipelines. In comparison with a baseline (top row), we consider our preprocessing in three variants: luminance, color-opponency and single-color. Our preprocessing begins with a color remapping (1) from RGB values to three different color spaces. Then follows the contrast extraction (2), where we simulate a convolution with a DoG-like kernel. Finally, we enforce sparsity (3) by setting low absolute values to zero, highlighted in pink. Each variant produces 3 channels that serve as a new input to different neural network architecures (right).

Crucially, this module has no learnable parameters and can be "plugged in" to any existing segmentation architecture (like DeepLabV3+ or Mask2Former) without changing the network's internal structure.

Datasets

The study focuses on semantic segmentation, a demanding task that requires pixel-perfect predictions.

  • Training: All models were trained on the standard Cityscapes dataset (daytime street scenes).
  • Evaluation (Out-of-Distribution): To test robustness, models were evaluated on Dark Zurich (nighttime) and ACDC (which includes subsets for Fog, Rain, Snow, and Night).

Key Insights and Ideas

  • Structure Over Intensity: By forcing the network to look at contrast rather than raw pixel values, the model becomes "blind" to the global lighting shifts that usually cause errors. It focuses on the stable structural features of the world.
  • The Sparsity "Aha" Moment: The most surprising finding is that you can throw away a huge portion of the input data without losing accuracy. In the contrast domain, structural information is concentrated in a small fraction of coefficients.
  • Biological Efficiency: This approach isn't just about software robustness; it has massive implications for hardware. If an image sensor (CMOS) performs this sparsification close to acquisition, it could reduce power consumption and data transmission bandwidth by over 30%.
Figure 3. Visualization and respective value distributions of an ex- ample image with our preprocessing and sparsification. (a) shows the contrast tensor obtained after our preprocessing, using the lu- minance variant (values around 0 appear gray); (b) the same con- trast image after sparsification with p=70%, with affected regions highlighted in pink; (c) shows the original RGB image, and (d) the original RGB image with the same sparsification mask applied as in (b). Important object structures and contours are preserved.

Key Results

The researchers tested their module across five different model families and found consistent improvements:

  • Nighttime Performance: On the ACDC Night benchmark, the biologically inspired preprocessing significantly outperformed the standard RGB baseline. For example, using UPerNet, the preprocessing variants preserved robust structural cues that the baseline missed entirely.
  • Extreme Sparsity: Models trained on these representations remained stable even when inputs were sparsified by up to 70% at inference time. In contrast, standard RGB models saw their performance collapse almost immediately when subjected to the same level of data removal.
  • Generalization: While designed for nighttime, the method also improved results in fog and snow, proving that a "contrast-first" bias is a general-purpose robustness tool.
Figure 4. Robustness improves without sacrificing i.i.d. perfor- mance. This summary view illustrates the main trend: our prepro- cessing maintains high Cityscapes performance while improving nighttime generalization on Dark Zurich.

Limitations and Future Work

The authors note that while their method narrows the gap between daytime and nighttime performance, a gap still exists. They suggest that combining this preprocessing with advanced data augmentation could close it further.

Another limitation is that the optimal "blur depth" (how much the image is blurred during contrast extraction) can vary slightly depending on the specific neural network architecture used. Future work could involve developing "sparsity-aware" architectures that are specifically designed to exploit these biologically inspired, sparse inputs from the ground up, potentially leading to even more efficient and reliable vision systems.

Summary updated

Abstract

Deep neural networks surpass humans on many vision benchmarks, yet remain far less robust to distribution shifts such as illumination and weather changes. Existing approaches address this challenge by additional training data, extensive augmentation, architectural modifications, or test-time adaptation. In this work, we explore a complementary direction: inspired by the human retina, we propose a fixed, model-agnostic preprocessing module that extracts signals that are more stable with respect to variations of illumination. Our method combines color remapping with local contrast extraction, producing sparse representations that emphasize structural features. We study its impact on semantic segmentation by training on Cityscapes and evaluating generalization under adverse conditions on Dark Zurich and ACDC. Our results show that the biologically inspired preprocessing preserves in-distribution performance while consistently improving robustness in challenging lighting scenarios, such as nighttime, where annotated training data are scarce. Moreover, the segmentation accuracy remains stable even when the contrast-based representation is sparsified by up to 70%. These gains suggest that rethinking the input representation itself can improve robustness while also opening opportunities for lower-latency, transmission-aware imaging sensors when sparsity can be exploited close to acquisition.
BibTeX
@misc{stracke2025improved,
  author = {Lorena Stracke and Lia Nimmermann and Shashank Agnihotri and Bhaskar Choubey and Margret Keuper and Volker Blanz},
  title = {{Improved Robustness from Biologically Inspired Sparse Contrast Representations}},
  year = {2025},
  eprint = {2509.24863},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.24863},
}
Figure