Abstract
Deep neural networks (DNNs) have achieved remarkable success in computer vision, yet they remain susceptible to environmental variations not well-represented in training data. This paper investigates the impact of incorrect white balance (WB)—a frequent failure in camera image signal processors (ISPs)—on DNN performance. We demonstrate that these global color shifts act as a form of adversarial attack, significantly degrading accuracy in classification and semantic segmentation. To address this, we propose a novel data-driven augmentation technique that realistically emulates camera-specific WB errors. Our method improves model robustness in real-world conditions by bridging the gap between idealized training sets and the "ugly" images encountered in the wild.
Introduction
Most state-of-the-art computer vision models are trained on large-scale datasets like ImageNet or COCO. These datasets, largely sourced from the web, suffer from a significant selection bias: they predominantly contain high-quality images with correct white balance. In contrast, real-world photography often involves challenging lighting conditions where a camera's computational color constancy fails, resulting in strong blue or orange color casts. Because DNNs are rarely exposed to these variations during training, they exhibit surprising fragility when deployed in uncontrolled environments.
The Impact of White Balance Errors
Incorrect white balance is more than just a cosmetic issue; it represents a global perturbation of image statistics. Unlike standard adversarial attacks that rely on local pixel changes, WB errors shift every pixel in a correlated, non-linear fashion. This can fundamentally alter the features a network relies on for recognition.
As shown in Figure 1, a simple shift in color temperature can cause a "Persian cat" to be misclassified as a "carton" or a "Walker Hound" as a "snorkel." In semantic segmentation tasks, these errors lead to significant mislabeling of regions, such as confusing road surfaces with grass or walls with plants. Our analysis shows that top-tier models like ResNet-50 experience a performance drop of over 10% when faced with these common camera errors.
Proposed Method: Realistic WB Emulation
Standard data augmentation techniques, such as random color jittering or channel swapping, often fail to improve robustness against WB errors. These methods frequently produce unrealistic colors—such as green skin or purple grass—that do not accurately reflect the physical process of camera ISP failures.
The complexity of WB errors in standard sRGB images arises from the non-linear operations (e.g., tone mapping, gamma correction) applied by the ISP after the initial white balance step. To capture this, we propose a data-driven framework that emulates realistic WB errors using a reference dataset of images rendered with diverse WB settings.
Our pipeline, detailed in Figure 5, involves three key components:
- Reference Dataset: A collection of images with both "correct" and "incorrect" WB settings.
- Non-linear Color Mapping: We utilize high-degree polynomial transformations to model the complex relationship between correctly and incorrectly balanced sRGB images.
- KNN-based Retrieval: For any given training image, we extract color histogram features and use K-Nearest Neighbors (KNN) to find similar examples in our reference set. We then apply the corresponding learned transformation matrices to generate realistic augmented versions of the input.
Experimental Results
We evaluated our method across several benchmarks, including CIFAR-10/100, ImageNet, and ADE20K. We also introduced a "Cat-2" dataset consisting of real-world images rendered directly from raw-RGB data to ensure authentic testing conditions.
Our results demonstrate significant improvements in robustness. On CIFAR-10, models trained with our WB augmentation saw accuracy on incorrectly balanced images increase from 65.5% to 78.6%. Furthermore, in semantic segmentation on the ADE20K dataset, our method improved both Mean Intersection-over-Union (mIoU) and pixel-wise accuracy, indicating that realistic color augmentation enhances a model's spatial understanding and feature consistency.
Conclusion
This research highlights a critical vulnerability in current deep learning models: their sensitivity to common camera-induced color constancy errors. By treating these errors as a global adversarial challenge and introducing a realistic, data-driven emulation framework for augmentation, we can significantly enhance the reliability of computer vision systems in real-world applications. Future work will explore integrating these simulations directly into hardware-level ISP pipelines to further bridge the gap between synthetic training and real-world deployment.
Summary updated