Abstract: AI-powered automatic camera scene detection mode is nowadays available in nearly any modern smartphone, though the problem of accurate scene prediction has not yet been addressed by the research community. This paper for the first time carefully defines this problem and proposes a novel Camera Scene Detection Dataset (CamSDD) containing more than 11K manually crawled images belonging to 30 different scene categories. We propose an efficient and NPU-friendly CNN model for this task that demonstrates a top-3 accuracy of 99.5% on this dataset and achieves more than 200 FPS on the recent mobile SoCs. An additional in-the-wild evaluation of the obtained solution is performed to analyze its performance and limitation in the real-world scenarios.
Introduction: The Scene Detection Challenge
While "AI Scene Detection" is a staple feature in modern smartphones—automatically adjusting exposure, white balance, and saturation—the research community has lacked a standardized, public benchmark. Most existing image classification datasets, such as ImageNet or Places, do not align with the specific requirements of a camera application. For instance, a mobile device must distinguish between "Gourmet food" to boost saturation, "Documents" to trigger OCR, and "Backlight" to activate HDR. Historically, manufacturers have relied on proprietary, "black box" solutions. This paper bridges the gap by formally defining the problem and providing open-source tools and datasets to solve it.
The CamSDD Dataset
The core contribution of this work is the Camera Scene Detection Dataset (CamSDD). It consists of over 11,000 images manually crawled from Flickr and rigorously inspected. The dataset covers 30 diverse classes essential for mobile photography, including Portrait, Macro, Gourmet, Underwater, Architecture, QR Code, and Monitor Screen.
CamSDD was specifically curated to include various angles, lighting conditions, and "in-the-wild" variations to ensure that models trained on it generalize effectively to real-world smartphone usage.
Proposed Architecture
The authors developed their solution using lightweight backbones: MobileNet-V1 and MobileNet-V2. To adapt these for scene detection, several key modifications were implemented:
- Custom Head: The standard ImageNet classification head was replaced with custom fully connected layers (e.g., 1024 units with 0.7 dropout).
- Activation Tuning: Empirical testing revealed that using a Sigmoid activation function in the final fully connected layer (prior to the Softmax output) yielded superior accuracy compared to standard ReLU or Tanh.
- Mobile Optimization: To ensure real-time performance, the models were optimized using INT8 quantization. This allows the models to leverage dedicated hardware like Neural Processing Units (NPUs) and Digital Signal Processors (DSPs) rather than relying solely on the CPU/GPU.
Key Results and Insights
The proposed models achieved a top-1 accuracy of 94.17% and a top-3 accuracy of 99.50%. Beyond raw accuracy, the research highlighted several critical insights for mobile deployment:
- Hardware Acceleration: On high-end SoCs like the MediaTek Dimensity 1000+, the model performance jumped from a few dozen FPS to over 233 FPS when properly accelerated on the NPU.
- Generalization vs. Precision: While MobileNet-V2 showed higher top-1 accuracy on the test set, MobileNet-V1 features were often found to be more "general," performing better in complex, "in-the-wild" conditions not explicitly covered in training.
- Quantization Efficiency: Moving from floating-point (FP16) to quantized (INT8) models significantly reduced model size (e.g., from 73MB to 19MB for MobileNet-V2) with negligible impact on accuracy.
Limitations and Future Work
Despite the high accuracy, the models face challenges in extreme lighting conditions, where high dynamic range (over- or under-exposure) can lead to misclassifications (e.g., an over-exposed sky being labeled as "Snow"). Additionally, the current implementation is sensitive to phone orientation and lacks a robust "out-of-distribution" handler for scenes that do not fit the 30 defined categories. Future work will explore combining features from both V1 and V2 backbones and incorporating gyroscope data to improve orientation robustness.
Summary updated