Abstract
AI-powered automatic camera scene detection mode is nowadays available in nearly any modern smartphone, though the problem of accurate scene prediction has not yet been addressed by the research community. This paper for the first time carefully defines this problem and proposes a novel Camera Scene Detection Dataset (CamSDD) containing more than 11K manually crawled images belonging to 30 different scene categories. We propose an efficient and NPU-friendly CNN model for this task that demonstrates a top-3 accuracy of 99.5% on this dataset and achieves more than 200 FPS on the recent mobile SoCs. An additional in-the-wild evaluation of the obtained solution is performed to analyze its performance and limitation in the real-world scenarios. The dataset and pre-trained models used in this paper are available on the project website.
Introduction
If you’ve used a modern smartphone, you’ve likely seen a small icon pop up—a moon for night mode, a fork for food, or a flower for a macro shot. This is automatic camera scene detection. It is a critical feature that allows the phone to adjust exposure, ISO, white balance, and "bokeh" effects on the fly to get the best possible shot. Despite being a staple in consumer tech, the research community has lacked a standardized, public benchmark for this task. Most manufacturers develop proprietary, "black box" solutions. This research fills that gap by defining the problem formally and providing a diverse dataset and efficient model architecture.
The Camera Scene Detection Dataset (CamSDD)
The core contribution of this work is the Camera Scene Detection Dataset (CamSDD). To ensure the model can handle the variety of real-world photography, the authors manually crawled and inspected over 11,000 images from Flickr.
Unlike datasets like CIFAR-10, which use low-resolution images, CamSDD uses higher-resolution images (resized to 576×384 for training) to preserve the fine textures necessary for scene recognition. The dataset covers 30 categories, including mobile-centric scenes like "QR Code," "Monitor Screen," "Document," "Candlelight," and "Gourmet." Each category contains roughly 350 images to prevent the model from developing a bias toward more common scenes.
Efficient Mobile Architectures
The authors focused on "NPU-friendly" architectures—models optimized for the Neural Processing Units found in modern mobile Systems on a Chip (SoCs). They leveraged MobileNet-V1 and MobileNet-V2, which use depthwise separable convolutions to drastically reduce computational overhead.
Key technical refinements included:
- Transfer Learning: Models were pre-trained on ImageNet. The best results came from keeping the original ImageNet-trained convolutional weights and only training new fully connected layers on top.
- Activation Functions: In a departure from standard practice, the researchers found that using a Sigmoid activation function in the final fully connected layer (just before the Softmax output) resulted in higher accuracy than ReLU or Tanh.
- Quantization: To ensure the models could run on mobile DSPs and NPUs, they utilized INT8 quantization, shrinking the model size (down to 19MB for MobileNet-V2) while maintaining high performance.
Key Results and Performance
The proposed models achieved impressive benchmarks on the CamSDD test set, with the best configurations reaching 94.17% top-1 and 99.5% top-3 accuracy. Beyond raw accuracy, the models demonstrated exceptional efficiency. On high-end SoCs like the MediaTek Dimensity 1000+ or Snapdragon 888, the models reached speeds of 200–233 FPS. Even on mid-range hardware, they comfortably exceeded 30 FPS, making them viable for real-time viewfinder integration.
Qualitative testing showed the models were robust against "intra-class variation," correctly identifying subjects like "Cat" regardless of breed, color, or lighting conditions. However, the researchers noted that MobileNet-V1 actually performed better in top-3 accuracy (99.5%) than V2, suggesting that V1 learns more "general" features that are more robust to messy, in-the-wild conditions.
Limitations and Future Work
While the results are promising, several challenges remain for real-world deployment:
- Hardware Constraints: On low-end devices with limited dynamic range, certain scenes (like waterfalls) can be misidentified as snow due to overexposed white areas.
- Orientation: Current models do not utilize the phone's gyroscope, which can lead to struggles if the user takes photos in landscape mode when the training data was primarily portrait-oriented.
- Out-of-Distribution Scenes: When pointed at a scene not covered by the 30 categories, the model produces near-random probabilities. Future iterations will require a robust "none of the above" class or thresholding to avoid flickering UI icons.
Future research directions include integrating temporal data from video frames to stabilize detection and exploring multi-backbone ensembles.
Summary updated