Abstract
Scene recognition is a complex computer vision task that requires a holistic understanding of an environment rather than just identifying individual objects. Traditional deep learning models often struggle with high inter-class similarity and significant intra-class variation. Furthermore, these models typically operate as "black boxes," offering little insight into their decision-making process. This paper introduces EnTri, a novel ensemble learning framework that addresses these challenges by processing images through three distinct hierarchical representations: pixel-level textures, semantic layouts, and object frequencies. EnTri not only achieves state-of-the-art accuracy on major benchmarks but also incorporates a Visual Textual Explanation Generator (VTEG) to provide transparent, natural language reasoning for its predictions.
The Challenge of Scene Recognition
Unlike object detection, scene recognition involves interpreting the context of an entire environment. Two primary hurdles hinder performance in this domain:
- Inter-class similarity: Different scene categories can appear remarkably similar. For example, an auditorium and a movie theater share many visual features like rows of seating and a central focal point.
- Intra-class variation: Instances of the same category can look vastly different. Two offices might share no common furniture or layout yet belong to the same class.
Methodology: Tri-Level Logic
The EnTri framework employs a "stacked generalization" approach, breaking down image analysis into three hierarchical levels to capture complementary information:
- Low-Level (Pixel-level): This stream focuses on color, texture, and background intensity. It utilizes state-of-the-art (SOTA) backbones like ConvNeXt-B and EfficientNetV2-L to extract fine-grained visual features.
- Mid-Level (Semantic-level): By using semantic segmentation models (such as DeepLab v2 and PSPNet), EnTri captures the shapes and spatial layouts of the environment. This level helps the model understand the structural composition of a scene without being distracted by pixel-level noise.
- High-Level (Object-level): This stream uses object detectors (YOLOv5, Faster R-CNN) to generate a "bag-of-objects" vector. It essentially counts the frequency of specific objects (e.g., sinks, chairs, toilets) present in the image.
These three streams are integrated by a Meta-model—a fully connected network that learns the optimal weighting for each perspective to produce a final classification.
Explainability via VTEG
To address the "black box" nature of deep learning, the authors developed the Visual Textual Explanation Generator (VTEG). This component identifies the most influential features from each of the three levels and synthesizes them into a natural language report. For instance, instead of just providing a heatmap, the model might explain: "The model agrees 99.96% with the bathroom category because of a dark gray sink in the bottom right..." This approach makes the AI's reasoning significantly more accessible and trustworthy for human users.
Experimental Results
EnTri was evaluated on three major benchmarks, consistently outperforming existing SOTA methods:
- MIT67: Achieved 87.69% accuracy (vs. 87.10% for previous leaders).
- SUN397: Achieved 75.56% accuracy, a significant improvement over the previous SOTA of 74.18%.
- UIUC8: Reached 99.17% accuracy.
Ablation studies confirmed that the integration of all three levels is critical for peak performance, as the combination of low, mid, and high-level features consistently yielded the highest accuracy across all datasets.
Conclusion
EnTri demonstrates that a hierarchical, multi-level approach is essential for mastering the nuances of scene recognition. By combining pixel-level detail with structural layouts and object-level context, the framework overcomes traditional hurdles of similarity and variation. Furthermore, the inclusion of the VTEG module sets a new standard for explainability in computer vision, paving the way for the deployment of these models in high-stakes domains where transparency is paramount.
Summary updated