EnTri: Ensemble Learning with Tri-level Representations for Explainable Scene Recognition

Amirhossein Aminimehr, Amirali Molaei, Erik CambriaarXiv 2023 · DOI · arXiv

Abstract

Scene recognition is a complex computer vision task that requires a holistic understanding of an environment rather than just identifying individual objects. Traditional deep learning models often struggle with high inter-class similarity and significant intra-class variation. Furthermore, these models typically operate as "black boxes," offering little insight into their decision-making process. This paper introduces EnTri, a novel ensemble learning framework that addresses these challenges by processing images through three distinct hierarchical representations: pixel-level textures, semantic layouts, and object frequencies. EnTri not only achieves state-of-the-art accuracy on major benchmarks but also incorporates a Visual Textual Explanation Generator (VTEG) to provide transparent, natural language reasoning for its predictions.

The Challenge of Scene Recognition

Unlike object detection, scene recognition involves interpreting the context of an entire environment. Two primary hurdles hinder performance in this domain:

  1. Inter-class similarity: Different scene categories can appear remarkably similar. For example, an auditorium and a movie theater share many visual features like rows of seating and a central focal point.
  2. Intra-class variation: Instances of the same category can look vastly different. Two offices might share no common furniture or layout yet belong to the same class.
Challenges in scene recognition: inter-class similarity between auditoriums and movie theaters (top) and intra-class variation among different office environments (bottom).

Methodology: Tri-Level Logic

The EnTri framework employs a "stacked generalization" approach, breaking down image analysis into three hierarchical levels to capture complementary information:

  • Low-Level (Pixel-level): This stream focuses on color, texture, and background intensity. It utilizes state-of-the-art (SOTA) backbones like ConvNeXt-B and EfficientNetV2-L to extract fine-grained visual features.
  • Mid-Level (Semantic-level): By using semantic segmentation models (such as DeepLab v2 and PSPNet), EnTri captures the shapes and spatial layouts of the environment. This level helps the model understand the structural composition of a scene without being distracted by pixel-level noise.
  • High-Level (Object-level): This stream uses object detectors (YOLOv5, Faster R-CNN) to generate a "bag-of-objects" vector. It essentially counts the frequency of specific objects (e.g., sinks, chairs, toilets) present in the image.
Visualization of mid-level semantic segmentation layouts and high-level object frequency vectors used in the EnTri framework.

These three streams are integrated by a Meta-model—a fully connected network that learns the optimal weighting for each perspective to produce a final classification.

Explainability via VTEG

To address the "black box" nature of deep learning, the authors developed the Visual Textual Explanation Generator (VTEG). This component identifies the most influential features from each of the three levels and synthesizes them into a natural language report. For instance, instead of just providing a heatmap, the model might explain: "The model agrees 99.96% with the bathroom category because of a dark gray sink in the bottom right..." This approach makes the AI's reasoning significantly more accessible and trustworthy for human users.

Experimental Results

EnTri was evaluated on three major benchmarks, consistently outperforming existing SOTA methods:

  • MIT67: Achieved 87.69% accuracy (vs. 87.10% for previous leaders).
  • SUN397: Achieved 75.56% accuracy, a significant improvement over the previous SOTA of 74.18%.
  • UIUC8: Reached 99.17% accuracy.

Ablation studies confirmed that the integration of all three levels is critical for peak performance, as the combination of low, mid, and high-level features consistently yielded the highest accuracy across all datasets.

Conclusion

EnTri demonstrates that a hierarchical, multi-level approach is essential for mastering the nuances of scene recognition. By combining pixel-level detail with structural layouts and object-level context, the framework overcomes traditional hurdles of similarity and variation. Furthermore, the inclusion of the VTEG module sets a new standard for explainability in computer vision, paving the way for the deployment of these models in high-stakes domains where transparency is paramount.

Summary updated

Abstract

Scene recognition based on deep-learning has made significant progress, but there are still limitations in its performance due to challenges posed by inter-class similarities and intra-class dissimilarities. Furthermore, prior research has primarily focused on improving classification accuracy, yet it has given less attention to achieving interpretable, precise scene classification. Therefore, we are motivated to propose EnTri, an ensemble scene recognition framework that employs ensemble learning using a hierarchy of visual features. EnTri represents features at three distinct levels of detail: pixel-level, semantic segmentation-level, and object class and frequency level. By incorporating distinct feature encoding schemes of differing complexity and leveraging ensemble strategies, our approach aims to improve classification accuracy while enhancing transparency and interpretability via visual and textual explanations. To achieve interpretability, we devised an extension algorithm that generates both visual and textual explanations highlighting various properties of a given scene that contribute to the final prediction of its category. This includes information about objects, statistics, spatial layout, and textural details. Through experiments on benchmark scene classification datasets, EnTri has demonstrated superiority in terms of recognition accuracy, achieving competitive performance compared to state-of-the-art approaches, with an accuracy of 87.69%, 75.56%, and 99.17% on the MIT67, SUN397, and UIUC8 datasets, respectively.
BibTeX
@misc{aminimehr2023entri,
  author = {Amirhossein Aminimehr and Amirali Molaei and Erik Cambria},
  title = {{EnTri: Ensemble Learning with Tri-level Representations for Explainable Scene Recognition}},
  year = {2023},
  eprint = {2307.12442},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.12442},
}
Figure