Exploring Object-Centric and Scene-Centric CNN Features and Their Complementarity for Human Rights Violations Recognition in Images

Grigorios Kalliatakis, Shoaib Ehsan, Aleš Leonardis, Klaus D. McDonald-Maier

Introduction

Human rights investigators today face an overwhelming volume of digital evidence. From social media uploads to NGO reports, the manual verification of thousands of images is a daunting task. While computer vision has excelled at identifying common objects and scenes, recognizing "human rights violations" represents a significantly more complex semantic challenge. This paper introduces the Human Rights Archive (HRA), a benchmark dataset designed to bridge this gap by training AI to understand the nuanced visual context of abuses.

The Human Rights Archive (HRA) Dataset

The primary contribution of this work is the HRA database, which consists of 3,050 images across eight violation categories: Arms, Child Labour, Child Marriage, Detention Centres, Disability Rights, Displaced Populations, Environment, and Out of School. To ensure high-quality "ground truth" labels, images were sourced from verified NGO repositories like Human Rights Watch and the United Nations, and were meticulously verified by experts.

Example image from the HRA dataset labeled as 'mining', representing labor-related violations.

The dataset also includes a "No Violation" control class to test the models' ability to distinguish between benign and harmful contexts.

Methodology: Objects vs. Scenes

The researchers employed a two-phase transfer learning strategy, fine-tuning state-of-the-art Convolutional Neural Networks (CNNs) for the specific task of human rights recognition. A key focus was the comparison between two types of visual features:

  • Object-Centric Features: Derived from models like VGG16 and ResNet50 pre-trained on ImageNet, focusing on specific items (e.g., weapons or tools).
  • Scene-Centric Features: Derived from models like VGG16-Places365, focusing on the broader environment (e.g., a factory or a refugee camp).
Example image from the HRA dataset labeled as 'garment', illustrating the importance of scene context in identifying labor rights issues.

To optimize performance, the study explored Early Fusion (merging features internally) and Late Fusion (averaging final predictions). Visualization tools like Grad-CAM were used to interpret which visual cues the models prioritized.

Key Insights and Results

The experiments revealed a "complementarity gap": object-centric models excel at identifying the "tools" of a violation, while scene-centric models are better at recognizing the "setting." The most robust results came from Late Fusion, which combines both perspectives to better understand the interaction between people and their environments.

The task proved exceptionally difficult; the top-performing model, an object-centric VGG19, achieved a top-1 accuracy of 35.18%. This relatively low figure underscores the complexity of the HRA benchmark compared to standard image classification tasks. To maintain reliability in high-stakes human rights monitoring, the authors implemented a confidence threshold (85%), prioritizing accuracy over total coverage.

Conclusion and Future Directions

The HRA dataset establishes a new frontier for computer vision in social justice. While current models struggle with the distinction between "evidence" (a building) and "action" (an attack), future work aims to incorporate multi-label classification and natural language descriptions to provide investigators with richer, more actionable context.

Summary updated

Abstract

Identifying potential abuses of human rights through imagery is a novel and challenging task in the field of computer vision, which will enable to expose human rights violations over large-scale data that may otherwise be impossible. While standard databases for object and scene categorization contain hundreds of different classes, the largest available dataset of human rights violations contains only four classes. Here, we introduce the human rights archive (HRA) database, a verified-by-experts repository of 3050 human rights violations photographs, labeled with human rights semantic categories, comprising a list of the types of human rights abuses encountered at present. With the HRA dataset and a two-phase transfer learning scheme, we fine-tuned the state-of-the-art deep convolutional neural networks (CNNs) to provide human rights violations classification CNNs. We also present extensive experiments refined to evaluate how well object-centric and scene-centric CNN features can be combined for the task of recognizing human rights abuses. With this, we show that the HRA database poses a challenge at a higher level for the well-studied representation learning methods and provide a benchmark in the task of human rights violations recognition in visual context. We expect that this dataset can help to open up new horizons on creating systems that are able to recognize rich information about human rights violations.
BibTeX
@article{kalliatakis2019exploring,
  author = {Grigorios Kalliatakis and Shoaib Ehsan and Aleš Leonardis and Klaus D. McDonald-Maier and Klaus D. McDonald-Maier},
  title = {{Exploring Object-Centric and Scene-Centric CNN Features and Their Complementarity for Human Rights Violations Recognition in Images}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2891745},
}
Figure