Exploring object-centric and scene-centric CNN features and their complementarity for human rights violations recognition in images

Grigorios Kalliatakis, Shoaib Ehsan, Ales Leonardis, Klaus D. McDonald-Maier

Introduction: The Digital Haystack

In conflict zones and impoverished regions, the proliferation of smartphones has led to a massive influx of visual data. Activists and witnesses upload millions of photos and videos, creating a "digital haystack" that human rights investigators must sift through to find evidence of atrocities. Manually identifying specific abuses—such as illegal detention or child labor—is a grueling and time-sensitive task. While computer vision has excelled at identifying common objects like cars or animals, recognizing human rights violations requires a deeper, more nuanced understanding of context and action.

The Human Rights Archive (HRA) Dataset

To address the lack of high-quality, expert-labeled data, researchers introduced the Human Rights Archive (HRA). Unlike previous datasets scraped broadly from the web, HRA consists of 3,050 images sourced from verified NGO records, including Human Rights Watch and the United Nations. This ensures that the "ground truth" labels are legally and ethically accurate.

Overview of the Human Rights Archive (HRA) dataset, showcasing categories such as child labor, detention centers, and displaced populations.

The dataset covers eight primary categories of violations: Arms, Child Labour, Child Marriage, Detention Centres, Disability Rights, Displaced Populations, Environment, and Out of School, along with a "No Violation" control class.

Methodology: Object-Centric vs. Scene-Centric Features

The study investigates which deep learning architectures are best suited for this context-heavy task. The researchers employed a two-phase transfer learning strategy, fine-tuning pre-trained models (VGG16, VGG19, and ResNet50) on the HRA dataset. Crucially, they compared two distinct AI perspectives:

  • Object-centric models: Trained on ImageNet to identify specific entities (e.g., a gun or a child).
  • Scene-centric models: Trained on Places365 to recognize environmental contexts (e.g., a refugee camp or a factory).

The team also explored Feature Fusion, testing whether merging data from both perspectives (Early Fusion) or averaging their final predictions (Late Fusion) would yield better results. To understand the models' decision-making, they used Grad-CAM to produce heatmaps showing which parts of an image the AI focused on.

Key Insights: The Power of Context

A central finding of the research is that human rights violations are defined by actions in context. For instance, a photo of a child is neutral, but a child in a mining pit signifies a violation.

An example from the 'Out of School' category, where the AI must distinguish between a child studying at home and a child deprived of formal education.

The study demonstrated that object-centric and scene-centric models are complementary. In a "displaced populations" image, an object-centric model might focus on a person's face, while a scene-centric model identifies makeshift tents in the background. Combining these views allows the AI to better understand the "tools" of abuse (like a shovel) in relation to their "setting" (a dirt pit).

Results and Future Directions

The VGG19 (object-centric) model emerged as the top individual performer with 35.18% accuracy. However, Late Fusion—averaging predictions from both object and scene models—significantly improved performance over scene-centric models alone.

Despite these gains, the task remains challenging. When high confidence (0.85) was required, models could only provide predictions for 42% to 64% of images. Future work will focus on expanding the dataset, addressing class imbalances (e.g., the scarcity of "Child Marriage" images compared to "Child Labour"), and moving from static images to video analysis to better capture the dynamic actions that constitute human rights abuses.

The HRA dataset and trained models are available on GitHub.

Summary updated

BibTeX
@article{kalliatakis2019exploring,
  author = {Grigorios Kalliatakis and Shoaib Ehsan and Ales Leonardis and Klaus D. McDonald-Maier},
  title = {{Exploring object-centric and scene-centric CNN features and their complementarity for human rights violations recognition in images}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2891745},
}
Figure