Exploring Object-Centric and Scene-Centric CNN Features and Their Complementarity for Human Rights Violations Recognition in Images

Grigorios Kalliatakis, Shoaib Ehsan, Ales Leonardis, Maria Fasli, Klaus D. McDonald-Maier

TL;DR

Researchers have introduced the Human Rights Archive (HRA), the first large-scale, expert-verified image dataset for identifying human rights violations. By comparing AI models trained to recognize "objects" versus those trained to recognize "scenes," the study demonstrates that human rights monitoring requires a sophisticated blend of both focal and contextual visual information.

What Problem Does This Paper Address?

Human rights investigators are currently drowning in digital evidence. Between social media uploads, NGO reports, and citizen journalism, the sheer volume of imagery makes manual verification nearly impossible. While computer vision has mastered identifying cats or living rooms, recognizing a "human rights violation" is a much higher-level semantic task.

The challenge is twofold:

  1. Data Scarcity: Prior to this work, the largest available dataset for this niche contained only four classes with 100 images each—far too small for robust deep learning.
  2. Contextual Ambiguity: A photo of a child holding a hammer might be "play" in one context but "child labor" in another. Standard AI models often struggle with these nuances because they focus either too narrowly on objects or too broadly on the overall scene.

Methods

The authors employed a two-phase transfer learning strategy. Instead of training a model from scratch (which requires millions of images), they took state-of-the-art Convolutional Neural Networks (CNNs) already "educated" on massive datasets and fine-tuned them for human rights recognition.

The study specifically compared two "flavors" of CNNs:

  • Object-Centric: Models (like VGG16 and ResNet50) pre-trained on ImageNet to recognize specific items and people.
  • Scene-Centric: Models (VGG16-Places365) pre-trained to recognize environments like "hospital," "classroom," or "refugee camp."

To find the best approach, they experimented with Early Fusion (merging the internal features of both models before making a decision) and Late Fusion (averaging the final predictions of two separate models). They also used Grad-CAM—a visualization tool—to produce heatmaps showing exactly which parts of an image the AI was looking at when it made a classification.

Datasets

The core of this paper is the Human Rights Archive (HRA).

  • Size: 3,050 images.
  • Categories: 8 violation types (Arms, Child Labour, Child Marriage, Detention Centres, Disability Rights, Displaced Populations, Environment, Out of School) plus a "No Violation" control class.
  • Source: Images were meticulously scraped from verified NGO repositories (Human Rights Watch and the United Nations) to ensure the "ground truth" labels were accurate and expert-approved.

Key Insights and Ideas

  • The Complementarity Gap: The authors found that object-centric models tend to focus on the "tools" of a violation (e.g., a weapon or a specific tool used in child labor), while scene-centric models look at the "setting" (e.g., the makeshift shelters in a displaced persons camp).
  • Context is King: The "aha moment" comes from the realization that human rights violations are rarely defined by a single object. The AI needs to see the interaction between a person and their environment to distinguish a child playing from a child working.
  • Coverage vs. Accuracy: In high-stakes fields like human rights, a "wrong" guess can be dangerous. The authors used a "confidence threshold"—if the AI isn't 85% sure, it refuses to guess. This introduces the metric of coverage: the percentage of images the AI actually feels comfortable labeling.

Key Results

  • Top Performer: The object-centric VGG19 model achieved the highest top-1 accuracy at 35.18%. While this sounds low compared to standard image tasks, it highlights the extreme difficulty of this new benchmark.
  • Fusion Wins: Late fusion (ensembling an object-centric model with a scene-centric one) consistently provided the most balanced results, significantly improving the reliability of the system.
  • VGG vs. ResNet: Interestingly, the older VGG architectures generally outperformed the more modern ResNet50 on this specific dataset, suggesting that VGG’s feature representations might be better suited for this type of transfer learning.

Limitations and Future Work

The authors acknowledge that 3,000 images is still a "small" dataset by modern AI standards. The current models also struggle with "evidence vs. action"—for example, they might correctly identify a school building (the evidence) but fail to understand if that school is currently being targeted by an armed group (the action).

Future Directions: * Multi-labeling: Real-world images often contain multiple violations (e.g., child labor in a displaced persons camp). Future versions of HRA will likely move toward multi-label classification. * Natural Language Descriptions: Moving beyond simple categories to generating full-sentence descriptions of violations to provide more context to investigators. * Web-Demo: The authors have already launched a web-based tool to allow investigators to upload photos and receive real-time classification suggestions.

Summary updated

Abstract

Identifying potential abuses of human rights through imagery is a novel and challenging task in the field of computer vision, which will enable to expose human rights violations over large-scale data that may otherwise be impossible. While standard databases for object and scene categorization contain hundreds of different classes, the largest available dataset of human rights violations contains only four classes. Here, we introduce the human rights archive (HRA) database, a verified-by-experts repository of 3050 human rights violations photographs, labeled with human rights semantic categories, comprising a list of the types of human rights abuses encountered at present. With the HRA dataset and a two-phase transfer learning scheme, we fine-tuned the state-of-the-art deep convolutional neural networks (CNNs) to provide human rights violations classification CNNs. We also present extensive experiments refined to evaluate how well object-centric and scene-centric CNN features can be combined for the task of recognizing human rights abuses. With this, we show that the HRA database poses a challenge at a higher level for the well-studied representation learning methods and provide a benchmark in the task of human rights violations recognition in visual context. We expect that this dataset can help to open up new horizons on creating systems that are able to recognize rich information about human rights violations.
BibTeX
@article{kalliatakis2019exploring,
  author = {Grigorios Kalliatakis and Shoaib Ehsan and Ales Leonardis and Maria Fasli and Klaus D. McDonald-Maier},
  title = {{Exploring Object-Centric and Scene-Centric CNN Features and Their Complementarity for Human Rights Violations Recognition in Images}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2891745},
}
Figure