Supporting Historical Photo Identification with Face Recognition and Crowdsourced Human Expertise (Extended Abstract)

Vikram Mohanty, David Thames, Sneha Mehta, Kurt LutherIJCAI 2020 · DOI

TL;DR

This paper introduces Photo Sleuth, a web platform that combines automated face recognition with crowdsourced human expertise to identify individuals in historical photographs, specifically focusing on American Civil War soldiers. By creating a novel pipeline that leverages user-generated tags and military records to narrow down search results before applying face recognition, Photo Sleuth significantly aids users in identifying previously unknown portraits.

What Problem Does This Paper Address?

Identifying individuals in historical photographs is crucial for understanding cultural heritage, correcting historical records, and even assigning economic value. However, this task is incredibly challenging due to the manual, time-consuming nature of traditional methods, which often involve sifting through vast archives of low-quality images and reference materials. While automated face recognition exists, it often struggles with the unique challenges of historical photos (e.g., low resolution, fading, monochrome) and is not widely adopted by experts. This paper addresses the gap by proposing a system that intelligently combines human domain expertise with AI to make historical photo identification more accessible and effective.

Methods

The core of Photo Sleuth is its novel person identification pipeline, designed to overcome the limitations of purely automated or purely manual approaches. The pipeline consists of three main stages:

  1. Building the Haystack: This involves creating a comprehensive database of historical photos. Users can upload unidentified portraits, and the system uses Microsoft Azure's Face API to detect faces. Crucially, users are encouraged to tag photos with metadata (format, inscriptions) and visual clues (uniform details, insignia). These tags are then mapped to military service information.
  2. Narrowing Down the Haystack: This is where the system intelligently filters potential matches. Instead of relying solely on face similarity, Photo Sleuth uses the user-provided visual tags to generate search filters based on detailed military service records (e.g., branch, rank). This significantly reduces the candidate pool. Facial similarity filtering via Azure's Face API is then applied within this narrowed space.
  3. Finding the Needle: The search results page displays potential matches, sorted by facial similarity. Users can then manually inspect these candidates, viewing military record highlights and comparing photos side-by-side. The system deliberately hides confidence scores to prevent bias. Users can then confirm an identification, attributing it to themselves.

Datasets

The paper mentions that Photo Sleuth's initial reference database contained over 20,000 identified Civil War soldier portraits from public sources like the US Army Military History Institute and the US Library of Congress. This is noted as a small fraction of the estimated four million Civil War portraits that exist. The evaluation used user-uploaded photos from the first month of the platform's public launch (2012 photos from 612 registered users).

Key Insights and Ideas

  • Human-AI Synergy is Key: The paper strongly advocates for a hybrid approach where AI (face recognition) acts as a powerful filter and sorter, but human expertise and judgment are essential for the final identification. This acknowledges the limitations of current AI in nuanced historical contexts.
  • Leveraging Domain-Specific Metadata: The innovative use of user-generated tags (visual clues, metadata) to query detailed military records is a significant contribution. This allows for highly specific filtering that goes beyond simple facial similarity.
  • Crowdsourcing as a Sustainable Model: By attributing user contributions and building a growing database of identified portraits, Photo Sleuth fosters a sustainable model for crowdsourced historical research.
  • The "Last Mile" Problem: The evaluation highlights that even with AI, the correct match doesn't always appear at the top of search results. This underscores the importance of human diligence in carefully reviewing a curated set of candidates.
  • Preventing Misinformation: The requirement for users to follow the entire workflow and provide sources for identification helps maintain data integrity and prevent the spread of incorrect information.

Key Results

  • During its first month of public launch, Photo Sleuth facilitated the identification of dozens of unknown portraits.
  • Of the 560 identified photos, 119 were "post-identified" (newly identified by users), matching 88 soldiers with existing reference photos.
  • An expert Civil War photo historian reviewed the post-identified photos and found that the majority of user-generated identifications were positive matches, particularly for photos with inscriptions or replicas.
  • The system's search workflow was successfully used by users, with most providing tags even though it was optional.

Limitations and Future Work

  • Automated Identification: The authors deliberately chose not to allow the system to directly identify photos, despite user requests, due to the potential dangers of fully automated approaches. This means the system relies on human confirmation.
  • Face API Limitations: The paper notes that while Azure's Face API yields near-perfect recall at a 0.50 similarity threshold, its precision is poor, leading to many similar-looking false positives.
  • Future Work: The authors mention developing an extension called "Second Opinion" that leverages cognitive science principles to help novice crowds systematically analyze search results. They also aim to build a more sustainable model for volunteer contributions, foster original research, prevent misinformation, and enhance the accuracy of person identification systems.

Summary updated

BibTeX
@inproceedings{mohanty2020supporting,
  author = {Vikram Mohanty and David Thames and Sneha Mehta and Kurt Luther},
  title = {{Supporting Historical Photo Identification with Face Recognition and Crowdsourced Human Expertise (Extended Abstract)}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/660},
}
Figure