TraffickCam: Explainable Image Matching For Sex Trafficking Investigations.

Abby Stylianou, Richard Souvenir, Robert PlessarXiv 2019 · OpenReview

TL;DR

TraffickCam is a specialized image-matching system designed to help law enforcement identify the hotel rooms where sex trafficking victims are photographed. This paper introduces improved deep metric learning techniques that double search accuracy and novel visualization tools that explain why the AI suggests a match, transforming a "black box" algorithm into a transparent investigative tool.

What Problem Does This Paper Address?

In sex trafficking investigations, photographs of victims are often the only clues to their location. Identifying the specific hotel room in these photos can help verify victim movement and predict where traffickers might operate next.

However, this is a massive "needle in a haystack" problem. Hotel rooms are notoriously difficult for AI to categorize: two rooms in the same hotel might look completely different (e.g., a bedroom vs. a bathroom), while two different hotel chains might use identical carpets or lamps. Furthermore, law enforcement officers cannot simply trust a "black box" result; they need to understand the reasoning behind a match to verify it, use it in reports, and potentially present it as evidence.

Methods

The authors built upon the existing TraffickCam system, focusing on three main areas:

  1. Improved Metric Learning: The system uses a ResNet-50 architecture to map images into a high-dimensional feature space. Previously, they used a "batch-all" loss function that forced every image from a single hotel to map to the same point. The authors found this forced the network to "memorize" disparate viewpoints, hurting its ability to generalize. They switched to "Easy Positive" triplet loss, which only forces the network to match the most similar images from a hotel, leading to better performance.
  2. Explainable Visualizations: To move beyond simple similarity scores, the authors developed two visualization techniques:
  3. Importance Maps: These highlight which 7x7 regions of an image contributed most to the similarity score.
  4. PCA-based Color Coding: By running Principal Component Analysis (PCA) on the feature vectors of a pair of images and mapping the top three components to RGB colors, the system creates a visual "map." If the curtains in the query image and the curtains in the result image are both colored red, it indicates the AI sees them as having similar features.
  5. Investigator Infrastructure: They developed a web platform that allows investigators to mask sensitive content (like victims' faces) before searching and a tool to generate formal reports for legal documentation.
Figure 1: The TraffickCam system consists of a smartphone application (a) to collect relevant imagery of hotel rooms to support investigations of human trafficking. To date, the TraffickCam image database (b) consists of over 3 million hotel room images collected from this application and other publicly available sources of hotel room photos. Deep met- ric learning is used to convert these images into a searchable index of image features (c). This index supports a law en- forcement platform where investigators can upload masked off images from trafficking investigations (d) to retrieve the most similar images in the TraffickCam database (e).

Datasets

  • TraffickCam Database: A massive collection of over 3 million hotel room images sourced from a public smartphone app and other online travel sites.
  • Hotels-50K: A large-scale, publicly available dataset specifically designed for the hotel identification problem, used here to benchmark improvements in search accuracy.

Key Insights and Ideas

  • Generalization over Memorization: Forcing an AI to treat a photo of a hotel toilet and a hotel bed as "the same" (because they are in the same building) is counterproductive. By allowing the network to focus on matching similar elements (the "Easy Positive" approach), the system becomes much more robust.
  • The Power of "Why": The PCA visualization is a standout feature. It allows an investigator to see if the AI is matching a room based on a unique headboard (useful) or just because both photos have white pillows (less useful).
  • Human-in-the-Loop Design: The authors emphasize that the AI isn't meant to replace the investigator. Instead, the tool is designed to provide "intuition for how the process works," allowing the human to make the final, informed call on whether a match is legitimate.
The investigator platform allows for masking sensitive areas and viewing multiple images from a candidate hotel.

Key Results

  • Accuracy Boost: The shift to "Easy Positive" triplet loss improved the "top-1" accuracy (the correct hotel appearing as the very first result) from 8% to 16% on training data. While 16% sounds low, it represents a 4,000x improvement over random chance in a database of 50,000 hotels.
  • Qualitative Verification: As seen in Figure 3, the PCA visualization successfully identifies matching curtains, floors, and sinks. Crucially, it also reveals errors—such as when the network incorrectly maps linens in a query to a carpet in a result—giving investigators a reason to be skeptical of certain matches.
Visualization results showing importance maps (b) and PCA color-coding (c) for correct and incorrect matches.

Limitations and Future Work

  • Shared Design Elements: The system still struggles when different hotels use the same furniture or decor, leading to high similarity scores for incorrect matches.
  • Human Perception Gap: An automated search might be effective at finding a match, but if the matching features aren't obvious to a human investigator (e.g., a specific wood grain pattern), the investigator might dismiss a correct result.
  • Next Steps: The authors plan to more deeply integrate these visualization tools into the live search interface and conduct formal studies to see how these explanations affect the speed and accuracy of real-world investigations.

Summary updated

Abstract

Investigations of sex trafficking sometimes have access to photographs of victims in hotel rooms. These images directly link victims to places, which can help verify where victims have been trafficked or where traffickers might operate in the future. Current machine learning approaches give promising results in image search to find the matching hotel. This paper explores approaches to make this end-to-end system better support government and law enforcement requirements, including improved performance, visualization approaches that explain what parts of the image led to a match, and infrastructure to support exporting the results of a query.
BibTeX
@misc{stylianou2019traffickcam,
  author = {Abby Stylianou and Richard Souvenir and Robert Pless},
  title = {{TraffickCam: Explainable Image Matching For Sex Trafficking Investigations.}},
  year = {2019},
  eprint = {1910.03455},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.03455},
}
Figure