Creating a Forensic Database of Shoeprints from Online Shoe-Tread Photos

Samia Shafique, Bailey Kong, Shu Kong, Charless C. Fowlkes

Abstract

Shoe-tread impressions are one of the most common types of evidence left at crime scenes. However, the utility of such evidence is limited by the lack of databases of footwear prints that cover the large and growing number of distinct shoe models. Moreover, the database is preferred to contain the 3D shape, or depth, of shoe-tread photos so as to allow for extracting shoeprints to match a query (crime-scene) print. We propose to address this gap by leveraging shoe-tread photos collected by online retailers. The core challenge is to predict depth maps for these photos. As they do not have ground-truth 3D shapes allowing for training depth predictors, we exploit synthetic data that does. We develop a method, termed ShoeRinsics, that learns to predict depth from fully supervised synthetic data and unsupervised retail image data. In particular, we find domain adaptation and intrinsic image decomposition techniques effectively mitigate the synthetic-real domain gap and yield significantly better depth predictions. To validate our method, we introduce 2 validation sets consisting of shoe-tread image and print pairs and define a benchmarking protocol to quantify the quality of predicted depth. On this benchmark, ShoeRinsics outperforms existing methods of depth prediction and synthetic-to-real domain adaptation.

Introduction

Shoeprints are among the most common types of evidence at crime scenes, yet their forensic utility is severely limited by a "database gap." Manually curating a database of every shoe model's tread is an expensive and slow process that cannot keep pace with the thousands of new models released monthly. While online retailers like Zappos or 6pm host high-resolution photos of nearly every shoe on the market, these are 2D images. Matching a crime scene print—which is essentially a 3D impression—requires understanding the 3D geometry (depth) of the tread. The primary challenge lies in the absence of ground-truth 3D scans for these millions of retail photos to train standard supervised models.

The ShoeRinsics Pipeline

To address this, the authors developed ShoeRinsics, a machine learning pipeline designed to predict 3D depth maps from standard 2D retail shoe photos. The architecture leverages synthetic data and intrinsic image decomposition to bridge the "domain gap" between computer-generated renders and real-world photography.

The ShoeRinsics architecture overview, showing the translator, decomposer, and renderer components.

The pipeline consists of several key stages:

  1. Synthetic Supervision: A massive synthetic dataset (syn-train) provides ground-truth for depth, lighting, and material (albedo).
  2. Domain Adaptation: GAN-based translators ($G_{S \rightarrow R}$) stylize synthetic images to match the visual characteristics of real retail photos.
  3. Intrinsic Image Decomposition: The model is tasked with decomposing images into physical components: albedo, depth, surface normals, and lighting. This "physics as a regularizer" approach ensures depth predictions are physically plausible.
  4. Differentiable Rendering: A Renderer ($R$) reconstructs the original image from the predicted components, providing a feedback loop for unsupervised learning on real data.

Datasets and Benchmarking

The study utilizes a combination of synthetic and real-world data to train and validate the system.

Examples from the datasets used: (a) synthetic training data, (b) real retail photos, (c) a new benchmark of shoe-print pairs including athletic, used, and formal shoes, and (d) validation pairs from the FID300 dataset.
  • syn-train: 88,408 synthetic images with full ground-truth annotations (depth, albedo, normals, light).
  • real-train: 3,543 unlabeled shoe-tread photos from online retailers.
  • real-val (New Benchmark): 36 pairs of shoe-tread photos and actual physical prints collected in a lab setting using forensic ink.
  • real-FID-val: 41 pairs derived from the existing FID300 forensic dataset.

Key Results

ShoeRinsics demonstrates significant improvements over existing domain adaptation methods. On the real-val benchmark, it achieved a 49.0% mIoU (mean Intersection-over-Union), significantly outperforming prior state-of-the-art methods like CyCADA (44.8%) and ADDA (41.4%).

A notable finding was the model's ability to generalize. Despite being trained primarily on athletic shoes, it performed remarkably well on formal and worn shoes. Ablation studies confirmed that both the GAN-based translation and the physical reconstruction (Renderer) are critical to the system's success.

Limitations and Future Work

The current model faces challenges with complex materials, such as translucent or highly reflective soles, which deviate from the "Lambertian" (matte) surface assumption used in the synthetic training data. Future research will focus on incorporating more complex material properties into the synthetic data and scaling the pipeline to process millions of retail images, potentially creating a comprehensive search engine for forensic shoeprints.

Summary updated

Abstract

Shoe-tread impressions are one of the most common types of evidence left at crime scenes. However, the utility of such evidence is limited by the lack of databases of footwear prints that cover the large and growing number of distinct shoe models. Moreover, the database is preferred to contain the 3D shape, or depth, of shoe-tread photos so as to allow for extracting shoeprints to match a query (crime-scene) print. We propose to address this gap by leveraging shoe-tread photos collected by online retailers. The core challenge is to predict depth maps for these photos. As they do not have ground-truth 3D shapes allowing for training depth predictors, we exploit synthetic data that does. We develop a method, termed ShoeRinsics, that learns to predict depth from fully supervised synthetic data and unsupervised retail image data. In particular, we find domain adaptation and intrinsic image decomposition techniques effectively mitigate the synthetic-real domain gap and yield significantly better depth predictions. To validate our method, we introduce 2 validation sets consisting of shoe-tread image and print pairs and define a benchmarking protocol to quantify the quality of predicted depth. On this benchmark, ShoeRinsics outperforms existing methods of depth prediction and synthetic-to-real domain adaptation.
BibTeX
@inproceedings{shafique2023creating,
  author = {Samia Shafique and Bailey Kong and Shu Kong and Charless C. Fowlkes},
  title = {{Creating a Forensic Database of Shoeprints from Online Shoe-Tread Photos}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=1QpF3j6Kix},
}
Figure