Creating a Forensic Database of Shoeprints from Online Shoe Tread Photos

Samia Shafique, Bailey Kong, Shu Kong, Charless C. Fowlkes

Abstract

Shoe tread impressions are one of the most common types of evidence left at crime scenes. However, the utility of such evidence is limited by the lack of databases of footwear prints that cover the large and growing number of distinct shoe models. Moreover, the database is preferred to contain the 3D shape, or depth, of shoe-tread photos so as to allow for extracting shoeprints to match a query (crime-scene) print. We propose to address this gap by leveraging shoe-tread photos collected by online retailers. The core challenge is to predict depth maps for these photos. As they do not have ground-truth 3D shapes allowing for training depth predictors, we exploit synthetic data that does. We develop a method termed ShoeRinsics that learns to predict depth by leveraging a mix of fully supervised synthetic data and unsupervised retail image data. In particular, we find domain adaptation and intrinsic image decomposition techniques effectively mitigate the synthetic-real domain gap and yield significantly better depth prediction. To validate our method, we introduce 2 validation sets consisting of shoe-tread image and print pairs and define a benchmarking protocol to quantify the quality of predicted depth. On this benchmark, ShoeRinsics outperforms existing methods of depth prediction and synthetic-to-real domain adaptation.

Introduction

In forensic science, shoeprints are a critical source of evidence. Identifying a specific shoe model from a crime scene print can provide vital leads. However, the sheer volume of new shoe models released monthly makes maintaining a manual database nearly impossible. While online retailers like Zappos host high-resolution photos of shoe treads, these are 2D images. To be useful for forensic matching, investigators need the 3D geometry (depth) of the tread to simulate how a shoe would leave a mark on various surfaces.

The primary obstacle is the lack of "ground truth" 3D data for the millions of retail photos available online. Without this data, standard supervised learning models cannot be trained. This paper introduces ShoeRinsics, a deep learning framework designed to bridge the gap between synthetic data (where 3D geometry is known) and real-world retail imagery.

The ShoeRinsics Framework

The ShoeRinsics pipeline is designed to overcome the domain gap between synthetic and real data using a combination of domain adaptation and intrinsic image decomposition.

The ShoeRinsics pipeline leverages synthetic data and intrinsic image decomposition to predict depth from retail photos.

Synthetic-to-Real Translation

The framework begins with a synthetic dataset where depth, albedo, and lighting are known. A "Translator" module, utilizing a CycleGAN-like architecture, learns to map synthetic images into the real-world retail image domain and vice versa. This allows the model to learn features that are invariant to the specific "look" of synthetic vs. real images.

Intrinsic Image Decomposition

Rather than directly predicting depth, the model acts as a Decomposer. It breaks down an input image into its fundamental physical components: albedo (base color), surface normals, lighting, and depth. This approach uses physics as a regularizer; by forcing the model to account for lighting and color, the resulting depth predictions are more physically plausible and less prone to the "hallucinations" seen in simpler depth-prediction networks.

The Renderer Loop and Pseudo-Supervision

To ensure accuracy on unlabeled real images, a Renderer module attempts to reconstruct the original image from the predicted components. A low reconstruction error suggests the model has correctly captured the underlying geometry. Additionally, the authors employ "pseudo-albedo" supervision using mean-shift clustering to provide a rough guide for the model when training on real, unannotated retail photos.

Datasets and Benchmarking

To train and validate the model, the researchers utilized a mix of existing and newly created datasets.

Overview of the datasets used for training and validation, including synthetic data and real-world shoe-print pairs.
  • syn-train: A large-scale synthetic dataset containing over 88,000 shoe-treads with perfect ground-truth data.
  • real-train: Over 3,500 unannotated shoe-tread images scraped from online retailers.
  • real-val: A new, high-quality validation set consisting of 36 shoe pairs. The authors used a block-printing technique with relief ink to capture actual physical prints, providing a real-world benchmark for depth-to-print conversion.
  • real-FID-val: 41 shoe-tread images matched with prints from the existing FID300 forensic dataset.

Results and Insights

ShoeRinsics demonstrates a significant improvement over previous state-of-the-art methods. On the real-val benchmark, it achieved a 49.0% mean Intersection-over-Union (mIoU), a substantial jump from the 30.4% achieved by the previous best intrinsic decomposition method.

Key findings include:

  • Local Texture Focus: Training on small 128x128 image patches improved performance by forcing the model to learn local tread textures rather than just the global shoe silhouette.
  • Generalization: The model performed well on shoe categories not present in the training set, such as formal shoes, indicating it learned general geometric principles.
  • Adaptive Thresholding: To evaluate depth maps against 2D prints, the authors developed an adaptive thresholding method that simulates how a shoe flattens under weight, which proved more effective than simple global thresholding.

Conclusion

ShoeRinsics provides a scalable way to build a forensic database from readily available online photos. While the model currently struggles with complex materials like translucent or highly reflective soles, it represents a major step forward in forensic footwear identification. Future work will focus on incorporating these complex materials and addressing "acquired characteristics"—the unique wear and tear patterns that distinguish individual shoes.

Summary updated

Abstract

Shoe tread impressions are one of the most common types of evidence left at crime scenes. However, the utility of such evidence is limited by the lack of databases of footwear prints that cover the large and growing number of distinct shoe models. Moreover, the database is preferred to contain the 3D shape, or depth, of shoe-tread photos so as to allow for extracting shoeprints to match a query (crime-scene) print. We propose to address this gap by leveraging shoe-tread photos collected by online retailers. The core challenge is to predict depth maps for these photos. As they do not have ground-truth 3D shapes allowing for training depth predictors, we exploit synthetic data that does. We develop a method termed ShoeRinsics that learns to predict depth by leveraging a mix of fully supervised synthetic data and unsupervised retail image data. In particular, we find domain adaptation and intrinsic image decomposition techniques effectively mitigate the synthetic-real domain gap and yield significantly better depth prediction. To validate our method, we introduce 2 validation sets consisting of shoe-tread image and print pairs and define a benchmarking protocol to quantify the quality of predicted depth. On this benchmark, ShoeRinsics outperforms existing methods of depth prediction and synthetic-to-real domain adaptation.
BibTeX
@inproceedings{shafique2023creating,
  author = {Samia Shafique and Bailey Kong and Shu Kong and Charless C. Fowlkes},
  title = {{Creating a Forensic Database of Shoeprints from Online Shoe Tread Photos}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=1QpF3j6Kix},
}
Figure