TL;DR
This paper introduces SCHISM, a novel unsupervised clustering method designed to group images of evolving subjects, like human decomposition, over time. By merging image sequences based on semantic similarity and temporal proximity, SCHISM effectively traces the progression of objects through different stages, outperforming existing methods on a challenging forensic dataset.
What Problem Does This Paper Address?
Analyzing large collections of images that capture the evolution of a subject over time presents a significant challenge for traditional clustering techniques. Standard methods often rely on static image similarity, which fails when subjects change drastically, like a human body progressing from a "fresh" state to a "skeletonized" one. This is particularly problematic in domains like forensic anthropology, where manual labeling of such evolving datasets is costly and time-consuming, and expert annotators are scarce. The paper aims to bridge this gap by developing an unsupervised approach that can automatically group and sequence these evolving images, enabling better analysis and research.
Methods
SCHISM tackles the problem by combining deep learning for feature extraction with a novel sequence merging strategy. The core of the method involves:
- Feature Extraction: Images are first passed through a pre-trained Convolutional Neural Network (CNN), specifically ResNet50, to extract rich feature representations. These features are then dimensionality-reduced to 256 dimensions using Principal Component Analysis (PCA) for efficiency.
- Window-based Sequencing (Snippets): A sliding window approach is employed across timesteps. Within this window, images are compared using cosine similarity. If an image's similarity to another image in a neighboring timestep exceeds a dynamic threshold (which adapts to varying similarities), they are grouped into "snippets" – short sequences of similar images. This dynamic threshold is crucial for handling the gradual changes in appearance over time.
- Stitching Short Sequences: These initial snippets are then "stitched" together to form longer, more comprehensive sequences. This stitching process occurs in three stages:
- Image-overlap stitching: Snippets sharing one or more images are merged.
- Temporal-overlap stitching: Snippets with overlapping time intervals are merged if their average image similarity is above a threshold. This leverages the fact that consecutive timesteps are generally more similar.
- Gap-filling stitching: This stage aims to fill temporal gaps by comparing snippets that might represent missing timesteps, using the average feature vectors of the snippets.
This multi-stage approach allows SCHISM to capture the full evolutionary trajectory of an object, from its initial to its final stages, while minimizing temporal gaps.
Datasets
The paper uses two primary datasets:
- Human Decomposition Dataset: A large, unlabeled dataset of over 1 million images collected over 8 years from the "Body Farm" at the Forensic Anthropology Center. This dataset captures various stages of human decomposition. For evaluation, a subset of 34,476 images from 10 randomly selected subjects was manually labeled.
- MORPH Dataset: A publicly available dataset of mugshots collected over 5 years, containing images of 13,618 subjects. This dataset was used to demonstrate SCHISM's applicability to other temporal evolving image collections, with age serving as the timestep concept.
Key Insights and Ideas
- Temporal Similarity is Key: The core insight is that for evolving subjects, temporal proximity often implies semantic similarity, even if visual appearances differ drastically. SCHISM leverages this by focusing on neighboring timesteps.
- Dynamic Thresholding for Robustness: The use of a dynamic threshold for snippet creation is critical. It allows the method to adapt to the varying degrees of similarity between images at different stages of decomposition, preventing premature or incorrect linking of images.
- Hierarchical Merging for Complete Sequences: The three-stage stitching process is a clever way to build long, coherent sequences. It first groups similar images, then merges overlapping sequences, and finally attempts to fill in missing temporal information, leading to more complete evolutionary traces.
- Unsupervised Approach for Scarce Labels: The unsupervised nature of SCHISM is a major advantage, as it significantly reduces the need for manual annotation in domains where expert labeling is a bottleneck.
Key Results
- Superior Purity: On the human decomposition dataset, SCHISM achieved an average purity of 91.93%, significantly outperforming pre-trained CNN-based image clustering (83.50%) and trained DeepCluster (85.99%) when using the same number of clusters.
- Improved Metrics: SCHISM demonstrated better performance across custom metrics like "essential clusters" (fewer essential clusters indicate more comprehensive sequences) and "inclusion" (higher inclusion means more timesteps captured per class), indicating longer and more complete evolutionary sequences.
- Reduced Gaps: Histograms of gaps showed that SCHISM generated clusters with significantly fewer missing timesteps compared to other methods.
- High Purity on MORPH: SCHISM achieved an impressive 99.87% average purity on the MORPH dataset for 11 subjects, demonstrating its generalizability.
Limitations and Future Work
- Manual Labeling for Evaluation: While the method is unsupervised, evaluation still required manual labeling of a subset of the decomposition data. The authors acknowledge that this metric can only be calculated with known class labels.
- Computational Cost: The paper mentions that increasing the sliding window size (W) can increase computation.
- Future Work: The authors suggest exploring more sophisticated methods for stitching and potentially incorporating explicit semantic constraints if available. They also highlight the potential for SCHISM in other domains with evolving image data.
Summary updated