Temporal Dynamics in Visual Data: Analyzing the Impact of Time on Classification Accuracy

Tom Pégeot, Eva Feillet, Adrian Popescu, Inna Kucher, Bertrand DelezoideWACV 2025

TL;DR

This paper introduces VCT-107, a new dataset of over 950,000 images spanning 14 years, to study how temporal shifts in visual data affect image classification models. They find that even state-of-the-art models experience accuracy degradation when trained on data from one period and tested on another, with human-made objects being more susceptible to these temporal changes.

What Problem Does This Paper Address?

Modern image classification models are typically trained on static datasets that represent a snapshot of visual data at a particular time. However, the real world is dynamic: the appearance of objects, trends, and even the distribution of classes can change over time. This temporal shift can lead to a significant drop in model performance when deployed in real-world scenarios where data distributions evolve. Existing datasets often lack temporal metadata, making it difficult to study and address this crucial problem. This paper aims to bridge this gap by providing a dataset and analysis specifically focused on the impact of temporal dynamics on visual classification accuracy.

Methods

The core of this work involves the creation and analysis of the VCT-107 dataset.

  • Dataset Creation: Images were collected from Flickr over five distinct two-year periods between 2007 and 2020. The dataset comprises 107 classes across nine topics, with careful attention paid to image rights and safety.
  • Preprocessing: Images were preprocessed by computing embeddings using a ViT-B/14 model pre-trained with DinoV2. Near-duplicates were removed, and images were clustered using K-means to facilitate annotation.
  • Annotation: A dedicated labeling interface was used, with participants annotating images based on LLM-generated class definitions. A rigorous process was followed to ensure annotation quality and to select classes with sufficient data.
  • Experimental Setup: The authors employed linear probing, a common technique where a pre-trained feature extractor is frozen, and only a linear classifier is trained. This allows for efficient evaluation of the impact of temporal shifts on the learned representations. They also explored full training and tested various continual learning algorithms.
  • Analysis: The study analyzes accuracy degradation across different temporal periods, investigates the impact of training strategies (e.g., pre-training vs. scratch training), and examines the influence of training set size. They also perform an embedding-based analysis to understand the underlying reasons for accuracy shifts.

Datasets

  • VCT-107 (Visual Classes through Time): A newly introduced dataset containing 951,176 images across 107 classes, collected from Flickr over five temporal periods (2007-2008, 2010-2011, 2013-2014, 2016-2017, 2019-2020). This dataset is unique in its explicit temporal dimension.
  • Existing Datasets (for comparison/context): CIFAR-100, ImageNet, YFCC100M, Google Landmarks, Google Open Images, Mapillary, CORE50, AmsterTime, AgeDB, Wild-Time, Yearbook, FMoW.

Key Insights and Ideas

  • Temporal Shift is a Real Problem: Even powerful, pre-trained models like DinoV2 ViT-B/14 show a noticeable drop in accuracy when tested on data from periods different from their training period. This highlights that "state-of-the-art" doesn't automatically mean "timeless."
  • Human-Made Objects are More Susceptible: Classes representing human-made objects (e.g., cars, buildings, electronics) exhibit larger temporal shifts than natural classes (e.g., animals). This is attributed to faster design evolution, technological advancements, and changing trends in these categories.
  • Pre-training is Crucial, but Not a Silver Bullet: While strong self-supervised pre-training (like DinoV2) significantly improves temporal robustness compared to training from scratch, it doesn't eliminate the accuracy gap entirely.
  • Continual Learning Offers Mitigation: Continual learning algorithms can help reduce the accuracy gap caused by temporal shifts, especially when combined with pre-trained models, offering a way to adapt without storing all historical data.
  • Temporal Distance Matters: The accuracy loss generally increases as the temporal distance between training and testing data grows, reinforcing the idea that models need to adapt to evolving distributions.
  • Embedding Analysis Can Predict Shifts: Distances between class embeddings (e.g., FID, Sinkhorn distance) show a correlation with accuracy loss, suggesting these metrics could be used to identify classes or topics that require more frequent updates.

Key Results

  • Accuracy Drop: Linear probing with DinoV2 ViT-B/14 on VCT-107 shows an accuracy loss when training and testing on different periods. For example, training on 07/08 and testing on 19/20 results in a performance decrease.
  • Pre-training vs. Scratch: Training from scratch on a single period leads to much larger accuracy variations across periods compared to using a pre-trained backbone.
  • DinoV2 vs. Other Backbones: DinoV2 ViT-B/14 generally shows better temporal stability than ResNet-18 pre-trained on ILSVRC or ImageNet-21k, with a maximum accuracy drop of 7.2% compared to 9.0% for ILSVRC pre-trained ResNet-18. CLIP models also showed good robustness, with a maximum loss of 5.5%.
  • Low-Shot Learning: Reducing training samples per class in low-shot settings exacerbates temporal shift issues, but strong pre-training still helps mitigate the relative accuracy loss.
  • Continual Learning Performance: DIL algorithms like RanPAC and FeCAM-n achieved the best accuracy, outperforming naive replay and accumulation strategies while requiring significantly less memory.

Limitations and Future Work

  • Dataset Source: The dataset is sourced from Flickr, which has inherent biases and limitations related to data availability and user demographics. The authors provide URLs, not image files, to respect copyright.
  • Temporal Scale: The dataset uses yearly intervals. Future work could explore finer temporal scales for more granular analysis.
  • Class Dynamics: The current dataset fixes classes over time. An enriched version could include classes that emerge or disappear over time.
  • Multi-label Annotation: The dataset is single-label. Extending it to multi-label annotations could provide insights into class co-occurrence dynamics.
  • Mitigation Strategies: While pre-training and CL were explored, other techniques like domain adaptation and imbalanced learning methods could be investigated further.
  • Broader Topics and Granularity: Expanding the dataset to include more topics and finer-grained classes would allow for a more comprehensive analysis of temporal shifts.

Summary updated

BibTeX
@inproceedings{pegeot2025temporal,
  author = {Tom Pégeot and Eva Feillet and Adrian Popescu and Inna Kucher and Bertrand Delezoide},
  title = {{Temporal Dynamics in Visual Data: Analyzing the Impact of Time on Classification Accuracy}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure