The Growing Strawberries Dataset: Tracking Multiple Objects With Biological Development Over an Extended Period

Junhan Wen, Camiel R. Verschoor, Chengming Feng, Irina-Mona Epure, Thomas Abeel, Mathijs de WeerdtWACV 2024

TL;DR

This paper introduces "The Growing Strawberries Dataset" (GSD), a novel long-term, in-the-wild dataset for Multiple Object Tracking (MOT). GSD challenges existing MOT algorithms by featuring significant biological development, appearance changes, and irregular movements over extended periods, highlighting the limitations of current methods in real-world, dynamic scenarios.

What Problem Does This Paper Address?

Multiple Object Tracking (MOT) is crucial for applications like autonomous driving and surveillance. However, most existing MOT datasets consist of short video clips, failing to capture the long-term variations and developmental changes common in real-world scenarios. This gap makes it difficult to evaluate and develop MOT algorithms for tasks requiring sustained tracking of objects that evolve over time, such as monitoring plant growth, livestock, or medical conditions. The paper specifically addresses the challenge of tracking objects that undergo significant biological development, leading to complex appearance changes and unpredictable movements, which are largely unexplored in current MOT research.

Methods

The paper introduces "The Growing Strawberries Dataset" (GSD) as its primary contribution. The dataset was collected hourly over 16 months (2021-2022) using six cameras (three RGB, three OCN) across two greenhouses, capturing the growth of 24 strawberry plants. This time-lapse approach, with hourly intervals, provides a unique challenge compared to typical MOT datasets that use higher frame rates. The authors also benchmarked several state-of-the-art online MOT algorithms (ByteTrack, OC-SORT, DeepSORT, StrongSORT) on GSD to demonstrate its difficulty. They analyzed the performance based on track identification, association, and overall MOT metrics, highlighting the trade-offs between appearance and motion-based tracking.

Datasets

  • The Growing Strawberries Dataset (GSD): A newly introduced dataset specifically designed for long-term MOT with biological development. It features hourly time-lapse images of strawberry plants over 16 months, encompassing 24 plants across two greenhouses. GSD includes human-annotated bounding boxes and track IDs for individual strawberries.
  • MOT20: Used for comparison to highlight the differences in challenges between GSD and a standard MOT dataset.

Key Insights and Ideas

  • Biological Development is a Major MOT Hurdle: The GSD dataset demonstrates that tracking objects undergoing significant biological changes (like fruit ripening) is substantially harder than tracking static objects or those with predictable movements.
  • Appearance Changes are More Than Just Lighting: While lighting variations are a known challenge, the intrinsic changes in an object's color, shape, and size due to development introduce a deeper level of complexity for appearance-based tracking.
  • Long-Term Tracking Exacerbates Existing Problems: The sparse nature of long-term data collection (hourly in GSD) makes it harder to maintain track continuity and increases the likelihood of identity switches and trajectory interruptions.
  • Current MOT Algorithms Struggle with GSD: Even leading MOT algorithms show a significant performance drop on GSD compared to datasets like MOT20, indicating a need for new approaches tailored to long-term, developmental tracking.
  • Motion vs. Appearance Trade-off is Critical: Adjusting the balance between motion and appearance cues in algorithms like StrongSORT reveals a trade-off: emphasizing motion can reduce identity switches but increase trajectory fragmentation, and vice-versa.

Key Results

  • Performance Degradation: State-of-the-art MOT algorithms exhibited significantly lower performance on GSD compared to MOT20, particularly in metrics like HOTA and IDF1, indicating intense drops in track identification and association.
  • Increased ID Switches and Interruptions: GSD resulted in exaggerated frequencies of identity switches (IDS/Tr) and trajectory interruptions (FM/Tr) for the tested algorithms, suggesting higher track discontinuity.
  • ByteTrack as a Strong Baseline: Among the tested algorithms, ByteTrack showed the best performance on GSD, but still highlighted the need for improvement.
  • Detection vs. Association: The paper suggests that while object detection on GSD is comparable to other datasets, the primary challenge lies in the association stage due to the unique characteristics of the data.
  • Downstream Application Impact: When evaluating GSD's utility for a downstream application like monitoring strawberry ripening (using the A* channel for color change), even the best-performing tracker (ByteTrack) struggled to maintain continuity, showing fragmented tracks and ID switches during crucial developmental transitions.

Limitations and Future Work

  • Limited Scope of Algorithms: The study primarily focused on online MOT algorithms due to the dataset's size and computational demands. Offline solvers were briefly explored but found to be resource-intensive.
  • "Daytime" Subset for Benchmarking: To ensure a fair comparison and mitigate extreme lighting variations, the benchmarking experiments were conducted on a "daytime" subset, excluding "darker images" that might occur during dawn/dusk. This means the full dataset's challenges, including more extreme lighting, were not fully explored in the benchmark.
  • Future Work: The authors call for the development of MOT algorithms that can better handle the challenges presented by GSD, particularly in improving track associations for long-term MOT tasks with sparse frame rates and in-the-wild data imperfections. They also suggest that insights from GSD could be applied to other long-term MOT tasks involving incremental changes, such as cellular growth or corrosion expansion, and that incorporating non-visual spectrum data could further enhance plant science applications.

Summary updated

BibTeX
@inproceedings{wen2024growing,
  author = {Junhan Wen and Camiel R. Verschoor and Chengming Feng and Irina-Mona Epure and Thomas Abeel and Mathijs de Weerdt},
  title = {{The Growing Strawberries Dataset: Tracking Multiple Objects With Biological Development Over an Extended Period}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure