Unraveling the Complexity of Splitting Sequential Data: Tackling Challenges in Video and Time Series Analysis

Diego Botache, Kristina Dingel, Rico Huhnstock, Arno Ehresmann, Bernhard SickarXiv 2023 · DOI · arXiv

Abstract

Standard machine learning practices like random 80/20 data splitting often fail when applied to sequential data because they ignore inherent temporal dependencies. This paper proposes a conceptual four-pillar framework—Acquisition, Representation, Split Ratio, and Quality Criteria—to guide researchers in building more reliable models for complex video and time-series tasks. By addressing issues like temporal leakage and non-i.i.d. distributions, the framework ensures that models generalize better to real-world scenarios.

Introduction: The Failure of Random Splitting

In traditional machine learning, data points are typically assumed to be independent and identically distributed (i.i.d.). Under this assumption, shuffling a dataset and splitting it into training and testing sets (e.g., an 80/20 split) is a robust way to evaluate performance. However, this approach collapses for sequential data, such as video feeds or sensor logs from a motor.

Sequential data points are highly correlated; the state at time $t$ is deeply linked to time $t-1$. Randomly shuffling these points leads to "temporal leakage," where a model effectively "sees the future" during training. This results in over-optimistic performance metrics that do not hold up in production. Conversely, a simple chronological split may fail to capture critical regimes or rare events that only appear in specific segments of the sequence, leading to poor generalization.

The Four-Pillar Framework

To address these challenges, the authors propose a structured approach to data splitting, broken down into four essential stages that form a conceptual roadmap for sequential data processing.

The four-pillar framework for sequential data splitting, illustrating the flow from data acquisition to quality assessment and selection strategy.
  1. Data Acquisition / Selection (I): This stage involves defining the "operational area" of the database. The goal is to ensure the dataset is diverse enough to cover all relevant scenarios, such as varying motor speeds or lighting conditions in video.
  2. Representation (II): Researchers must choose appropriate sequence lengths, features, and periodicities. The choice of "window size" in sliding window approaches is a critical hyperparameter that determines the temporal patterns the model can perceive.
  3. Split Ratio (III): Rather than a blind percentage-based split, the authors argue that training, testing, and validation sets must all contain representatives of relevant scenarios to prevent bias and ensure the test set is truly representative.
  4. Quality Criteria (IV): This stage introduces statistical tests and distance measures to assess the similarity between data splits before training. This ensures that the test set provides a fair and sufficiently challenging evaluation of the model's capabilities.

Case Studies

The framework's utility is demonstrated through two distinct real-world technical use cases:

  • Particle Tracking in Liquids (Video): High-frame-rate microscopic video data is used to track particles in lab-on-a-chip systems. The high redundancy between consecutive frames makes traditional splitting difficult, as nearly identical images might end up in both training and test sets.
  • Motor Test Benches (Time Series): High-dimensional sensor data is used for fault detection. These datasets are often characterized by "rare events"—faults are infrequent, making it difficult to maintain a functional split ratio that includes enough failure states in both sets.

Key Insights and Discussion

A central theme of the paper is the "i.i.d. trap." Because sequential data violates independence assumptions, standard cross-validation often yields misleading results. The authors advocate for using similarity as a metric to identify "unexploited areas" in the data distribution. If the test set is statistically too divergent from the training set, the model is likely to fail.

Furthermore, the framework supports a closed-loop experimentation strategy. As shown in the framework diagram, if the "Quality Criteria" stage identifies a gap—such as a lack of fast-moving particle examples—the system can trigger new simulations or experiments to fill that specific gap, creating an active learning loop that improves the dataset over time.

Conclusion and Future Work

The proposed four-pillar framework formalizes the path toward a comprehensive splitting strategy for sequential data. By applying these principles, researchers can avoid common pitfalls like data redundancy in video tracking and the neglect of rare events in motor monitoring.

While powerful, the framework currently assumes that labeling is accurate and that new data can be obtained if gaps are found. Future research will focus on defining specific similarity measurements that generalize across different sequential data types and automating the selection of quality criteria to further robustify the machine learning lifecycle.

Summary updated

BibTeX
@misc{botache2023unraveling,
  author = {Diego Botache and Kristina Dingel and Rico Huhnstock and Arno Ehresmann and Bernhard Sick},
  title = {{Unraveling the Complexity of Splitting Sequential Data: Tackling Challenges in Video and Time Series Analysis}},
  year = {2023},
  eprint = {2307.14294},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.14294},
}
Figure