Fifty-Two Shades of Pig: What the UCR Archive Really Asks Classifiers to Do

Omar Swelam, Jaris Küken, Shi Bin Hoo, Frank Hutter, Lennart PuruckerarXiv 2026 · arXiv

TL;DR

The UCR Archive is the gold-standard benchmark for Time Series Classification (TSC), but this provenance audit reveals that many of its tasks are "pretext" problems—like identifying which specific pig produced a heartbeat rather than detecting if the pig is ill. With only 30% of the archive representing real-world temporal tasks and significant data leakage found in a third of the datasets, high benchmark scores may reflect a model's ability to exploit recording artifacts rather than its readiness for actual deployment.

What Problem Does This Paper Address?

In the world of machine learning, benchmarks are the compass that points toward "progress." For researchers working on Time Series Classification (TSC), that compass is almost always the UCR Archive. However, the authors argue that a benchmark only drives meaningful progress if its tasks resemble the problems encountered in practice.

The paper tackles a growing suspicion in the field: that the UCR Archive has become a "convenience-driven" collection rather than a representative one. If a model ranks first on the archive by exploiting specific sensor noise or memorizing subject identities, does that actually mean it’s a better classifier for clinical monitoring or satellite sensing? To answer this, the authors performed a deep-dive "provenance audit" to see where these 128 datasets actually came from and what they are really asking models to do.

Methods

The authors developed a rigorous Provenance Audit Protocol to trace each dataset through three stages of its lifecycle: the original source study, the construction of the archive task, and the evaluation (train/test) split.

To handle the scale of 128 datasets, they employed a novel five-stage pipeline involving Language Model (LLM) assistance:

  1. Scaffolding: Automated scripts to scrape basic metadata.
  2. Agentic Drafting: LLMs read source publications to fill in audit fields, citing specific evidence.
  3. Agentic Review: A second, independent LLM pass to verify claims and quotes.
  4. Fact-check: Two more LLMs (Claude Opus and GPT Astra) re-derived verdicts from the records.
  5. Human Verification: The authors personally reviewed every case where the AI agents disagreed or expressed low confidence.

The audit evaluated datasets across six fields, including whether the task was "real" or "toy," whether the data was truly temporal, and whether the train/test splits were independent.

Figure 3: The six audit fields, two per stage of a dataset’s life. Green marks the value that passes, red a value that fails or flags leakage, grey a value the verdict does not read.

Datasets

The study focuses exclusively on the 128 univariate datasets in the UCR Time Series Classification Archive. While it mentions the multivariate UEA archive and newer collections like MONSTER, the UCR archive remains the primary target due to its status as the de facto standard in TSC research.

Key Insights and Ideas

  • The "Pretext Task" Problem: Many datasets use "Instance Identity" as a label. For example, the PigAirwayPressure dataset was originally created to detect hemorrhage (bleeding). In the archive, however, the task is redefined to identify which of 52 specific pigs produced the trace. This is a "pretext" task: it tests if a model can recognize a specific source, not if it can solve the underlying medical problem.
  • Realism is Rare: Only 38 out of 128 datasets were identified as "real temporal tasks"—meaning they answer a source question, use data sampled over time, and show no evidence of leakage.
  • Non-Temporal "Time Series": About 38% of the archive consists of ordered sequences that aren't actually temporal, such as image outlines (contours) or spectroscopy (wavelengths). The authors argue that treating these as "time series" assumes a transfer of capability that the archive never actually tests.
  • The Pre-segmentation Trap: Most UCR datasets are pre-aligned and pre-segmented. This turns time steps into features, allowing "tabular" classifiers (which ignore order) to perform as well as sophisticated sequence models.
Figure 1: We show the archive datasets distribution over the different realism criterion. Our audit identifies only 38/128 UCR datasets as real temporal tasks; about a quarter of these are prone to split leakage. We indicate the proportion of datasets failing our leakage criteria (white boxes) and with no leakage found (black boxes) within each category, although most splits cannot be checked (53 of the 76 real-task datasets are Unclear).

Key Results

  • Task Construction Failures: For 52 of the 128 datasets, the classification target is not a real-world prediction problem. These include "toy" tasks (17), metadata used as labels (13), and redefined tasks (13).
  • Widespread Leakage: 41 datasets (32%) were flagged for "Concern" regarding train/test leakage. This includes 17 datasets with exact duplicates across the split.
  • Performance Inflation: The authors demonstrated that removing duplicates significantly drops the accuracy of top-tier models like MiniRocket on several datasets, proving that "state-of-the-art" performance is sometimes just a result of memorization.
Figure 2: MiniRocket accuracy with the origi- nal training set (with duplicates, grey) and af- ter removing the duplicates (blue). On several datasets, duplication inflates test accuracy.

Limitations and Future Work

The authors acknowledge that their leakage counts are likely "lower bounds." For 65 datasets, the information needed to check for shared subjects or sessions is simply missing from the documentation.

The paper concludes with a call to action for the TSC community:

  1. Move beyond aggregate scores: Stop interpreting a single average score across the whole archive as a measure of "deployment readiness."
  2. Provenance-aware evaluation: Future benchmarks should prioritize "real" tasks and use evaluation protocols (like subject-disjoint splits) that reflect how models will actually be used.
  3. New Benchmarks: There is an urgent need for more realistic, multivariate, and unsegmented time series benchmarks to close the gap between research and reality.

Summary updated

Abstract

The pigs were bleeding. That was the point of the experiment: could airway pressure reveal hemorrhage before it became critical? In the UCR Time Series Classification Archive, the de facto benchmark for Time Series Classification (TSC), the same recordings ask a different question: "which of 52 pigs produced this trace?". The dataset PigAirwayPressure is not an outlier. Although the archive is the standard benchmark, whether its tasks resemble the problems TSC is meant to solve has rarely been examined. We find that several datasets are repurposed from their original task, built from toy setups, or have flawed test splits. We further question how representative the archive's ordered non-temporal sequences (e.g. spectra, shape outlines) are of real TSC, and the potential downsides from the decomposition of multivariate problems. Our findings call for more careful interpretation of archive-wide scores and greater attention to deployment-relevant evaluation. The audit is at https://github.com/automl/ucr-provenance-audit.
BibTeX
@misc{swelam2026fifty,
  author = {Omar Swelam and Jaris Küken and Shi Bin Hoo and Frank Hutter and Lennart Purucker},
  title = {{Fifty-Two Shades of Pig: What the UCR Archive Really Asks Classifiers to Do}},
  year = {2026},
  eprint = {2611.09843},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.09843},
}
Figure