The Erosion of URL-Based Machine Learning Benchmarks

17 concepts17 papers1 proposal

Modern machine learning thrives on scale, but scale often relies on the open web. To bypass copyright restrictions, navigate paywalls, and avoid massive hosting costs, dataset creators frequently distribute benchmarks as collections of external URLs rather than raw media files.

However, this creates a critical vulnerability: the web is volatile. This phenomenon, known as link rot or dataset decay, means that over time, users delete videos, change privacy settings, or take down websites. When a dataset relies on these external links, it undergoes a gradual, irreversible erosion.

[Dataset Release] ---> 100% Data Available (Baseline Evaluation)
       |
       v (Time Passes: Deletions, Privacy Changes, Expired Domains)
       |
[1 Year Later]   ---> 90% Data Available  (Incomplete Replication)
       |
       v
[3 Years Later]  ---> 80% Data Available  (Biased Benchmarking)

This digital decay is not just an administrative nuisance; it undermines the core foundation of empirical machine learning: reproducibility.


Why Dataset Decay Silently Breaks Science

When a benchmark shrinks, it introduces a subtle but severe form of evaluation bias. If a model developed in 2020 was evaluated on 100% of a dataset, and a model developed in 2024 is evaluated on only 80% of the remaining data, a direct comparison between their accuracy scores is no longer scientifically valid.

The missing 20% is rarely missing at random. Content that disappears from the web often correlates with specific user demographics, geographic regions, or content types, meaning the benchmark's underlying data distribution shifts over time.

The scale of this problem spans multiple modalities:

  • Video and Audio Platforms: Video-sharing platforms are particularly susceptible to rapid decay. In a comprehensive survey, Gijs Wijngaard et al. found that nearly one-fifth of original samples in video-sourced audio-language datasets become unavailable over time due to deletions or privacy changes. This volatility directly impacts tasks like dense video captioning (Vladimir Iashin et al.), sign language recognition (Dongxu Li et al.), and video highlight detection (Hang Wang et al.), where shrinking training sets make it difficult to maintain consistent evaluation standards.
  • Web-Curated Images: Social media platforms suffer from similar fragility. For instance, the RedCaps dataset (Karan Desai et al.) curates millions of image-text pairs from Reddit. Because it is distributed via URLs to respect user data ownership, it faces gradual data loss whenever users delete their posts or images. As Laurent Mertens et al. note in their work on emotion recognition, relying on third-party image hosting creates a structural dependency that threatens the long-term accessibility of computer vision research.
  • Text and News Media: In natural language processing, cross-lingual and news-based datasets frequently break down. Xi Chen et al. and Joseph Hajjar et al. demonstrate that distributing news article datasets via URLs leads to severe data loss as articles are moved behind paywalls or domains expire. This prevents subsequent researchers from accessing the complete original training sets, impacting model robustness.
  • Web Security Data: The issue even compromises security benchmarks. In an analysis of phishing landscapes, Sidharth Chhabra et al. observed that a large majority of URLs in historical phishing datasets go offline rapidly, making it difficult to verify the original status assigned to those links or replicate the security analyses derived from them.

Emerging Strategies to Halt the Rot

To ensure benchmarks remain permanent and reproducible, researchers are shifting away from raw URL distribution toward architectures and curation methods designed for permanence.

1. Static, Anonymized Hosting

The most direct solution to link rot is to eliminate the link entirely and host the raw media. To do this without violating privacy or copyright, the data must be heavily processed. For example, the AViD dataset (A. J. Piergiovanni et al.) provides a massive video action recognition benchmark by completely anonymizing faces and directly hosting the clips. This creates a "static" alternative that bypasses the volatility of platforms like YouTube.

2. Feature-Space Distribution

If copyright or bandwidth prevents the distribution of raw media files (such as commercial music tracks), creators can distribute precomputed model embeddings instead of source URLs.

Traditional Pipeline (Volatile):
[Dataset of URLs] ---> Download Media (Fails if Link Rots) ---> Extract Features ---> Train Model

Feature-Space Pipeline (Robust):
[Dataset of Precomputed Embeddings] -------------------------------------------------> Train Model

In the DISCO-10M music dataset, Emil Funke et al. leverage this strategy by providing precomputed CLAP embeddings for over 15 million tracks. Even if the original audio links disappear from third-party platforms, the mathematical representation of the audio remains fully intact and usable for training downstream models (Aurian Quelennec et al.).

3. Transmutation into Text Scripts

For multimodal tasks, another alternative is to transmutate volatile media into highly descriptive, permanent text. In the ExFunTube dataset, Dayoon Ko et al. address the ephemeral nature of short-form videos by converting video content into rich, fine-grained text scripts—including speaker-diarized transcripts, visual descriptions, and sound tags. This ensures that even if the source video is deleted, the semantic context is preserved for language models to analyze.

4. Time-Aware and Incremental Training

When datasets inevitably change or visual concepts evolve over time, models must be built to adapt rather than break. Frédéric Lin et al. demonstrate that static pretraining degrades when faced with temporal shifts and link inactivity. They propose time-incremental pretraining strategies as a mitigation framework, acknowledging that datasets are living, shifting entities rather than frozen snapshots.

Key Foundational Work

Multi-modal Dense Video Captioning (2020) — contributed 1 concept elements to this direction.
Transferring Cross-Domain Knowledge for Video Sign Language Recognition (2020) — contributed 1 concept elements to this direction.
AViD Dataset: Anonymized Videos from Diverse Countries (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_dataset_link_rot_480e8d58,
  title = {dataset link rot},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dataset-link-rot/rdr_480e8d5801a048edaf13c748b689fca5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_dataset_link_rot_480e8d58_v1,
  title = {dataset link rot},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dataset-link-rot/rdr_480e8d5801a048edaf13c748b689fca5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake