AutoDataset: A Lightweight System for Continuous Dataset Discovery and Search

Junzhe Yang, Xinghao Chen, Yunuo Liu, Zhijing Sun, Wenjin Guo, Xiaoyu Shen

Abstract

The continuous expansion of task-specific datasets has become a major driver of progress in machine learning. However, discovering newly released datasets remains difficult, as existing platforms largely depend on manual curation or community submissions, leading to limited coverage and substantial delays. To address this challenge, we introduce AutoDataset, a lightweight, automated system for real-time dataset discovery and retrieval. AutoDataset adopts a paper-first approach by continuously monitoring arXiv to detect and index datasets directly from newly published research. The system operates through a low-overhead multi-stage pipeline. First, a lightweight classifier rapidly filters titles and abstracts to identify papers releasing datasets, achieving an F1 score of 0.94 with an inference latency of 11 ms. For identified papers, we parse PDFs with GROBID and apply a sentence-level extractor to extract dataset descriptions. Dataset URLs are extracted from the paper text with an automated fallback to LaTeX source analysis when needed. Finally, the structured records are indexed using a dense semantic retriever, enabling low-latency natural language search. We deploy AutoDataset as a live system that continuously ingests new papers and provides up-to-date dataset discovery. In practice, it has been shown to significantly reduce the time required for researchers to locate newly released datasets, improving dataset discovery efficiency by up to 80%.

Introduction

In the fast-moving world of machine learning, datasets are the essential fuel for progress. However, as the volume of research explodes—with a nearly 95-fold increase in papers introducing new datasets since 2014—finding the right benchmarks has become a manual, time-consuming chore. Traditional platforms like Hugging Face or Papers with Code often suffer from lag times and inconsistent metadata due to their reliance on manual community submissions. AutoDataset addresses these issues by treating the research paper as the "ground truth," monitoring preprint servers like arXiv directly to find datasets the moment they are announced.

System Architecture

AutoDataset uses a modular, four-stage pipeline designed for speed and low operational cost, moving from broad filtering to precise semantic indexing.

The four-stage AutoDataset workflow: (a) Dataset detection, (b) Description extraction, (c) Download link extraction, and (d) Retrieval.
  1. Dataset Detection (BERT-GATE): Instead of processing every PDF with heavy models, the system runs a lightweight binary classifier on titles and abstracts. This "gatekeeper" identifies papers likely to introduce a dataset in just 11ms.
  2. Description Extraction (BERT-DESC): For papers that pass the gate, the system parses the text using GROBID. A fine-tuned BERT model identifies specific sentences describing the dataset using a class-conditional sliding window to maintain context.
  3. Link Extraction: To overcome the difficulty of finding URLs buried in footnotes, AutoDataset uses a hybrid approach: a rule-based scoring system (prioritizing GitHub or Hugging Face) with a fallback to the paper's LaTeX source code to find "hidden" links.
  4. Indexing and Retrieval: Extracted data is embedded using a dense semantic retriever (gte-large), allowing users to search for datasets using natural language queries rather than just keywords.

Implementation and Results

The system is deployed as a live, continuous crawler that provides researchers with an up-to-date interface for dataset discovery and management.

The AutoDataset system interface, showing the configuration panel, crawl control, and search/retrieval functionality.

Key performance metrics include:

  • High Precision: The BERT-GATE detector achieved an F1 score of 0.94.
  • Superior Extraction: BERT-DESC outperformed heavy long-context models like Longformer and BigBird while being 96% faster than API-based LLMs.
  • Real-World Impact: In pilot studies, AutoDataset reduced the time to locate a newly released dataset from 4–9 minutes down to just 0.5–2 minutes—an 80% improvement in efficiency.

Conclusion

AutoDataset demonstrates that a "paper-first" philosophy, combined with specialized lightweight models, can significantly streamline the discovery of machine learning benchmarks. By anchoring discovery to canonical introductory papers, the system avoids deduplication issues and provides a scalable solution to the current dataset explosion. Future work will expand the system's coverage to other scientific fields and extract finer-grained metadata such as data modality and task applicability.

Summary updated

Abstract

The continuous expansion of task-specific datasets has become a major driver of progress in machine learning. However, discovering newly released datasets remains difficult, as existing platforms largely depend on manual curation or community submissions, leading to limited coverage and substantial delays. To address this challenge, we introduce AutoDataset, a lightweight, automated system for real-time dataset discovery and retrieval. AutoDataset adopts a paper-first approach by continuously monitoring arXiv to detect and index datasets directly from newly published research. The system operates through a low-overhead multi-stage pipeline. First, a lightweight classifier rapidly filters titles and abstracts to identify papers releasing datasets, achieving an F1 score of 0.94 with an inference latency of 11 ms. For identified papers, we parse PDFs with GROBID and apply a sentence-level extractor to extract dataset descriptions. Dataset URLs are extracted from the paper text with an automated fallback to LaTeX source analysis when needed. Finally, the structured records are indexed using a dense semantic retriever, enabling low-latency natural language search. We deploy AutoDataset as a live system that continuously ingests new papers and provides up-to-date dataset discovery. In practice, it has been shown to significantly reduce the time required for researchers to locate newly released datasets, improving dataset discovery efficiency by up to 80%.
BibTeX
@misc{yang2026autodataset,
  author = {Junzhe Yang and Xinghao Chen and Yunuo Liu and Zhijing Sun and Wenjin Guo and Xiaoyu Shen},
  title = {{AutoDataset: A Lightweight System for Continuous Dataset Discovery and Search}},
  year = {2026},
  eprint = {2603.07271},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.07271},
}
Figure