Just Select Twice: Leveraging Low Quality Data to Improve Data Selection

Yifei Zhang, Yusen Jiao, Jiayi Chen, Zhaoyang Li, Huaxiu Yao, Jieyu Zhang, Frederic SalaOpenReview 2025 · Submission to ICLR 2025 · Rejected · OpenReview

Abstract

Data valuation is crucial for assessing the impact and quality of individual data points, enabling the ranking of data by importance for efficient data collection, storage, and training. Many data valuation methods are sensitive to outliers and require a certain level of noise to effectively distinguish low-quality data from high-quality data, making them particularly useful for data removal tasks. In particular, optimal transport-based methods exhibit notable performance in outlier detection but show only moderate effectiveness in high-quality data selection, due to their sensitivity to outliers and insensitivity to small variations. To mitigate the issue of insensitivity to high-quality data and facilitate effective data selection, in this paper, we propose a straightforward two-stage approach, JST, that initially does data valuation as usual, but then performs a second-round data selection where the identified low-quality data points are designated as the validation set to perform data valuation again. In this way, high-quality data become outliers with respect to the new validation set and can be naturally identified. We empirically evaluate an instantiation of our framework based on optimal transport method for data selection and data pruning on several standard datasets and our framework demonstrates superior performance compared to pure data valuation, especially under small noise conditions. Additionally, we show the general applicability of our framework to influence function based and reinforcement learning based data valuation methods.
BibTeX
@misc{zhang2025just,
  author = {Yifei Zhang and Yusen Jiao and Jiayi Chen and Zhaoyang Li and Huaxiu Yao and Jieyu Zhang and Frederic Sala},
  title = {{Just Select Twice: Leveraging Low Quality Data to Improve Data Selection}},
  year = {2025},
  url = {https://openreview.net/forum?id=dugoA2gfhs},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2025},
}

Loading related content...

Figure