Abstract
Fine-tuning large-scale language models for specialized tasks like text editing typically requires vast amounts of annotated data. We present DEFT-UCS, a framework that utilizes Unsupervised Core-Set Selection (UCS) to reduce data requirements by up to 70% while maintaining or exceeding state-of-the-art performance. By leveraging unsupervised clustering and geometric sampling strategies, DEFT-UCS identifies high-utility "hard" samples that drive model generalization, offering a cost-effective path for domain-specific model adaptation.
Introduction
The prevailing paradigm in instruction-tuning Pre-trained Language Models (PLMs) emphasizes large-scale datasets. However, for nuanced tasks such as professional text editing—encompassing bias neutralization and coherence improvement—high-quality labels are expensive and scarce. Existing data pruning techniques often rely on supervised metrics, necessitating a pre-existing reference model. DEFT-UCS addresses this by demonstrating that the most informative training samples can be identified using only raw input text through unsupervised methods.
Methodology: Unsupervised Core-Set Selection
The core of our approach is the Unsupervised Core-Set Selection (UCS) algorithm. Unlike supervised methods, UCS operates on the latent representations of raw input sentences to identify a representative subset for fine-tuning.
The process involves four distinct stages:
- Embedding: Sentences are mapped to a vector space using Sentence-T5, which provides superior task separation compared to other embedding models.
- Clustering: K-Means clustering groups data into $K$ clusters, where $K$ corresponds to the known task intentions in the dataset.
- Geometric Sampling: Within each cluster, samples are categorized based on their cosine distance from the centroid. "Hard" samples (those furthest from the centroid) are prioritized as they represent diverse task boundaries and edge cases.
- Fine-Tuning: A Flan-T5 Large model is trained on a combination of a small stratified base set ($D_{base}$) and the selected core-set ($D_c$).
Experimental Setup
We evaluated DEFT-UCS using the CoEDIT dataset, which contains 82,000 samples. Performance was measured across eight benchmarks covering simplification (TurkCorpus, Asset), coherence/clarity (Iterator series), grammar (JFLEG), and neutralization (WNC).
Results and Discussion
DEFT-UCS demonstrates remarkable data efficiency. By training on only 32.5% of the original CoEDIT data, the model achieved comparable or superior SARI and ROUGE-L scores on 6 out of 8 evaluation datasets.
The Value of "Hard" Samples
A key insight from our study is that "hard" samples—those furthest from cluster centers—are significantly more valuable than "easy," prototypical samples when data is constrained. These outliers likely capture the complex linguistic boundaries necessary for robust generalization. Human evaluation confirmed this, with DEFT-UCS edits perceived as accurate 83.8% of the time, compared to 70.5% for the model trained on the full dataset.
Task-Specific Data Requirements
The utility of data pruning varies by task subjectivity. Objective tasks like simplification reached peak performance with just 12% of the data. In contrast, highly subjective tasks like neutralizing bias (WNC) required over 80% of the data to match the full-set baseline, suggesting that diversity in subjective tasks is harder to capture in small core-sets.
Conclusion
DEFT-UCS proves that unsupervised geometric clustering is a viable alternative to supervised data pruning. By reducing the reliance on massive labeled datasets, this framework enables more accessible fine-tuning for specialized NLP tasks. Future work will focus on automating hyperparameter selection for $K$ and sampling weights to further streamline the pipeline.
Summary updated