GradientSpace: Unsupervised Data Clustering for Improved Instruction Tuning

Shrihari Sridharan, Deepak Ravikumar, Anand Raghunathan, Kaushik RoyarXiv 2025 · DOI · arXiv

Introduction

Instruction tuning is the critical process that transforms raw pretrained Large Language Models (LLMs) into capable assistants. However, real-world instruction datasets are often highly heterogeneous, containing a "Data Mix" of mathematics, code, creative writing, and more. Training a single model on such diverse data frequently leads to gradient interference—a phenomenon where conflicting learning signals from different tasks (e.g., the precise logic of math vs. the fluid nature of creative writing) pull model parameters in opposing directions, resulting in "negative transfer" and degraded performance.

GradientSpace is a novel framework designed to solve this problem by identifying and isolating these conflicting signals. By clustering data in full-dimensional gradient space using an efficient online algorithm, it creates specialized experts that outperform existing methods while maintaining high inference efficiency.

Methods

The GradientSpace pipeline consists of three distinct stages designed to move beyond simple semantic grouping and instead cluster data based on how the model actually learns:

  1. Stage I: LoRA Warm-up: The model undergoes brief training on a small data subset using Low-Rank Adaptation (LoRA). This phase ensures the adapters produce meaningful, task-specific gradient representations.
  2. Stage II: Online SVD-based Clustering: To handle the memory constraints of full-dimensional gradients, the framework employs an online Singular Value Decomposition (SVD) algorithm. It identifies dominant directions of gradient variance to determine the optimal number of clusters ($K$) and refines them using an exponential moving average (EMA) update.
  3. Stage III: Expert Specialization & Routing: Specialized LoRA experts are trained for each discovered cluster. A lightweight semantic router (based on a DistilBERT-like encoder) is then trained to predict the appropriate expert for any given input at inference time.
The GradientSpace workflow: from initial LoRA warm-up to online centroid calibration and final expert routing.

Key Insights

  • Gradient Space vs. Semantic Space: A core finding of this research is that semantic similarity is a poor proxy for learning dynamics. Prompts that appear similar often produce vastly different gradients, while unrelated tasks may reinforce each other. GradientSpace organizes data by these underlying "learning dynamics."
  • The Power of Full Dimensions: Unlike prior methods that use random projections to reduce dimensionality (and lose information), GradientSpace operates in the full gradient dimension. The SVD-based initialization makes this computationally feasible without sacrificing accuracy.
  • Single Expert Efficiency: While many Mixture-of-Experts (MoE) approaches use ensembles, GradientSpace demonstrates that routing to a single, well-trained expert is both faster and more accurate, as it prevents the dilution of specialized knowledge.

Key Results

GradientSpace consistently outperformed standard finetuning and existing gradient-based clustering methods across multiple benchmarks using LLAMA-2-7B and LLAMA-3.2-1B models.

  • Accuracy Gains: On the MATH benchmark with Llama-2, GradientSpace achieved 46.8% accuracy, a significant jump from standard LoRA finetuning (28.4%) and the previous state-of-the-art method, ELREA (41.8%).
  • Inference Speed: By utilizing a lightweight semantic router rather than computing gradients on-the-fly, GradientSpace adds only 3.9ms of latency. This is a massive improvement over gradient-based routing methods, which can add over 2000ms of latency.
Accuracy comparison across Finance, Math, Code, and Creative Writing tasks, showing GradientSpace outperforming semantic and random clustering.

Limitations and Future Work

The framework currently relies on a warm-up phase; if the initial data subset is unrepresentative, the resulting clusters may be suboptimal. Additionally, while the SVD helps estimate the number of clusters, the expert count remains fixed after Stage II. Future research could explore dynamic expert merging or scaling these techniques to 70B+ parameter models in production environments.

Summary updated

Abstract

Instruction tuning is one of the key steps required for adapting large language models (LLMs) to a broad spectrum of downstream applications. However, this procedure is difficult because real-world datasets are rarely homogeneous; they consist of a mixture of diverse information, causing gradient interference, where conflicting gradients pull the model in opposing directions, degrading performance. A common strategy to mitigate this issue is to group data based on semantic or embedding similarity. However, this fails to capture how data influences model parameters during learning. While recent works have attempted to cluster gradients directly, they randomly project gradients into lower dimensions to manage memory, which leads to accuracy loss. Moreover, these methods rely on expert ensembles which necessitates multiple inference passes and expensive on-the-fly gradient computations during inference. To address these limitations, we propose GradientSpace, a framework that clusters samples directly in full-dimensional gradient space. We introduce an online SVD-based algorithm that operates on LoRA gradients to identify latent skills without the infeasible cost of storing all sample gradients. Each cluster is used to train a specialized LoRA expert along with a lightweight router trained to select the best expert during inference. We show that routing to a single, appropriate expert outperforms expert ensembles used in prior work, while significantly reducing inference latency. Our experiments across mathematical reasoning, code generation, finance, and creative writing tasks demonstrate that GradientSpace leads to coherent expert specialization and consistent accuracy gains over state-of-the-art clustering methods and finetuning techniques.
BibTeX
@misc{sridharan2025gradientspace,
  author = {Shrihari Sridharan and Deepak Ravikumar and Anand Raghunathan and Kaushik Roy},
  title = {{GradientSpace: Unsupervised Data Clustering for Improved Instruction Tuning}},
  year = {2025},
  eprint = {2512.06678},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.06678},
}
Figure