Selecting Representative Data via Cluster Centroids

43 concepts41 papers1 proposal

When building machine learning workflows under strict data or annotation budgets, a foundational question arises: which data points should we prioritize? Traditional active learning frameworks often lean heavily on uncertainty sampling, hunting for boundary cases where a model is most confused. However, this edge-seeking strategy has a dangerous blind spot—it is highly sensitive to outliers, label noise, and idiosyncratic data quirks.

An alternative paradigm has emerged across computer vision, natural language processing, and system optimization: centroid-based representative selection. Instead of searching for the most uncertain examples at the fringes of a distribution, this approach maps data into a latent feature space, clusters it, and selects the geometric centers—the centroids, medoids, or medians. The core intuition is simple: the center of a cluster represents the most typical, prototypical instance of a concept, maximizing the generalization of any learning signal applied to it.

The Geometry of Representativeness

Why does selecting a centroid work so well? To understand the mechanism, consider how data is distributed in a high-dimensional embedding space. A cluster represents a dense region of semantically or structurally similar items. The mathematical center of this cluster—whether computed as a mean vector (centroid) or an actual data point that minimizes the distance to all others (medoid)—acts as a statistical summary of that entire neighborhood.

As shown by Huang et al. in Tsinghua University at TAC 2009: Summarizing Multi-documents by Information Distance, selecting an object that minimizes the sum of distances to all other documents captures shared, central ideas far better than attempting to be exhaustively comprehensive. This geometric property means that a single label or optimization step applied to a centroid inherently generalizes to its neighbors. For instance, in few-shot medical landmark detection, Quan et al. introduce a framework in Which images to label for few-shot medical landmark detection? that calculates similarity scores in a latent space to identify samples at the center of the data distribution. This prioritizes average examples that generalize robustly across an entire dataset without requiring initial human annotations.

Immunity to Noise and Outliers

The primary failure mode of uncertainty-based sampling is its attraction to noise. An outlier sitting between two class boundaries will perpetually confuse a model, causing active learning algorithms to waste precious annotation budgets on uninformative anomalies. Centroid-based selection flips this dynamic.

By definition, outliers reside far from cluster centers. In their work on active learning for vertex classification, Burr et al. demonstrate in Active Few-Shot Learning for Vertex Classification Starting from an Unlabeled Dataset that selecting cluster medoids captures global structural information far more effectively than uncertainty sampling, which gets trapped by noisy boundary cases. Similarly, Doucet et al. introduce a heuristic in Bridging Diversity and Uncertainty in Active learning with Self-Supervised Pre-Training that targets central, typical samples within self-supervised clusters to solve the "cold start" problem, ensuring that early training phases are not derailed by anomalous data.

This approach also prevents models from overfitting to source-specific quirks during domain adaptation. As demonstrated by Cui in Multi-Source Attention for Unsupervised Domain Adaptation, selecting pseudo-labeled target instances based on their proximity to the target domain's centroid ensures that the model learns from representative examples rather than high-confidence outliers.

This principle extends to filtering out extreme variations that could degrade performance. For example, Roli et al. utilize centroid-based selection in A novel classification-selection approach for the self updating of template-based face recognition systems to retain only the core, dominating modes of a user's appearance, successfully blocking impostor distributions that typically overlap with extreme facial variations. When dealing with messy, real-world data, Xia et al. demonstrate in Moderate Coreset: A Universal Method of Data Selection for Real-world Data-efficient Deep Learning that selecting data points near the median of a distribution provides a robust statistical proxy for the entire dataset, balancing predictive information while remaining completely insulated from label noise and corruption.

Maximizing Label Efficiency in Few-Shot Learning

When training data is scarce, every single labeled example must carry maximum informational weight. Centroid selection ensures that human expertise is broadcasted efficiently across the unlabeled pool.

In low-resource natural language processing, once a specific category or confusion zone is flagged, Chaudhary et al. show in Reducing Confusion in Active Learning for Part-Of-Speech Tagging how a centroid-based approach selects the most prototypical instance for human annotation, maximizing the contextual information gained. For cross-domain keyphrase extraction, Pattanaik et al. establish a strategy in Clustering-based Sampling for Few-Shot Cross-Domain Keyphrase Extraction that embeds documents into a vector space and prioritizes those with the highest average cosine similarity to cluster centers, allowing models to rapidly master the core features of a target domain with minimal labels.

This strategy is equally powerful in complex spatial and behavioral domains. In 3D scene understanding, Hou et al. show in Exploring Data-Efficient 3D Scene Understanding With Contrastive Scene Contexts that performing k-means clustering on self-supervised features and selecting cluster centroids as manual annotation targets captures non-redundant geometric structures with a tiny fraction of the labels. For tracking animal behavior, Li et al. demonstrate in OpenLabCluster: active learning based clustering and classification of animal behaviors based on kinematic body keypoints that a single human label applied to a segment at the center of an unsupervised cluster can be instantly generalized to all similar points within that cluster, drastically reducing manual annotation overhead.

Streamlining Optimization, Prompting, and Evaluation

Beyond active learning, centroid-based selection serves as a core mechanism for compressing datasets, optimizing generative models, and structuring prompts for large language models (LLMs).

In dataset distillation, where massive datasets are condensed into tiny synthetic versions, Liu demonstrates in DREAM: Efficient Dataset Distillation by Representative Matching that matching synthetic data to the centroids of sub-clusters—rather than random real batches—provides significantly smoother gradient signals by focusing on typical feature representations. For LLM pretraining, Tirumala et al. introduce a method in D4: Improving LLM Pretraining via Document De-Duplication and Diversification that ranks documents by their proximity to cluster centers, concentrating the training set on central concepts and discarding uninformative outliers to boost efficiency.

The paradigm also enhances how we interact with and evaluate modern models. In prompt engineering, Zhang et al. automate Chain-of-Thought prompting in Automatic Chain of Thought Prompting in Large Language Models by identifying the most central question in a clustered dataset to serve as the prototypical reasoning demonstration. When evaluating LLMs on highly constrained budgets, Yauney et al. find in How Reliable is Language Model Micro-Benchmarking? that selecting evaluation examples that act as centroids in prediction space maximizes the information gained from tiny micro-benchmarks.

Conclusion

Whether the task is predicting perceived age (Ueki et al.), forecasting trajectories (Porrello et al.), or establishing relative representations across disparate latent spaces (Hüttebräucker et al.), the underlying mechanism remains identical. By shifting our focus from the uncertain boundaries to the geometric centers of data distributions, centroid-based representative selection provides a stable, noise-resistant, and highly generalizable foundation for data-efficient machine learning.

Key Foundational Work

A Semi-Supervised Approach to Perceived Age Prediction from Face Images (2010) — contributed 2 concept elements to this direction.
Tsinghua University at TAC 2009: Summarizing Multi-documents by Information Distance (2009) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_centroid_based_representative_sampling_d5d027fa,
  title = {centroid-based representative sampling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/centroid-based-representative-sampling/rdr_d5d027faee2b464aa6f784d7c3751b48},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_centroid_based_representative_sampling_d5d027fa_v1,
  title = {centroid-based representative sampling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/centroid-based-representative-sampling/rdr_d5d027faee2b464aa6f784d7c3751b48/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake