Can a Vision-Language Model Tune Clustering In Context?

Javith Ranatunga, Paimon Goulart, Evangelos E. PapalexakisarXiv 2026 · arXiv

TL;DR

This paper explores whether Vision-Language Models (VLMs) can act as "visual consultants" to automate the tuning of clustering algorithms. By visually inspecting cluster plots and using in-context learning, the VLM (Qwen3.5-27B) successfully guided a k-means algorithm to optimal configurations in significantly fewer iterations than traditional grid search or Bayesian optimization.

What Problem Does This Paper Address?

Clustering is a cornerstone of unsupervised learning, yet it remains notoriously difficult to evaluate. Because there are no "ground truth" labels, researchers often rely on internal metrics like the Silhouette Coefficient or the Elbow Method. However, these metrics can be ambiguous or even contradictory. Consequently, the "gold standard" for many researchers is still manual visual inspection: looking at a plot and deciding if the clusters "make sense."

Traditional Hyperparameter Optimization (HPO) methods like Grid Search or Bayesian Optimization are effective but require a clear mathematical objective to maximize. If a researcher isn't sure which metric best captures their domain-specific needs, HPO becomes a guessing game. This paper asks: Can we automate the human-in-the-loop visual inspection process using a VLM?

Methods

The authors developed an agentic loop that treats the VLM as both an evaluator and a controller. The process follows these steps:

  1. Visual Input: The VLM is shown a plot of a current k-means clustering result.
  2. In-Context Learning (ICL): To calibrate the model, it is provided with three "fixed" visual examples ranging from "worst" to "best" possible clustering for that scenario.
  3. Scoring & Feedback: The VLM assigns a score (1 to 10) to the clustering and proposes new hyperparameters—specifically the number of clusters ($k$) and the initialization method (k-means++ vs. random).
  4. Iteration: If the score is below 8, the system reruns k-means with the new parameters and feeds the new plot back to the VLM. The loop terminates when the VLM is satisfied or reaches 10 iterations.

The researchers tested this using Qwen3.5-27B, a powerful open-source VLM, with "thinking" disabled and temperature set to 0 to ensure deterministic, reproducible results.

Datasets

The study utilized five controlled synthetic clustering scenarios (borrowed from Schubert’s prior work) that represent common challenges in the field:

  • Well-Separated: Clear, distinct groups.
  • Overlapping: Clusters that bleed into one another.
  • Many-Blobs: A high number of small, dense groups.
  • Uniform-Noise & Normal-Noise: Increasingly ambiguous cases where clear clusters don't actually exist.

The evaluation was performed across 100 total clustering tasks (20 datasets per scenario).

Key Insights and Ideas

  • VLMs Mimic Human Intuition: The VLM performed almost perfectly on "Well-Separated" and "Overlapping" data but struggled with noisy, ambiguous plots. This suggests that VLMs share similar visual biases and limitations as human researchers when interpreting data.
  • Visual-Metric Correlation: Surprisingly, the VLM’s subjective 1–10 scores showed a strong positive correlation with the Silhouette Coefficient for clear datasets, even though the VLM was never shown the underlying numerical data or the formula for the metric.
  • Efficiency over Exhaustion: The VLM-driven agent reached "good enough" solutions much faster than traditional methods. While Grid Search might check 50+ combinations, the VLM usually found a high-quality result in fewer than 5 iterations.
  • Preserving Interpretability: Unlike "Deep Clustering" methods where a neural network directly assigns clusters in a "black box" fashion, this framework keeps the clustering algorithm (k-means) explicit. The VLM only acts as the "tuner," making the final logic easy for a human to audit.

Key Results

  • Accuracy: In pairwise comparisons (choosing the better of two plots), the VLM matched the "correct" choice (defined by Silhouette score) nearly 100% of the time for well-separated data.
  • Performance vs. Baselines: The agentic framework produced average Silhouette scores that were "relatively close" to those achieved by exhaustive Grid Search and Bayesian Optimization.
  • Speedup: The VLM agent terminated in fewer than 5 iterations on average across all scenarios. In contrast, Grid Search required roughly 18 evaluations for simple scenarios and up to 58 for the "Many-Blobs" scenario.

Limitations and Future Work

  • Algorithm Scope: The study was limited to k-means. Future work is needed to see if VLMs can tune more complex algorithms like DBSCAN or HDBSCAN, which have more sensitive hyperparameters (like epsilon or minimum cluster size).
  • Dimensionality: The experiments were conducted on 2D synthetic data. Real-world data is often high-dimensional, requiring dimensionality reduction (like PCA or t-SNE) before a VLM can "see" it. The impact of these projections on the VLM's judgment remains unexplored.
  • Noise Sensitivity: The model’s performance degraded significantly in high-noise environments, indicating that VLMs might need more specialized "visual prompts" or ICL examples to handle messy real-world datasets.

Summary updated

Abstract

Clustering is a widely used unsupervised learning method, but selecting an appropriate algorithm and set of hyperparameters for a specific dataset can require many possible configurations and evaluations. In this work, we investigate whether a Vision-Language Model can instead visually evaluate clustering results and use those decisions to then tune an explicit clustering algorithm. We study this through three progressive experiments using Qwen3.5-27B and five controlled synthetic clustering scenarios. First, we test whether the VLM can distinguish the best clustering result from a pair of plots. Second, we test whether it can assign numerical scores to those plots where a higher score represents a better result. Finally, we use these judgments within an agentic loop in which the VLM scores a k-means clustering and proposes updated hyperparameters that may result in better clustering. Our exploratory results show that the VLM performs well when clear cluster structure is present, while struggling on more ambiguous and noisy settings. In the full agentic experiment, the VLM produces clustering configurations with silhouette scores relatively close to those of grid search and Bayesian optimization, while terminating in much fewer iterations. Overall, these results provide an initial proof of concept that VLM-based visual feedback can be used to autonomously tune an explicit clustering algorithm.
BibTeX
@misc{ranatunga2026can,
  author = {Javith Ranatunga and Paimon Goulart and Evangelos E. Papalexakis},
  title = {{Can a Vision-Language Model Tune Clustering In Context?}},
  year = {2026},
  eprint = {2611.18768},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18768},
}
Figure