Simplex Clustering via sBeta With Applications to Online Adjustment of Black-Box Predictions

Florent Chiaroni, Malik Boudiaf, Amar Mitiche, Ismail Ben Ayed

Abstract

Deep learning models frequently encounter performance degradation due to domain shift. This paper introduces K-SBETAS, a novel probabilistic clustering method designed to adjust the predictions of "black-box" neural networks in real-time. By modeling softmax outputs using a new "scaled Beta" (sBeta) distribution, the authors enable unsupervised domain adaptation and few-shot learning without requiring access to or retraining of the underlying model's weights.

Introduction

Deep learning models often struggle when encountering data that differs from their training distribution—a phenomenon known as "domain shift." For example, a segmentation model trained on synthetic imagery may fail when deployed on real-world streets. Traditional solutions involve fine-tuning model weights, which is computationally expensive and often impossible when dealing with "black-box" models accessible only via APIs.

This research addresses the challenge of fixing a model's predictions solely by analyzing its output probabilities. By treating these predictions as data points on a probability simplex, the authors propose a clustering approach to recover the true underlying classes in an unsupervised or few-shot manner.

The Geometry of Black-Box Predictions

Standard clustering algorithms like K-Means assume data follows a Gaussian distribution. However, the softmax outputs of a classifier are fundamentally different; they are constrained to a simplex where coordinates sum to one and are typically "peaky," clustering near the vertices (corners) of the space.

Clustering behavior on the simplex under domain shift.

As shown in the figure above, when a domain shift occurs, the clusters move away from the vertices. While standard argmax classification fails to account for this shift, Probabilistic Simplex Clustering (PSC) can identify the new cluster centers, effectively "re-calibrating" the model's predictions without weight updates.

The sBeta Distribution and K-SBETAS

The core innovation of this work is the sBeta (scaled Beta) distribution. Traditional distributions are ill-suited for the asymmetric, highly-peaked nature of softmax data. The sBeta distribution generalizes the Beta distribution to provide the flexibility needed to model these outputs accurately.

Comparison of sBeta density to other distributions.

The K-SBETAS algorithm utilizes this distribution within a probabilistic clustering objective. Key features of the algorithm include:

  • Unimodality Constraints: To prevent mathematical collapse, the authors enforce that each cluster has only one peak.
  • Vertex Initialization: Clusters are initialized at the simplex vertices (representing 100% confidence), aligning with the original training objective of classifiers.
  • Efficiency: Using block-coordinate descent, the algorithm alternates between cluster assignment and parameter updates, making it suitable for real-time applications.

Experimental Results

The authors evaluated K-SBETAS across several benchmarks, demonstrating its versatility and speed:

  1. Unsupervised Domain Adaptation (UDA): On the challenging VISDA-C benchmark, K-SBETAS achieved 56.0% accuracy, significantly outperforming standard K-Means (47.9%).
  2. Few-Shot Learning with CLIP: When applied to the CLIP foundational model, K-SBETAS consistently improved zero-shot and one-shot performance across eight diverse datasets, including ImageNetv2 and Stanford-Cars.
  3. Real-Time Segmentation: In a GTA5-to-Cityscapes road segmentation task, the method maintained a high processing speed of 45 frames per second (fps) on a standard GPU, proving its viability for edge devices.

Conclusion and Future Work

K-SBETAS provides a powerful, model-agnostic framework for adapting black-box predictions on the fly. By leveraging the specific geometry of the probability simplex and the flexible sBeta distribution, it offers a privacy-preserving and computationally efficient alternative to traditional fine-tuning.

While highly effective, the method currently treats data points independently, ignoring spatial or temporal context. Future research may integrate these dimensions or use K-SBETAS as a pseudo-labeling mechanism for more advanced self-training loops.

Summary updated

Abstract

We explore clustering the softmax predictions of deep neural networks and introduce a novel probabilistic clustering method, referred to as k-sBetas. In the general context of clustering discrete distributions, the existing methods focused on exploring distortion measures tailored to simplex data, such as the KL divergence, as alternatives to the standard Euclidean distance. We provide a general maximum a posteriori (MAP) perspective of clustering distributions, emphasizing that the statistical models underlying the existing distortion-based methods may not be descriptive enough. Instead, we optimize a mixed-variable objective measuring data conformity within each cluster to the introduced sBeta density function, whose parameters are constrained and estimated jointly with binary assignment variables. Our versatile formulation approximates various parametric densities for modeling simplex data and enables the control of the cluster-balance bias. This yields highly competitive performances for the unsupervised adjustment of black-box model predictions in various scenarios. Our code and comparisons with the existing simplex-clustering approaches and our introduced softmax-prediction benchmarks are publicly available: https://github.com/fchiaroni/Clustering_Softmax_Predictions.
BibTeX
@article{chiaroni2022simplex,
  author = {Florent Chiaroni and Malik Boudiaf and Amar Mitiche and Ismail Ben Ayed},
  title = {{Simplex Clustering via sBeta With Applications to Online Adjustment of Black-Box Predictions}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2022},
}
Figure