Which Pipeline Choices Decide a Cluster's Name? Sensitivity-Pruned Provenance for LLM Cluster Labeling

When a large language model names a cluster of customer messages "card not arriving," that name looks like a finding about the data. Often it is partly a record of choices the analyst made and never wrote down: which prompt template was used, which example messages were shown to the model, how many clusters were requested, which embedding grouped the messages, and what filtering threshold removed outliers. This proposal treats automated cluster labeling as a small scientific procedure. We would vary its undocumented choices systematically, measure how much each one moves the resulting labels, and publish a provenance record (a machine-readable account of how the label was produced) that contains only the choices that actually changed the outcome. Each label then carries a stability score and a short list of the decisions it depends on.

The method answers a question that the provenance literature leaves open: how fine-grained the record of a research process should be. We let the data set the level of detail, so a decision enters the record only if changing it changes the label. It also gives cluster labeling something it lacks, which is a way to tell a robust name from one that depends on the prompt, without asking a second LLM to vouch for the first.

Core Claim

In modern labeling pipelines, much of the variation in the final names comes from analyst decisions rather than from the data. We hypothesize that this decision-induced variation is larger than the variation from re-sampling the same model. We further hypothesize that a per-cluster stability score, computed across a designed sweep of pipeline choices, predicts which labels are wrong better than re-sampling agreement or perturbation-based self-checks. If both hold, the right provenance record for a labeling run is the short list of high-sensitivity decisions together with the values they took. A full log of every parameter is not needed.

Grounding in Prior Work

The labeling side supplies the pipelines we audit. SynCID (Synergizing Large Language Models and Pre-Trained Smaller Models for Conversational Intent Discovery) has an LLM write short "intent descriptors" for each user query. It then trains a smaller encoder to match those descriptors, filters clusters by neighbor consistency, and asks the LLM to name each cluster from its central examples. Its labels are judged only through clustering accuracy against benchmark categories. The prompts, exemplar choices and filter thresholds that shape each name go unrecorded and unvaried. Text-Guided Image Clustering converts images to captions or keywords with a vision-language model, clusters the text, and labels clusters by counting distinctive keywords. It shows that changing the steering prompt restructures the partition. That paper motivates our study, because it shows decision sensitivity exists, but it reports no way to document or quantify it. Hierarchical Level-Wise News Article Clustering via Multilingual Matryoshka Embeddings produces LLM summaries at several levels of granularity. It is another pipeline whose many choices would fall under the same audit.

The closest prior work is Explainable Mapper. In that system, GPT-4o writes hypotheses about clusters in an embedding space, and a second agent perturbs input texts to test whether they stay in the cluster, yielding a consistency score. It holds the pipeline fixed and perturbs inputs. We do the reverse: we hold the data fixed and perturb the analyst's decisions. Its verifier is also an LLM, which risks the explainer and the verifier sharing the same biases. Our fidelity check uses a separate embedding model and the held-out cluster membership instead.

The transparency side supplies the problem framing and the output format. Documenting the research process and the related Polifonia work by Daga et al. build machine-readable research records on W3C PROV-O, a standard vocabulary for describing how something was produced. They name the "granularity dilemma" as unsolved: logging everything produces unreadable graphs, while summaries hide decisive steps. A model for capturing provenance of assertions about chemical substances argues for provenance attached to each individual claim rather than to the whole publication. We adopt that unit, one record per cluster label. A Survey of Domain Knowledge Elicitation in Applied Machine Learning finds that 75% of elicited expert inputs in ML papers go unvalidated. This motivates treating prompts and exemplar choices, which are elicitation by another name, as decisions to be audited. Finally, Label Errors in BANKING77 flags roughly 14% of BANKING77 training utterances as possibly mislabeled. Zhu and Rzeszotarski argue that such errors come from how annotation tasks were designed. Both constrain our evaluation: gold intent names cannot be treated as uncontested truth, so we report results separately on flagged and unflagged regions.

Research Program

We would build a labeling harness that wraps a SynCID-style pipeline and exposes its decisions as explicit factors. The factors are the embedding model, the number of clusters $k$, the descriptor prompt template, the neighbor-filter threshold, the exemplar selection rule (nearest the cluster center, random, or near the boundary), the number of exemplars, the labeling prompt, the labeling LLM, and the sampling temperature. A fractional factorial design, which runs a structured subset of factor combinations, keeps the sweep to a few hundred runs. The prompt-template levels are fixed in advance, so the sweep does not become an open-ended search.

Labels must be compared across runs whose clusters differ. To do this, we match each reference cluster to its counterpart in every run by maximum utterance overlap, using the Hungarian algorithm, and record the match quality. We then embed every matched label with a fixed encoder that is not used anywhere in the pipeline.

For each cluster $c$ and decision $j$, we compute a first-order sensitivity index:

$$S_{c,j} = \frac{\mathrm{Var}_{x_j}\left(\mathbb{E}[\,y_c \mid x_j\,]\right)}{\mathrm{Var}(y_c)}$$

Here $x_j$ is the value taken by decision $j$ and $y_c$ is the label embedding of cluster $c$, measured as cosine distance to the label's centroid across runs. The index is the share of a cluster's label variation that is explained by decision $j$ alone. Cluster stability is one minus the normalized total variance. Sampling temperature and seed form a separate "sampling" factor, so decision-induced and sampling-induced variation are directly comparable.

The provenance record for a cluster is a PROV-O graph in the style of Daga et al. It records the final label, the decisions whose $S_{c,j}$ exceeds a threshold together with the values used, and the stability score. All other decisions are collapsed into a single "immaterial at this tolerance" node. The threshold is chosen on a validation split, as described below.

Baselines

The naive reference point is a single SynCID-style run with default settings. Its label is reported with no stability information, as current papers do. As a confidence score, it is a constant, so its ability to separate good labels from bad ones is chance by construction. We include it to show what a user currently receives.

The strongest existing cluster-discovery system is SynCID itself. Liang et al. report 72.89% unsupervised clustering accuracy on BANKING77, against 67.43% for IDAS. We first reproduce that pipeline and that number, so that our sweep audits a faithful system rather than a weakened one. For SynCID, the file gives no result on label quality as opposed to clustering accuracy: no reported result found on Lacuna for SynCID's label correctness. Our label-fidelity numbers for it will therefore be new measurements and cannot be compared against a published figure.

The strongest competitor for the specific question of which labels to trust is Explainable Mapper's perturbation consistency. We run it on the same reference clusters, with GPT-4o-class models generating perturbations. Its reported reference point is a consistency of 0.83 on possessive-pronoun clusters in BERT-base, a different task from ours, so it is not directly comparable. The second competitor is self-consistency: the same pipeline is re-sampled at nonzero temperature, and agreement among the resulting labels serves as the confidence score.

Fairness is enforced by giving every confidence method the same budget of LLM calls per cluster as our sweep uses, together with the same data, the same reference clusters, and the same fixed evaluation encoder. We also run SimpleXAI-style keyword labels from Text-Guided Image Clustering as a deterministic, fully transparent labeling baseline.

Two ablations isolate the key ingredient. The first computes stability from the sampling factor alone. If it matches the full sweep, decision variation adds nothing. The second keeps the full sweep but replaces the independent fidelity check with an LLM judge, which measures how much circular validation inflates scores.

Our method beats the competitors if three things hold. On held-out clusters, its stability score must flag wrong labels with an area under the ROC curve (AUROC) significantly above both self-consistency and perturbation consistency, using a paired bootstrap over clusters. Its pruned record must predict the label of a fresh run at least as well as a full parameter log. And it must do both under the matched call budget.

Why not just…?

Why not just log every parameter, as experiment trackers already do? A full log is complete but answers no question. It does not tell a reader which of thirty settings made the label "card not arriving" rather than "delivery delay." We test this directly. Given the provenance record of a held-out run, how well can one predict the label of a new run? We compare a full log with our pruned record, both used as features in the same simple predictor. If the full log is clearly more predictive, sensitivity pruning is discarding real information and the threshold is wrong. If the two are equal, the pruned record wins on readability at no cost. That would be an answer to the granularity dilemma posed by Daga et al.

Why not just re-sample the LLM several times and use agreement, which is far cheaper? This is the main threat to the proposal. If label variation comes mostly from sampling noise rather than from analyst decisions, our sweep is an expensive form of self-consistency. The variance decomposition settles the question, because it puts the sampling factor beside every decision factor. The first ablation settles the practical version, by testing whether sampling-only stability predicts label errors as well as the full sweep. We commit in advance to reporting it. If sampling dominates, the paper's conclusion becomes that self-consistency is enough and decision provenance can be dropped for this pipeline class.

Why not just use keyword labels, which are deterministic and transparent by construction? Keyword counting removes prompt sensitivity, but SynCID's results suggest that LLM synthesis is what yields usable names for fine-grained intents. Keyword extractors cannot map colloquial queries onto abstract category names. We include SimpleXAI-style labels under the same fidelity metric. If they match LLM labels on fidelity, the honest recommendation is to use them, and our audit becomes a tool for the cases where they lose.

Why not just verify labels with perturbations, as Explainable Mapper does? Input perturbations ask whether the cluster is stable around a fixed pipeline. They cannot reveal that a different but equally defensible prompt would have produced a different name. Because their generator and judge are LLMs, they can also agree with the explainer for the wrong reasons. The head-to-head AUROC comparison and the LLM-judge ablation measure both concerns.

Proposed Experiments

The primary testbeds are BANKING77, CLINC150 and StackOverflow, which SynCID already uses. A label's correctness is judged in two ways. First, its embedding is compared with the benchmark's intent names after matching. Second, the label is used as a classifier: held-out utterances are assigned to the nearest label under the independent encoder, and this assignment is scored against the cluster's actual membership. The second measure does not depend on the gold names.

For BANKING77, every result is reported separately for clusters dominated by the utterances that Ying and Thomas flag as likely mislabeled and for clean clusters. Those authors found that trimming the flagged utterances raised agglomerative clustering ARI by 8%. We expect low stability to concentrate in the flagged regions. If it does, some apparent "label errors" by the pipeline are honest uncertainty about contested categories rather than failures of the pipeline. CLINC150's out-of-scope queries give a second stress test. Larson et al. report that BERT recalls only 40–60% of out-of-scope queries, so clusters that absorb out-of-scope traffic should receive low stability. As a smaller extension, we apply the sweep to the text-guided image pipeline on LSUN and Sports10. There the steering prompt is a single dominant decision, which provides a sanity check that the method recovers a sensitivity already known to be large.

Three decisive checks determine whether the idea holds. The first is whether decision-induced variance exceeds sampling variance for a substantial share of clusters. The second is whether full-sweep stability beats self-consistency and perturbation consistency at identifying wrong labels under a matched call budget. The third is whether the pruned provenance record predicts fresh-run labels as well as the full log. If the first check fails, the proposal reduces to self-consistency, and we would say so. If the first two succeed and the third holds, labeling papers would have a concrete and cheap reporting standard: each name accompanied by its stability and the few decisions it depends on.