When Is a Pragmatic Similarity Score Reproducible Enough to Be a Test?

Speech researchers now have automatic models that estimate pragmatic similarity: whether two spoken utterances do the same social job, such as reassuring, closing a topic, or sounding confident, regardless of the exact words or pitch. The best-known model averages features from a self-supervised speech network (HuBERT, a model trained to predict masked chunks of audio) and reports correlations with human judges close to the level at which humans agree with each other. That closeness is the reason one would want to use such a model as an automatic oracle, for example a regression test that fails when a new speech synthesizer stops sounding reassuring. It is also the reason the model's reproducibility matters. When the gap between model and humans is a few hundredths of a correlation point, small changes in how audio is decoded, resampled, padded, or computed on different hardware could move the result by a similar amount. Nobody has measured this.

This note proposes to measure it, and to use the measurement to define a pass/fail rule for pragmatic tests. The central idea is a variance budget with two terms. One term is environment variance: how much the score for a fixed utterance pair moves across sealed, versioned software containers that differ only in audio library, model checkpoint loader, numerical precision, or device. The other term is human disagreement: how much the 6–9 human ratings of that same pair spread. A pragmatic score is usable as a test only where the first term is small relative to the second. Where humans themselves disagree widely, the test should abstain instead of issuing a verdict. The work treats container-based reproducibility as the measuring instrument and the subjectivity of pragmatic judgment as the yardstick.

Core Claim

Current reproducibility practice for agents and scientific software, built on sealed containers and deterministic state checks, assumes a single correct answer. Pragmatic similarity has no single correct answer. CEI, a text benchmark of indirect social inference, argues that annotator disagreement reflects real social ambiguity and is not noise. We claim these two views fit together once they are put on one scale. Environment-induced variation should be judged against the width of human disagreement for the same item, not against zero. A pragmatic oracle should give a verdict only when its calibrated score, together with its environment spread, lies clearly inside or clearly outside the band of human ratings. We expect this rule to predict fresh human verdicts on new speech systems better than a fixed threshold on the mean score does. We also expect it to show that some published model-versus-human comparisons fall within reproduction noise.

To make this concrete, let $s_{i,e}$ be the calibrated pragmatic score, mapped onto the 1–5 human rating scale, for utterance pair $i$ computed in container environment $e$. Let $\sigma^{env}_i$ be its standard deviation across environments, and let $\sigma^{hum}_i$ be the standard deviation of the human ratings of pair $i$. The per-item reproducibility ratio is

$$\rho_i = \sigma^{env}_i / \sigma^{hum}_i ,$$

and a test on pair $i$ is called environment-flaky when $\rho_i$ exceeds a threshold fixed in advance (we will report 0.1 and 0.25). This borrows the flaky-test distinction from software engineering but replaces "any nondeterminism is a bug" with "nondeterminism that is large relative to human ambiguity is a bug."

Relation to Prior Work

Three papers supply the object of study. PragSim collected human similarity ratings for seed utterances paired with re-enactments under six conditions, from close vocal mimicry to text-to-speech synthesis. It reports average inter-judge correlation of 0.72 in English and 0.66 in Spanish, and finds that disagreement concentrates on context-only re-enactments and on seeds containing laughter or falsetto. Towards a General-Purpose Model of Perceived Pragmatic Similarity and its demo built the estimator: HuBERT Large layer 24, mean-pooled over time, reduced by greedy search to 103 features and compared by cosine similarity. It reports r = 0.74 with mean human judgments on English Session 2 and beats mel-cepstral and pitch dynamic time warping. We reuse PragSim and this estimator as-is. What we add is an account of how stable the estimator is outside the environment it was built in. The data-dependent greedy feature selection is a second source of instability that the original work did not quantify. A note on the numbers: the 0.74 figure is correlation with the mean judgment, while the human 0.72 figure is correlation between individual judges. Part of our contribution is to compare these two quantities on a common footing.

Two papers motivate the verdict rule and constrain it. CEI shows that pragmatic disagreement is intrinsic, with Fleiss' κ of 0.06–0.25. That rules out any design that treats a single averaged label as ground truth. Refining sentence similarity with discourse information combines a lexical score with a dialog-act distance and falls back to the lexical score when the act classifier is unsure. That confidence-gated fallback is the direct ancestor of our abstention rule. We gate on human disagreement and environment spread instead of classifier confidence.

The reproducibility side supplies method and contrast. A qualitative study of flaky tests finds that practitioners rely on reruns (half of interviewees) as a stopgap and name hermetic environments as the real fix. Benchmarks such as τ-bench and Agent-Diff verify success by deterministic state comparison inside sealed environments. τ-bench also introduces pass^k, the probability of passing all k trials, to expose inconsistency. We adopt its logic by asking whether a test's verdict holds across all k environments rather than across k agent runs. Unlike these benchmarks, our target is a distribution of human perceptions, not a database state. Establishing Best Practices for Building Rigorous Agentic Benchmarks audits agent benchmarks and names extension to "subjective tasks and social interaction" as open. This proposal is one concrete step in that direction for speech. ReSEval makes human listening tests reproducible through configuration files and seeded stimulus order. We use it to collect fresh judgments. It does not isolate model execution and does not relate automatic metrics to human spread, and those are the two things this proposal does.

Research Program

The first component is an environment matrix. We will build sealed container images, in the style of OpenHands's fresh-image-per-run sandboxes and Bencher's isolation of dependency-sensitive benchmarks. Each image computes PragSim scores end to end, from raw audio file to cosine similarity. The images vary one factor at a time from a reference image. The factors are the audio decoder version, the resampling library used to reach 16 kHz, the checkpoint loader (the original research toolkit versus a general model hub), fp32 versus half precision, CPU versus GPU, and batched inference with or without correct masking of padded frames during mean pooling. We suspect the padding factor matters most, because unmasked padding changes an average-pooled vector directly. Each image is run twice to confirm it is deterministic on its own, so that all measured variance can be attributed to the factor under study.

The second component isolates estimator variance. We will re-run the greedy 103-feature selection on bootstrap resamples of the tuning data and on each environment's embeddings. This yields a distribution of selected feature sets and of resulting correlations. It tells us whether "103 features" is a stable object or an artifact of one run.

The third component is the verdict rule. We fit a monotone calibration from estimator score to the 1–5 scale on training seeds. For each held-out pair, the rule returns "similar" when the calibrated score's environment interval lies above a human-derived acceptance level, returns "dissimilar" when it lies below, and abstains otherwise. The acceptance level is the lower quartile of ratings for close vocal mimicry pairs. The rule also abstains whenever $\sigma^{hum}_i$ for comparable training items is in the top decile, which covers the laughter and context-only cases PragSim flagged. Because the estimator cannot see per-item human spread at test time, we will predict that spread from the condition label and from embedding distance to high-disagreement training items. How accurate that prediction is gets reported as its own result.

Baselines

The naive reference point is a single run of the published estimator on one unpinned machine, with a fixed threshold on its score, validated only by correlation with mean human ratings. The strongest existing method is the 103-feature HuBERT cosine estimator of Towards a General-Purpose Model of Perceived Pragmatic Similarity, which reports Pearson r = 0.74 with mean human judgments on English Session 2 and 0.72 with individual judges on lexically identical pairs. It is the strongest competitor because it is the only published estimator tuned on these exact judgments. We will reproduce it on the same Session 2 split, with the same layer, pooling, and feature count, in our reference container. If we cannot recover roughly 0.74 there, that failure is itself a finding and becomes the anchor for everything else.

Two further baselines come from the same paper: mel-cepstral dynamic time warping and pitch dynamic time warping. The file reports that HuBERT beats them but gives no numbers, so no reported result was found on Lacuna for either DTW baseline, and we will compute them ourselves. They matter here for a different reason: they are deterministic signal-processing measures that may be nearly immune to environment variance. They therefore bound how much stability one can buy by giving up accuracy. A full-embedding cosine baseline with no feature selection serves as the ablation that isolates the selection step's contribution to instability.

For the verdict rule, the key ablation compares three versions on identical held-out pairs and identical fresh human judgments. The first is a fixed-threshold rule tuned on the training split with the same tuning budget. The second is our rule with environment intervals but no human-disagreement abstention. The third is the full rule. Beating the fixed threshold means higher accuracy against the majority verdict of fresh human raters on the items where the rule does decide, with coverage reported alongside. It also means matching or exceeding the fixed threshold's accuracy on everything once abstentions are counted as errors. Otherwise abstention is just hiding difficulty.

Why not just…?

Why not just pin one Docker image and declare the estimator reproducible? Pinning makes results identical inside one image. It says nothing about whether a result obtained in that image transfers to the image another lab builds, and it does not tell us whether the image's choices, such as a particular resampler or unmasked padding, quietly shift correlations. The environment matrix directly tests whether pinning is enough. If every $\rho_i$ comes out below 0.1, we will say so plainly, and the contribution narrows to the disagreement-calibrated verdict rule and the feature-selection stability result. That outcome is plausible for pure floating-point effects, which is why the matrix deliberately includes preprocessing and padding factors, where larger effects are expected.

Why not just rerun and average, as practitioners do with flaky tests? Averaging across runs in the same environment removes nothing if each image is deterministic, and averaging across environments hides which factor caused the drift. The flaky-test study reports that practitioners themselves regard reruns as a palliative. Our per-factor design reports attribution, which averaging cannot.

Why not just use a fixed threshold on the mean human rating, or ask a language model to judge pragmatic appropriateness? A fixed threshold treats every pair as equally determinate, while PragSim shows disagreement clusters on predictable conditions. The fixed-threshold ablation tests whether ignoring this costs accuracy on fresh judgments. Language-model judges are a weaker option here for two reasons. They read transcripts, and lexically identical pairs differ only in delivery. CEI also reports that even Llama-3.1-70B reaches 25.0% against 54.3% human majority agreement on text pragmatics, with no gain from chain-of-thought prompting. We will not build on them, though an audio-capable model judge could be added as an extra row if compute allows.

Why not just re-collect human ratings with ReSEval every time? For a one-off study that is the gold standard, and we use it for validation. As a regression test run on every synthesizer change it is slow and costly. The purpose of the verdict rule is to decide which items need humans at all: the rule's abstentions are exactly the items to send to a ReSEval study.

Proposed Experiments and Decisive Checks

All experiments use the 689 PragSim pairs across two English sessions and one Spanish session. Seeds are split so that no seed appears in both the tuning and evaluation sets. The first decisive check reports the distribution of $\rho_i$ per environment factor and per re-enactment condition. The question is whether any factor pushes a meaningful share of pairs past $\rho_i = 0.25$. The second check asks whether the estimator's correlation with mean judgments, and its ranking against the full-embedding and DTW baselines, stays the same across all environments and bootstrap feature selections. A ranking flip, or a correlation interval that overlaps the human inter-judge level, would mean current model-versus-human claims cannot be resolved at present reproducibility. The third check puts the estimator and humans on one scale. It compares the estimator's correlation with held-out individual judges against leave-one-judge-out human correlation, with bootstrap intervals, to settle whether "approaching human agreement" survives.

The applied test uses new stimuli. For a held-out set of English seeds, we generate outputs from two or three current speech synthesizers, which are not in PragSim, and collect fresh ratings from at least six judges per pair through a pinned ReSEval configuration. The full verdict rule, its two ablations, and the naive single-run threshold then each predict the human majority verdict of "similar" or "dissimilar" for every synthesizer. The proposal succeeds if the full rule decides a majority of pairs, is more accurate than the fixed threshold on those pairs, and does not lose accuracy once abstentions are counted. It also succeeds if its abstentions fall disproportionately on the pairs where the fresh judges split, which is the direct test that environment spread and human subjectivity have been put on one usable scale. A Spanish replication with English-tuned calibration checks whether the variance budget transfers across languages, as the original estimator partly did.