Is Style Matching Inconsistent, or Just Measured Inconsistently?

People unconsciously copy each other's small words, such as pronouns, articles and prepositions, and a decade of studies uses this "linguistic style matching" to read power, rapport and therapeutic quality from text. The findings do not agree. Wikipedia administrators coordinate more than non-administrators, against the hypothesis that powerful people accommodate less (Echoes of power). Forum leaders accommodate less than non-leaders (Finding Zelig in Text). In counseling, overall matching is higher in high-empathy sessions while turn-by-turn coordination is lower (Understanding and Predicting Empathic Behavior). Large language models (LLMs) appear to match their users far less than therapists do, but the therapist transcripts were spoken and the LLM chats were typed (Using Linguistic Entrainment to Evaluate LLMs for CBT). Each paper used a different corpus, a different matching formula, a different way of correcting for a speaker's habits, and a different definition of a "good" outcome.

Clinical meta-analysis has standard tools for this situation. Random-effects models estimate how much true effects vary between studies, the between-study variance $\tau^2$, and what share of the observed spread is real rather than sampling noise, the inconsistency index $I^2$. Moderator analysis then asks which study features explain that spread. We propose a harmonized re-analysis of the public and accessible style-matching corpora named below. Accommodation is recomputed with one estimator that corrects for length and chance. Then $\tau^2$ is split into the part explained by how matching was measured (formula, normalization, modality) and the part explained by what was studied (power structure, outcome perspective). The claim is falsifiable. If the field's contradictions are mostly measurement artifacts, harmonization should shrink $\tau^2$ sharply and flip or erase some reported effects. If they are real, the residual heterogeneity should line up with substantive moderators such as who rates the outcome.

Core Claim

We hypothesize that more than half of the between-corpus variance in two headline relationships disappears once every corpus is scored with the same chance-corrected estimator. The first relationship is status versus accommodation. The second is accommodation versus rated conversation quality. We expect what remains to be explained mainly by outcome perspective: client-rated, expert-coded, or behavioral engagement. Interaction setting should matter less. A direct corollary is that the reported LLM "entrainment deficit" (mean LSM 0.75 versus 0.92 for therapists) will shrink substantially when LLMs answer the same client turns as the therapists and both are scored against the same chance baseline.

The study-heterogeneity literature supplies the method and the warning. Pooling cohorts with drifting definitions creates structured bias, and extreme heterogeneity pushes reviews into narrative summary. PyHealth's response to inconsistent task definitions in health prediction was to standardize the task definition before any model comparison (PyHealth). We do the same for conversational alignment. Style matching is the core object here; heterogeneity analysis is the lens.

Research Program

Unified estimator. We will build one accommodation estimator and apply it to every corpus. It starts from the conditional-probability coordination measure of Echoes of power. That measure asks how much more likely a reply is to contain a marker when the preceding turn contains it, compared with the replier's base rate. Like the Zelig Quotient (Finding Zelig in Text), it subtracts personal habit, but neither measure corrects for corpus-level artifacts. Two such artifacts matter here. Turn length changes how likely any marker is to appear. Spoken transcripts are structurally different from typed text. We therefore add a surrogate correction. For corpus $k$, marker $m$ and replier $a$, the corrected coordination is

$$C^{*}_{k,m,a} = C_{k,m,a} - \mathbb{E}_{\pi}\left[C_{k,m,a}^{(\pi)}\right],$$

where $C$ is the observed coordination and $C^{(\pi)}$ is the same quantity recomputed after pairing each reply with a stimulus turn drawn at random from a non-partner conversation in the same corpus. The random stimulus is matched on turn-length bin and position in the conversation. The subtracted term is therefore the coordination expected from corpus genre, modality and length alone. We will report, alongside $C^*$, the raw LSM score, the uncorrected coordination, the Zelig Quotient, and the word-embedding distance nCLiD used in the CBT entrainment study. Each paper's original metric is then a coded "measurement choice."

Two separate meta-analyses. For each corpus $k$ and each question, we estimate a standardized effect $y_k$ with sampling variance $v_k$. For the power question, this is the standardized difference in accommodation toward higher-status versus lower-status targets. For the quality question, it is the correlation between accommodation and the corpus's quality outcome. We then fit a meta-regression:

$$y_k = \mu + \beta^\top x_k + u_k + \epsilon_k,\quad u_k \sim \mathcal{N}(0,\tau^2),\ \epsilon_k \sim \mathcal{N}(0, v_k),$$

where $\mu$ is the average effect and $x_k$ codes moderators. The measurement moderators are metric, normalization and modality. The substantive moderators are status type, outcome perspective and setting. The quantity of interest is how much $\tau^2$ falls when measurement moderators are equalized, by re-scoring every corpus with $C^*$, versus when only substantive moderators are added.

Corpora and access. The power meta-analysis uses Wikipedia talk pages and Supreme Court oral arguments (Echoes of power). It adds Reddit karma and moderator status from the multi-subreddit setting of Exploring Linguistic Style Matching in Online Communities, split into community-level sub-studies so that $\tau^2$ is estimable. The vBulletin forums from the Zelig work come in only if their authors share them. The quality meta-analysis uses four sources. The first is the 259-session MI quality corpus (What Makes a Good Counselor?); MI, motivational interviewing, is a structured counseling style. The second is the 276-session empathy corpus (Understanding and Predicting Empathic Behavior), which spans five clinical settings and is treated as five sub-studies. The third is the three-trial MI corpus (Predicting Counselor Behaviors), split by trial. The fourth is the CBT entrainment benchmark. Crisis Text Line data (genuine concern study; orientation study) is restricted. It enters only through a data-use agreement. Without it, the "client-rated" moderator level is thin, and we will say so.

Matched-stimulus LLM test. Client turns from the CBT transcripts will be fed to several current LLMs. Their replies are scored with $C^*$ against the same corpus surrogate as the therapists' real replies. Holding the stimulus constant separates relational skill from modality and source.

Baselines

The do-nothing reference point is the field's current state: each paper's reported effect, taken at face value and summarized narratively. Concretely, these are the 0.92 versus 0.75 LSM gap with $p<0.001$ (CBT entrainment), the leader −0.432 versus non-leader −0.181 Zelig Quotients, and the 57.1% cross-domain status accuracy (Echoes of power). We will code these into the same meta-regression to obtain the "as-published" $\tau^2$ and $I^2$. Harmonization is judged against those values.

The naive pooling baseline concatenates all corpora and fits one regression with no corpus term. This is the fixed-effects aggregation the heterogeneity literature warns against. It shares our data and estimator, so any difference isolates the value of modeling between-corpus variance.

The strongest existing method is the mixed-effects approach of Exploring Linguistic Style Matching in Online Communities. It uses raw LSM with controls for utterance length and community baselines, and treats context as random effects. We will fit it to exactly the same corpora with corpus as a random intercept and slope, the same covariates, and no surrogate correction. The tuning budget is identical: the same moderator set and the same leave-one-corpus-out protocol.

For predictive comparisons on single corpora, the reported numbers set the bar. These are 88.03% accuracy for MI quality (What Makes a Good Counselor?), 80.07% for linguistic empathy prediction (Understanding and Predicting Empathic Behavior), and 66.2% balanced accuracy for genuine concern (Crisis counselor language). We do not aim to beat them in-corpus. We aim to show that $C^*$ features lose less when moved to a held-out corpus. We found no reported cross-corpus number for the quality question on Lacuna.

Beating the strongest baseline means two things. The first is a lower residual $\tau^2$ with a narrower 95% prediction interval for a new corpus. The second is better leave-one-corpus-out prediction of the held-out corpus's effect sign and size, with an absolute error that falls by at least a third. Two ablations isolate the key ingredient. The first removes the surrogate subtraction while keeping everything else. The second keeps the surrogate but drops length and position matching.

Why not just…?

Why not just add corpus as a random effect, as existing mixed-effects work does? This is the most serious alternative. Statistically, our one-stage model is a close relative of it. The difference lies in the outcome variable. Raw LSM carries modality and length artifacts into every corpus's effect, so a random intercept absorbs the artifact as if it were real context. The head-to-head comparison in the Baselines section settles this directly. If raw-LSM mixed models reach the same residual $\tau^2$ and transport error as $C^*$, surrogate correction adds nothing and we will report that.

Why not just z-score each corpus's matching scores? Standardizing within corpus removes mean and scale differences. It cannot remove artifacts that change which dyads look aligned, such as long turns mechanically containing more markers. It also cannot resolve sign disagreements like administrators versus forum leaders. A z-scored variant is included as an extra measurement condition, so its $\tau^2$ is directly comparable.

Why not just compare LLMs to humans on embedding similarity, or ask an LLM judge? Unadjusted similarity confuses habit with adaptation. The Zelig work needed baseline normalization even to see that forums center on divergence (about −0.4). Reflexive function-word matching and deliberate formality matching are nearly uncorrelated (−0.00171; Exploring LSM in Online Communities), so a single similarity score blurs two channels. The matched-stimulus test reports embedding similarity next to $C^*$, which shows whether the simpler measure gives the same verdict.

Proposed Experiments and Decisive Checks

The first decisive check is the variance decomposition. We compare the as-published $\tau^2$ and $I^2$ with those obtained after re-scoring every corpus with $C^*$, separately for the power and quality questions. A drop of more than half supports the measurement-artifact claim. A drop below a fifth refutes it and shifts the burden to substantive moderators.

The second check is directional. We test whether the Wikipedia-administrator and forum-leader results still point in opposite directions under $C^*$. If they do, we test whether status type (elected role versus emergent leadership) explains the reversal.

The third check is the matched-stimulus LLM experiment. The pre-registered prediction is that the therapist–LLM gap in $C^*$ is less than half the published raw-LSM gap. Failure would strengthen, not weaken, the original deficit finding.

The fourth check is the outcome-perspective moderator. If client-rated outcomes show a smaller or negative matching–quality slope than expert-coded outcomes, consistent with the valence "synchrony trap" in crisis texting, outcome definition becomes a primary source of heterogeneity. That result would argue against treating these endpoints as interchangeable.

All analyses use leave-one-corpus-out evaluation. Prediction intervals are reported alongside pooled means, because the deliverable is a calibrated statement of how much a style-matching effect found in one corpus should be expected to change in the next.