Does a Speech Scorer Stay Valid Over a Month at 4,700 m?

Automatic speech scoring systems are certified by showing that their scores track expert human ratings about as closely as two experts track each other. That certification almost always comes from one recording session per speaker, made at sea level. High-altitude observatories, glaciology camps and Andean detector sites are where such a scorer would be most useful. Trained raters cannot be kept on site, and teams need a cheap daily check on whether personnel can still speak clearly and follow radio procedures. These are also the places where the certification is least likely to carry over. Thin air shifts vocal resonances, hypoxia changes breathing and articulation, and people acclimatize over days and weeks. Every existing altitude speech study has watched speakers for at most a few hours.

This note proposes a multi-week field study at a high-altitude research site with a narrow aim. We measure how agreement between machine and human scores changes over a deployment, rather than within a single exposure. We then test two ways to keep that agreement intact. The first is a scorer built on relational features: measures of how speech sounds relate to one another, which do not depend on their absolute frequencies. The second is a small weekly sample of recordings rated remotely by humans, used to catch drift. In two sentences: machine-human agreement measured on day one at altitude may not hold by week four, and nobody has checked. We will check, and show whether invariant features plus sparse remote human audits keep a scorer valid for the whole deployment.

Core Claim

The quantity of interest is the agreement ratio on day $t$ of deployment:

$$\rho(t) = \frac{r_{hm}(t)}{r_{hh}(t)}$$

Here $r_{hm}(t)$ is the Pearson correlation between machine scores and the mean human rating on recordings made on day $t$. $r_{hh}(t)$ is the correlation between two independent human raters on the same recordings. A ratio near 1 is what "human parity" means in this literature. Examples are the fluency scorer of Li et al. at 0.817 against a human ceiling of 0.831, and the structure-based pronunciation scorer of Suzuki et al. at 0.75 against 0.77.

We make three claims. First, $\rho(t)$ measured in the first two days at altitude is a poor predictor of $\rho(t)$ after three to four weeks. Second, scorers whose acoustic features are relational degrade much less than scorers built on absolute spectral templates. Third, a remote human audit of a few percent of recordings detects the cases where they do degrade. The new element is the longitudinal agreement curve itself, together with a deployable rule for when to distrust the machine. Neither exists for any extreme environment.

What Exists and What It Leaves Open

Three studies motivate the gap, and none measures human-machine agreement. Liberman et al. exposed 13 people to 3,000 m and heat for three hours in a chamber. They computed formant-trajectory correlation eigenspectra and classified stressed versus control speech with 0.90 accuracy, but only 0.65 for altitude alone. Their labels were chamber settings, not perceptual ratings. Laaridh et al. fused breath-sound features with near-infrared brain-oxygenation and ECG signals under normobaric hypoxia. They reached about 72.4% held-out accuracy on non-speech segments at a simulated 5 km, and also scored against gas settings over minutes. The HAPP dataset records blood-oxygen saturation (SpO₂), heart rate and mask gases at seven simulated altitudes. Each altitude was held for three minutes, and no audio was recorded. All three are acute and none has a human rating target. Our study adds both a multi-week horizon and a human rating target.

Scoring work supplies the methods. Van Moere et al. show that a radiotelephony-oriented scorer reaches an overall human-machine correlation of 0.94 across six ICAO subskills. That result was obtained in static, sea-level testing, and we borrow its task design rather than assume its validity. Suzuki et al. build speaker-invariant "speech structure" maps from distances between phoneme distributions. Their scorer stayed stable under simulated vocal-tract frequency warping while Goodness-of-Pronunciation scores collapsed. Liberman's eigenspectrum features were likewise reported to be insensitive to absolute formant shifts from air density. Together these motivate our hypothesis that relational features resist altitude, but both were tested only on synthetic or acute shifts.

Two results constrain the design. Liu et al. cut Whisper's error on one child corpus from 17.19% to 4.88% by fine-tuning, but performance degraded on another corpus. They also found the machine scored about three points stricter than clinicians. This warns that fine-tuning to the deployment site can buy local agreement at the cost of transfer, and it tells us to track systematic offsets as well as correlation. Kerz et al. found transcript-based complexity measures robust at an 18% word error rate. Hakkani-Tür et al. found content-unit recall correlating 0.82–0.95 with clinical scores despite elevated error in older speakers. Both predict that scores based on content will survive altitude better than scores based on acoustics, and we test that prediction directly.

The follow-up literature supplies the cautionary shape. In ADL+, broad cognitive gains at six months had largely faded three months later, with only processing speed holding up. Yang et al. note that 3–6 month windows misclassify late recurrences. By analogy, early agreement may reflect a transient state rather than a stable property. van Sluis et al. show that multi-year acoustic tracking of the same voices is feasible, which is the precedent for repeated-measures voice data.

Research Program

The site will be a staffed high-altitude station. Chacaltaya at 4,740 m, one of the SWGO candidate sites, is the model. That survey notes that reduced human work capacity above 4,400 m drives designs needing little on-site intervention, and the same constraint is why on-site raters are impractical.

We will recruit about 40 adults on planned stays of four weeks. Each participant gets one sea-level baseline session before ascent, daily ten-minute sessions at altitude, and a post-descent session two weeks after return. Each session contains a read-aloud passage, a radiotelephony-style readback task modelled on the ICAO dimensions, an immediate story retelling scored for content units, and a fingertip SpO₂ and heart-rate reading taken at the same time.

Audio goes over the station uplink to a pool of at least six trained raters at sea level. They score intelligibility, fluency and pronunciation without knowing the recording day or altitude. Every recording in a stratified 20% subsample receives two independent ratings, so that $r_{hh}(t)$ can be estimated on each day. Raters also rescore a fixed set of anchor clips every week, which separates drift in the machine from drift in the raters.

We compare scorers that differ mainly in feature type. The absolute-feature scorers use MFCC or GOP inputs and a Whisper transcript. The relational scorers use structure maps and eigenspectra. The content scorer uses unit recall from transcripts. For each scorer we estimate the curve $\rho(t)$ and the mean machine-minus-human offset over time. A mixed model with participant random effects separates within-person drift from cohort composition.

Attrition matters here. People who descend early with altitude sickness are probably the most affected speakers, and dropping them would flatter every scorer. We record departure reasons, report results with and without inverse-probability weighting on baseline SpO₂ and symptoms, and keep the recordings made before each person departed.

Finally, we simulate a sparse audit rule. Each week a random 2–5% of recordings is rated, and the rule raises a flag if the audited $\rho$ falls below a preset threshold. We measure how quickly and how reliably the rule detects the drift that the full ratings reveal.

Baselines

The do-nothing reference is the scorer as it would actually be deployed: an off-the-shelf sequential fluency scorer trained at sea level, in the style of Li et al.. It takes phone identities, silence tokens and durations from forced alignment, feeds them to a two-layer bidirectional LSTM, and is validated once on day one. Li et al. report a PCC of 0.817 against a human ceiling of 0.831 on their read-aloud benchmark. It is the strongest published human-parity scorer in the file and the most likely to be picked up, which makes it the strongest existing competitor.

We will retrain it on the public LibriSpeech portion plus whatever non-native data we can license. Because we cannot reproduce their 5,000-hour training set, we will report its sea-level PCC on our baseline sessions as its effective starting point. Our proposal beats it if the relational scorer keeps $\rho(t) \ge 0.9$ through week four while this baseline falls below that level, with non-overlapping participant-bootstrap intervals.

The second baseline is site adaptation: LoRA fine-tuning of Whisper on the first three days of altitude recordings, following Liu et al.. All scorers get the same audio, the same human-label budget of the first three days, and the same hyperparameter search budget. Beating it requires better week-four agreement and better post-descent agreement. Post-descent is where the generalization trap Liu et al. describe should show up.

The third baseline uses physiology alone: a regression from SpO₂ and heart rate to human ratings. It is fitted on the same three days.

The key ablation holds the relational scorer's architecture and training fixed and swaps its features for absolute MFCC/GOP inputs. This isolates feature invariance as the cause of any stability.

For context, there is no reported result on Lacuna for human-machine score correlation on speech recorded at altitude. The altitude numbers in the file (Liberman's 0.65 and Laaridh's 72.4%) are classification accuracies against exposure condition, so they are not comparable baselines for $\rho$.

Why not just…?

Why not just use the pulse oximeter? HAPP shows that SpO₂ falls with pressure and correlates with heart rate (r < −0.4). It is cheap and needs no raters. This is a genuine threat. If SpO₂ predicts human-rated communication quality as well as any speech scorer does, the speech scorer is not worth deploying for this purpose. The physiology-only baseline settles it directly. We will report the incremental variance in human ratings that speech scores explain beyond SpO₂. If that increment is negligible, we will report the negative result and redirect the work to the audit question.

Why not just fine-tune the scorer on site data? Liu et al.'s cross-corpus degradation suggests that adapting to early-exposure speech fits a state that acclimatization then changes. Doing so also spends scarce human labels; their paper notes seven to eight minutes of expert labour per minute of audio. The LoRA baseline measures this directly. If it holds agreement as well as relational features, then cheap periodic re-adaptation is the answer, and we will say so.

Why not just run a longer chamber study? Chambers control altitude but not weeks of sleep loss, cold, workload and acclimatization together, and multi-week chamber stays are rarely feasible. The Liberman and HAPP protocols are capped at hours for this reason. The field site is what makes the follow-up duration real. Its cost is less control, which the per-day SpO₂ readings partly offset.

Why not just score content and ignore acoustics? Kerz et al. and Hakkani-Tür et al. suggest content measures survive transcription noise. They may well hold agreement best, and the content scorer is included for exactly this reason. They cannot, however, capture the slurring and breath-driven disfluency that human raters penalize. Our per-dimension analysis will show which rating dimensions each scorer actually tracks.

Decisive Checks

The study succeeds or fails on four results. The first is whether the day 1–2 value of $\rho$ predicts the week-4 value, measured as the correlation of per-participant agreement across the two periods. A weak relationship would falsify the practice of one-off validation. The second is whether the relational-versus-absolute ablation shows a gap in $\rho$ at week four that is larger than the gap at day one. The third is whether speech scores add explained variance in human ratings beyond SpO₂. The fourth is the audit-budget curve: the smallest weekly fraction of human-rated recordings that detects a 0.1 drop in $\rho$ within one week at least 80% of the time.

Each of these can come out against us, and each outcome changes what a remote station should deploy.