Where the Talker Faces: Geometry-Aware Voice Onset Time Measurement from Smart Glasses

Voice onset time (VOT) is the gap, usually a few to about a hundred milliseconds, between the "pop" that releases a stop consonant like /p/ or /t/ and the moment the vocal folds start to buzz. Phoneticians use it to track dialect change, second-language learning, child acquisition, and motor-speech disorders. It is still measured almost entirely from clean recordings of word lists. Smart glasses now record thousands of hours of natural conversation, which could turn them into passive field instruments for phonetics. The obstacle is physical rather than statistical. The release burst is a brief high-frequency event, and a human mouth sends high frequencies mostly forward. When a conversation partner turns their head away from the wearer's microphones, the burst should fade relative to the low-frequency voicing, and an automatic measurer should mark the burst late and report a VOT that is too short. Our claim is that this error is predictable from where the talker is and which way they face relative to the wearer, and that a measurer told this geometry, and allowed to abstain when the geometry is bad, will give trustworthy VOT from egocentric recordings.

The two directions play unequal roles. VOT measurement is the core task. Egocentric spatial reasoning supplies one ingredient: estimating a partner's facing direction in the wearer's own coordinate frame. This is a perspective-taking problem that vision-language models are known to handle poorly, so we solve it geometrically. Nothing in the file measures sub-phonemic timing on wearable audio. The egocentric audio-visual systems it describes use speech only for direction finding, overlap detection, or transcription.

Core Claim

Automatic VOT error on wearable recordings of other people depends systematically on the talker's facing angle relative to the microphone and on their distance. The direction of the bias should be predictable from how the mouth radiates sound: shortened long-lag VOT when the talker faces away, and missed prevoicing at a distance or in babble. A measurer conditioned on video-derived facing angle and distance, trained with physically simulated directional radiation and room responses, and equipped with a learned abstention rule should beat geometry-blind noise-robust measurers on the tokens it keeps. Its abstention should also track true error better than audio-only quality estimates. The wearer's own speech, recorded at a fixed near-field position, acts as a within-session control where geometry is constant.

Prior Work and Its Role

The measurement methods come from three generations of VOT tools. Automatic Measurement of Positive and Negative Voice Onset Time split the problem into separate predictors for voicing lead and voicing lag. It found that errors in the sign of VOT inflate mean absolute error from 4.4 ms to about 10 ms. DeepVOT recast measurement as frame labeling with a bidirectional recurrent network, reaching 75.3% of tokens within 2 ms of experts against 50.5% for AutoVOT. Dr.VOT added sign prediction as an auxiliary task and adversarial training for invariance across corpora, reporting a 9.2-point absolute gain at 2 ms tolerance on unseen corpora. All three treat acoustic degradation as nuisance variation to be averaged out, and all were evaluated on word-list or tongue-twister corpora. We reuse Dr.VOT's backbone. The difference is that we model one structured source of degradation, talker geometry, explicitly instead of trying to make the network invariant to it. Invariance is the wrong goal when the distortion shifts the quantity being measured in a known direction.

The egocentric audio-visual work shows that wearable speech is a hard setting and supplies the sensing pieces we need. Audio Visual Speaker Localization from EgoCentric Views estimates where a speaker is, including when they are out of view, and releases a Unity simulator. Audio-Visual Approach For Multimodal Concurrent Speaker Detection detects overlap on EasyCom. WearVox reports accuracy drops of up to 15% from indoor to outdoor wearable audio. AVATAR finds lip-reading useless for the wearer's own speech. None of these models measures anything finer than a word. We take their location estimates as inputs and add what they omit: which way the talker is facing.

The spatial-reasoning literature both constrains the design and supplies part of the method. Seeing Through Their Eyes reports an average drop of about 35% when vision-language models must reason from another agent's viewpoint. ViewSpatial-Bench reports 34.98% for GPT-4o on multi-perspective localization. Both results rule out asking a vision-language model "is she facing me?". SymPL shows that explicit geometry built from detection, depth, and orientation estimation (GroundingDINO, Depth Pro, OrientAnything) makes such questions easy, with 91.50% on "facing" tasks. We reuse only the part of that pipeline that returns a talker's yaw and distance, not the full layout reasoning. TEP takes about 13 s per image, which warns against heavy pipelines. Because we need geometry only near stop tokens, not on every frame, the cost stays small.

Research Program

We first build the missing ground truth. In controlled sessions, pairs of speakers converse and read embedded stop-initial targets. One person wears multichannel camera glasses, and the partner wears a lapel microphone. Partners are prompted to vary distance and head yaw, including turning fully away. Expert phoneticians annotate the lapel audio, with Dr.VOT used as an assistant, to give reference VOT. The wearable-channel measurement is then scored against that reference. The sample should cover a language with a prevoicing contrast, since Tamim et al. document one in Palestinian Arabic, as well as an aspirating language, so that both regimes of error are tested. Stop place, vowel context, and speech rate are recorded because VOT depends on them and on the speaker (Speaker-specific Structure in German Voiceless Stop VOT).

Second, we characterize the error surface. We plot signed measurement error against facing yaw, distance, and estimated signal-to-noise ratio, separately for positive and negative VOT. This descriptive result is the first test of the idea.

Third, we build the measurer. We train Dr.VOT's architecture on clean corpora that have been augmented by convolving the audio with frequency-dependent mouth directivity patterns and simulated rooms at sampled geometries. The same geometry values are fed to the model as inputs. At test time, yaw and distance come from the SymPL-style orientation and depth estimate on the video frame nearest each token. When the talker is out of view, the model falls back on the array-based direction estimate with yaw unknown. A small head predicts each token's absolute error, and tokens above a threshold are rejected.

Baselines

The naive reference applies AutoVOT and DeepVOT unchanged to the wearable channel. It shows how far measurement falls from its clean-speech level, where DeepVOT reports 75.3% within 2 ms (DeepVOT). No reported result exists on Lacuna for any VOT measurer on egocentric wearable audio.

The strongest existing method is Dr.VOT (Dr.VOT). Its adversarial invariance training is designed precisely to handle unseen recording conditions. The file reports its gain as a 9.2-point absolute improvement at 2 ms over the previous best, with no absolute accuracy given. For a fair comparison we give Dr.VOT a stronger setup than its published one. It is retrained on the same augmented clean data, with the wearable-recording condition treated as an extra domain in its adversarial branch, using the same hyperparameter search budget and the same abstention head, which is fed audio features only. Beating it means a higher within-2 ms and within-5 ms rate, and fewer sign errors, at matched coverage on held-out speakers and rooms, with a lower area under the risk–coverage curve.

A beamformed Dr.VOT, where the multichannel front end is steered to the talker's estimated direction before measurement, tests whether better audio alone removes the need for geometry.

Two ablations isolate the key ingredient. The first trains with directivity augmentation but gives the model no geometry at test time. This separates the value of the training data from the value of the conditioning. The second replaces the estimated yaw with a shuffled yaw from another token in the same session. This checks that the model uses the geometry itself rather than some session-level correlate.

Why not just…?

Why not just make the measurer noise-robust and skip the geometry? This is the most serious alternative. Directivity augmentation alone may teach the network to cope with attenuated bursts. The first ablation above tests exactly this. If the geometry-blind model with the same augmentation matches the conditioned one, we will report augmentation as the contribution and drop the video input. Our reason to expect a gap is that invariance cannot undo a bias whose size depends on a hidden variable: a weak burst from a talker facing away and a weak burst from a quiet talker look the same in the audio but call for different corrections.

Why not just estimate recording quality from the audio and abstain on bad tokens? Signal-to-noise and direct-to-reverberant ratio estimates are cheap. They may capture most of what yaw captures, since facing away lowers direct-path energy. We test this with a nested comparison: does adding yaw and distance to an audio-only error predictor reduce its error-prediction loss and improve selective accuracy? If not, the visual branch is unnecessary. One place audio alone should fail is broadband background noise, which masks a burst regardless of which way the talker faces.

Why not just beamform toward the talker? Beamforming raises signal-to-noise for sound arriving from the talker's direction. It cannot restore high-frequency burst energy that the talker's head sent away from the glasses. Array quality is also tied to hardware layout, as WearVox notes. The beamformed baseline answers this empirically. If the error-versus-yaw curve flattens after beamforming, the directivity hypothesis is wrong.

Why not just ask a vision-language model which way the talker faces? Seeing Through Their Eyes and ViewSpatial-Bench show that such models fall back on the camera's viewpoint. We include one prompted vision-language model yaw estimate as a baseline input to show that orientation quality matters.

Proposed Experiments and Decisive Checks

The first decisive check is the error surface. If signed VOT error on partner speech does not vary with facing yaw beyond what distance and signal-to-noise explain, the core claim fails and the project reduces to robust VOT measurement. The wearer's own tokens should show no such dependence, which serves as a negative control.

The second check is held-out generalization. We train on some rooms and speakers and test on new ones. Selective accuracy within 2, 5, and 10 ms, prevoicing detection accuracy, and risk–coverage curves are compared against the Dr.VOT variants above.

The third check is scientific usefulness. For each speaker, the voicing contrast is summarized as the difference between mean lead and lag VOT, and we compute this separately from the lapel reference, the naive wearable pipeline, and ours. The question is whether our kept tokens recover the reference contrast without bias, while the naive pipeline shrinks it. Shrinkage would mimic the kind of contact-induced shortening reported by Kelly et al., so this is the error that would mislead a field linguist.

A secondary probe uses Fake Audio Detection with Microfeatures, which reports 25.2% equal error rate from VOT features alone on ASVSpoof2019 LA. We ask whether geometry-induced VOT bias on replayed wearable audio would raise false alarms for such low-cost detectors, and whether geometry correction reduces them.