When people listen to a story, their brain activity in the slowest rhythms (delta, roughly 1 to 4 cycles per second) rises and falls in step with the speech. Neuroscientists disagree about what drives this. One view is that it follows sound: pauses, pitch resets and phrase-final lengthening. The other is that it follows meaning, meaning the listener's internal parse of phrases and discourse. This proposal brings in a tool from a separate field, implicit temporal inference: natural-language-processing (NLP) methods that recover when events happened from text that never states it, for example "she had already left" or a flashback with no date. We use these tools to build a variable that sound cannot supply. That variable marks the moments in a spoken story when the listener must move the story's clock, such as jumping back to an earlier event, skipping ahead, or holding time still. We then ask whether slow brain rhythms recorded with magnetoencephalography (MEG, a non-invasive recording of the brain's magnetic fields at millisecond resolution) respond to these timeline updates after sound and word-level language-model predictors are removed.
In two sentences: we apply an event-ordering extractor, validated on human annotations, to the transcripts of spoken stories, and mark where the story's chronology departs from the order of telling. We then test whether delta-band MEG responses carry information about those departures, especially the ones signalled by no explicit word and no acoustic boundary. If they do, slow speech-brain synchronization tracks a mental timeline and not only acoustic phrasing. If they do not, that is a clean negative result on a long-running debate.
Core Claim
The open problem this targets is stated directly in the tracking literature. Complex Mapping between Neural Response Frequency and Linguistic Units in Natural Speech highlights that low-frequency neural responses do not map cleanly onto linguistic units in natural speech, so it remains unclear whether delta tracking reflects abstract structure or acoustic boundaries. The evidence for a top-down contribution is real but indirect, and it motivates this work.
- Speech Rhythms and Multiplexed Oscillatory Sensory Coding in the Human Brain shows that delta and theta tracking, and the coupling between rhythms, weaken for backward speech. Backward speech destroys intelligibility, but it also changes acoustic edges.
- Neural Tuning to Low-Level Features of Speech throughout the Perisylvian Cortex finds frontal areas tuned to slow modulations at word-to-sentence scales. That tuning was measured against acoustic modulation, not meaning.
Neither manipulation isolates a variable that is purely interpretive. Story chronology is such a variable. The same pause can precede a flashback or a continuation. A flashback can also arrive with no pause and no connective, signalled only by event knowledge: in "He paid the bill. The fish had been excellent," the eating happened before the paying.
Our claim is that a story-time discontinuity variable, built from an implicit temporal relation extractor, explains held-out delta-band MEG activity beyond three nuisance sets: acoustic envelope and edge regressors, prosodic boundary regressors, and contextual language-model embeddings. The decisive version of this claim concerns implicit shifts, those lacking both explicit temporal markers and acoustic boundary cues.
The roles of the two directions are deliberately unequal. Speech-brain synchronization supplies the question and the data. Implicit temporal inference supplies the measurement instrument.
Research Program
Data. The main dataset is SMN4Lang, described in A synchronized multimodal neuroimaging dataset for studying brain language processing. Twelve listeners heard 60 naturalistic Mandarin stories, about six hours in total, with MEG recorded and with millisecond word timestamps, dependency trees and BERT/GPT-2 embeddings provided. The authors verified delta- and theta-band tracking through inter-subject correlation, which tells us the MEG carries story-locked signal at the frequencies we care about.
Mandarin is a useful test case rather than a nuisance. It has no grammatical tense, so chronology is inferred mostly from aspect particles, adverbs and event knowledge. That gives an unusually large share of implicit shifts.
Step 1: human annotation of story time. Two annotators label every clause-level event in the 60 transcripts. Each event is anchored at its start point relative to the previous narrated event, following the MATRES convention of start-point relations (before, after, equal, vague). We apply logical closure, the transitivity reasoning that Automating Temporal Annotation with TARSQI shows multiplies the number of recoverable links fourfold, to derive a partial order.
From that partial order, each event onset receives one of four codes: continuation, backward jump, forward jump, or unresolved. Annotators also mark whether the shift is explicitly signalled by a connective, a time adverb or an aspect particle such as 过. This annotation is the gold standard. At this corpus size it is feasible in a few annotator-weeks.
Step 2: an automatic extractor. The extractor tests whether the variable can be produced at scale without hand labels and supports later replication. It has three parts.
- The encoder is a multilingual model with linguistically informed pretraining in the style of LEAF: Linguistically Enhanced Event Temporal Relation Framework. LEAF continually pretrains on syntactically mined temporal pairs.
- Relations are read out over dependency paths, as in Extracting Temporal Event Relation with Syntax-guided Graph Transformer, which attends along dependency paths between event tokens. SMN4Lang already ships verified dependency trees.
- A global integer-linear-programming step, as in Joint Reasoning for Temporal and Causal Relations, enforces transitivity and cause-before-effect.
We train on MATRES and adapt on half of the annotated stories. The extractor also yields a timeline-uncertainty score: the entropy of its relation distribution at each event.
Step 3: brain encoding. We fit temporal response functions (linear models that predict each MEG sensor's band-limited signal from time-lagged stimulus features) separately in delta, theta and gamma bands. Predictors enter in nested sets:
- Acoustic: broadband envelope, envelope edges and pitch.
- Prosodic boundaries: pause duration, pitch reset and syllable lengthening, extracted from the audio.
- Lexical and contextual: word onsets, GPT-2 surprisal, and BERT layer-7 embeddings reduced in dimension. Layer 7 is the layer SMN4Lang reports as best correlated with brain activity.
- Story-time codes: impulses at event onsets for each shift type, plus timeline uncertainty.
Models are fit with leave-one-story-out cross-validation within each subject. We report the gain in held-out prediction correlation that set 4 adds over sets 1 to 3.
Predictions. If delta tracking indexes timeline updating, three things should hold:
- The gain is concentrated in delta, not theta or gamma.
- The gain is carried by backward and forward jumps, not continuations.
- It survives when restricted to implicit, acoustically unmarked shifts.
Speech Rhythms and Multiplexed Oscillatory Sensory Coding in the Human Brain reports stronger slow-rhythm entrainment in the right auditory cortex. We therefore predict sensor topographies with right-temporal and frontal emphasis, using source reconstruction from the provided structural MRIs.
Baselines
Naive reference. The do-nothing reference is the acoustic-only response model, using the envelope as the main scaffold. Dynamic time-locking mechanism in the cortical representation of spoken words shows the envelope gives better reconstruction than other spectral properties. Every gain is reported relative to this model, and it also sets the noise ceiling we compare against.
Strongest competitor. This is the full acoustic, prosodic and contextual language-model encoding model. It is the most serious rival because contextual embeddings already predict activity across temporal and frontal language networks in SMN4Lang. Any timeline effect could simply be something the language model already encodes. It receives the same cross-validation folds, the same regularization search and the same number of hyperparameter trials as our full model. We also give it extra capacity: extra principal components of the embeddings matching the number of story-time regressors, so the comparison is not won by parameter count.
No reported result was found on Lacuna for held-out MEG encoding accuracy on SMN4Lang. SMN4Lang reports only that BERT layer 7 correlates best. We will therefore establish this baseline's number ourselves before testing additions.
Beating it means a statistically reliable increase in held-out delta-band prediction correlation in at least 8 of 12 subjects. Reliability is tested against a permutation null in which the story-time labels are shuffled among event onsets within each story. This preserves the number and timing of events and destroys only the chronology.
Ablation of the key ingredient. The ablation replaces the chronology codes with a generic event-onset regressor that has the same onsets but no shift information. This isolates whether chronology matters beyond "a new event started."
Extractor reference points. The extractor matters only as a measuring instrument. Its known operating points on English are an F1 of 52.1 on TB-Dense with causal joint reasoning (Joint Reasoning for Temporal and Causal Relations), and a MATRES contrast-set consistency of 49.8% against a previous 40.6% (Extracting Temporal Event Relation with Syntax-guided Graph Transformer). We will report its agreement with the human Mandarin annotations. The brain analysis uses human labels if the extractor falls well short of human agreement.
Why not just…?
Why not just use language-model embeddings, which already encode context? This is the real threat, and the proposal is built to settle it rather than argue around it. A deep model's state may well track chronology implicitly. If so, adding explicit story-time codes will give no unique variance, and we will report that. We would then probe whether the embeddings themselves linearly encode shift type. That would locate the timeline signal inside the language-model predictor instead of declaring the hypothesis false. The ablation against a generic event-onset regressor guards against the opposite error: crediting chronology for what is merely event-boundary structure.
Why not just prompt a large language model to order the events? Sloppy labels would weaken the measuring instrument. LEAF: Linguistically Enhanced Event Temporal Relation Framework reports that three-shot ChatGPT trails a structured extractor by over 37 F1 on MATRES-style ordering. Large models also tend to default to the order of telling, which is exactly the confound we need to break. We nevertheless include an LLM labeler as a cheap comparison against the human gold labels, since its quality on these Mandarin stories is an empirical question.
Why not just regress out pauses and pitch resets and look at phrase boundaries? That is the existing debate restated. Phrase boundaries and acoustic boundaries co-occur almost perfectly, which is why Complex Mapping between Neural Response Frequency and Linguistic Units in Natural Speech finds the mapping hard to resolve. Story-time shifts can be split by whether they coincide with an acoustic boundary, which phrase boundaries cannot. The comparison of implicit, unmarked shifts against matched continuations is the test those analyses lack.
Why not use brain signals to improve the temporal extractor, as earlier cognitive-signal work did? Bridging between Cognitive Processing Signals and Linguistic Features via a Unified Attentional Network predicted sentence tense from reading EEG. However, 12 listeners provide far too little supervision compared with text corpora, and brain data are unavailable at test time. The more defensible direction is the reverse: NLP structure as a probe of the brain.
Decisive Checks
The central test is the implicit-shift contrast. We select backward and forward jumps with no explicit marker and no pause or pitch reset above the corpus median. Each is matched to continuation events on pause duration, word frequency, GPT-2 surprisal and position in the story. We then compare delta-band responses, measured as cross-validated response amplitude and as delta phase concentration across subjects in a window after the event onset. A reliable difference with right-temporal and frontal emphasis supports timeline tracking. No difference while the acoustic and language-model predictors still perform normally argues that delta tracking does not index story chronology beyond what sound and words provide.
Two secondary checks follow.
- Graded response. Timeline uncertainty should predict delta response magnitude in a graded way among unresolved events.
- Out-of-order explicit markers. The effect for explicitly marked shifts should precede the effect for implicit ones relative to event onset. Explicit markers allow earlier updating, which echoes the "leading" versus "following" dissociation reported in Leading and Following: Noise Differently Affects Semantic and Acoustic Processing during Naturalistic Speech Comprehension.
A small English check uses the Kymata Soto dataset (Kymata Soto Language Dataset): 20 English listeners and about 7 minutes of podcast speech. It will contain too few shifts for a standalone test, so it is reported only as a direction-of-effect check with the automatic extractor.
References
- Complex Mapping between Neural Response Frequency and Linguistic Units in Natural Speech
- Speech Rhythms and Multiplexed Oscillatory Sensory Coding in the Human Brain
- Neural Tuning to Low-Level Features of Speech throughout the Perisylvian Cortex
- A synchronized multimodal neuroimaging dataset for studying brain language processing
- Automating Temporal Annotation with TARSQI
- LEAF: Linguistically Enhanced Event Temporal Relation Framework
- Extracting Temporal Event Relation with Syntax-guided Graph Transformer
- Joint Reasoning for Temporal and Causal Relations
- Dynamic time-locking mechanism in the cortical representation of spoken words
- Bridging between Cognitive Processing Signals and Linguistic Features via a Unified Attentional Network
- Leading and following: Noise differently affects semantic and acoustic processing during naturalistic speech comprehension
- Kymata Soto Language Dataset: an electro-magnetoencephalographic dataset for natural speech processing