When a person listens to speech, their brain activity tracks the sound with a delay of a few hundred milliseconds. A model that has to judge how a listener perceived that speech, for example how natural a synthetic voice sounded or how well the listener followed it, should therefore care most about the relationship between the sound and the brain response. The absolute shape of either signal matters less. Current audio–EEG systems do not represent that relationship directly. They encode audio and electroencephalography (EEG, scalp recordings of electrical brain activity) in separate networks and merge the two only at the end, hoping the fusion layer will find the coupling on its own. We propose to compute the coupling explicitly, before any learning, as a small two-dimensional image. A plain CNN then reads that image.
The tool for building the image is the Gramian Angular Field (GAF). A GAF maps each value of a rescaled series to an angle and fills an image cell $(i,j)$ with the cosine of the sum, or the sine of the difference, of the angles at times $i$ and $j$. So far the literature computes GAFs only within one series. We define a cross-modal version that pairs the speech envelope at time $t$ with the EEG at time $t+\ell$. We keep only lags $\ell$ inside the window where neural tracking actually happens. The core claim is that this time-by-lag coupling image gives better cross-subject predictions of perceived speech quality and attention than late fusion or cross-attention of separately encoded streams. The reason it should is that the coupling image is relational: the stimulus is shared across listeners, and normalizing each window removes much of the per-subject amplitude variation that makes EEG models fail on new people.
What exists and where this sits
Two lines of work almost meet but do not. The first uses audio together with EEG. Combining Audio and Brain Activity for Predicting Speech Quality runs a mel-spectrogram CNN on the audio and a separate CNN on time–frequency EEG. It fuses the two scalar predictions with a small multilayer network and evaluates only subject-dependently. This paper motivates our work: it shows that EEG adds information over audio alone, but its fusion never lets the network compare the two signals in time.
The second line applies GAFs to EEG. Deep Neural Networks on EEG Signals to Predict Auditory Attention Score Using Gramian Angular Difference Field stacks 14 per-electrode difference-field images and regresses attention. It is purely unimodal and never sees the speech it is meant to explain.
The GAF construction itself comes from Imaging Time-Series to Improve Classification and Imputation, which defines the summation and difference fields within a single series. We reuse that construction, but compute it across two series and restrict it to a lag band.
Other work fuses a raw sequence view with an image view of the same sensor rather than of two physical sensors. An intelligent diagnosis method for PMSM radial misalignment based on current signal and Swin-BiGRU multimodal fusion pairs a 1D current stream with Markov-transition images through cross-attention. Cross-Modal Knowledge Transfer in Time Series AI via Large Vision Models combines pretrained vision models on imaged series, including GAF, with numerical models. Its finding that vision models over-favor periodic texture and miss trends is a constraint on our design. Our lag-band image has no long time axis to fold periodic texture into, but we still check for that failure mode.
Gramian Multimodal Representation Learning and Alignment is the only other "Gramian multimodal" work in the file, and it is not the same thing. It aligns learned embeddings by minimizing Gram-matrix volume, whereas we build the input from raw synchronized signals.
Core Claim and Construction
Let $a_t$ be the speech amplitude envelope and $e^{(c)}_t$ the EEG on electrode $c$. Both are resampled to a common rate (for example 64 Hz) and min-max rescaled to $[-1,1]$ within each analysis window. Each rescaled value becomes an angle, $\phi_t = \arccos(a_t)$ for audio and $\psi^{(c)}_t = \arccos(e^{(c)}_t)$ for electrode $c$. The banded cross-modal fields are
$$S^{(c)}_{t,\ell} = \cos\big(\phi_t + \psi^{(c)}_{t+\ell}\big), \qquad D^{(c)}_{t,\ell} = \sin\big(\phi_t - \psi^{(c)}_{t+\ell}\big), \qquad 0 \le \ell \le L.$$
Here $t$ indexes time in the window and $\ell$ is the neural delay. $L$ covers about 500 ms, the range in which EEG follows speech. Restricting to this band removes the usual quadratic cost of GAFs: a 10 s window becomes a 640×33 image per electrode and field type rather than 640×640. Stacking all electrodes and both field types gives a compact tensor that a small 2D CNN can read. In that tensor, a vertical stripe at a fixed lag means the brain is consistently tracking the sound at that delay.
We are explicit about what this does and does not add. Expanding the cosine shows that $S$ equals $a_t e_{t+\ell} - \sqrt{1-a_t^2}\sqrt{1-e_{t+\ell}^2}$. The full field is therefore a rank-two outer product, and it contains no information beyond the two series. Any benefit has to come from where it puts information: lagged pairwise interactions become local, translation-invariant patterns that a small convolutional network learns efficiently from a few hundred labeled trials. This is precisely what makes the idea testable against simpler relational encodings, as described below.
Research Program
We will build the cross-field tensor for every audio–EEG trial and train a compact CNN on it, alone and alongside an audio-only spectrogram branch. Training a CNN from scratch is the default because the labeled data are small. A pretrained vision backbone is a secondary arm, tested because Cross-Modal Knowledge Transfer in Time Series AI via Large Vision Models reports that such backbones transfer to imaged series.
We will evaluate under leave-one-subject-out splits, which is the regime where Combining Audio and Brain Activity for Predicting Speech Quality was not tested and where subject variance is the known failure. We will also report the subject-dependent split, for direct comparability with that paper.
The analysis has three goals. The first is to show whether the network's evidence concentrates at physiologically sensible lags; we will test this by occluding lag bands. The second is to find whether the benefit comes from the cross term or simply from imaging EEG. The third is to measure whether the relational input shrinks the gap between subject-dependent and cross-subject performance.
Baselines
The naive reference point is the audio-only mel-spectrogram CNN. It ignores the listener entirely. Combining Audio and Brain Activity for Predicting Speech Quality reports RMSE 0.862 for it on PhySyQX mean opinion score prediction. We will retrain it under our splits rather than copy the number.
The strongest existing method is that paper's late-fusion model, which combines independent audio and EEG CNNs through a two-hidden-layer network. It reports RMSE 0.732 ± 0.017 on PhySyQX, subject-dependent. It is the strongest competitor because it is the only published system that already exploits both signals on this exact task. We will reimplement it and give it the same input windows, the same splits, the same number of hyperparameter trials, and a matched parameter count. Beating it means a lower RMSE on the subject-dependent split, outside the seed-to-seed spread. More importantly, it means a clearly lower RMSE under leave-one-subject-out, checked with a paired Wilcoxon test over subjects, which is the same test that paper used.
A stronger feature-level fusion baseline follows the Swin-BiGRU design of An intelligent diagnosis method for PMSM radial misalignment based on current signal and Swin-BiGRU multimodal fusion. It applies bidirectional cross-attention between an audio sequence branch and an EEG branch, and in principle lets the model learn lagged coupling itself. We also add the adaptive gradient blending of XSleepNet: Multi-View Sequential Model for Automatic Sleep Staging to the late-fusion model. XSleepNet reports 1.2–1.4% accuracy gains over naive fusion in sleep staging, so this check rules out the possibility that our gains merely come from better-balanced training of the two branches.
On Phyaat, the reference is the 14-channel difference-field 2D CNN of Deep Neural Networks on EEG Signals to Predict Auditory Attention Score Using Gramian Angular Difference Field, with reported validation MAE 29.43, against an SVM at 29.65. The margin between those two is small, so we will report seed variance before claiming any improvement.
Three ablations isolate the key ingredient. The first replaces the cross fields with per-stream GAFs, an audio GAF plus stacked EEG GAFs concatenated, so there is no cross term; this tests whether coupling matters or imaging alone does. The second feeds the network a mismatched stimulus, pairing each EEG trial with a different trial's audio. If performance survives this pairing, the model is not using coupling at all. The third is the lagged-correlation family described next.
Why not just…?
Why not just compute a lagged cross-correlation map or a temporal response function instead of a Gramian field? This is the most serious threat, because the rank-two identity above shows the cross field is a nonlinear reparameterization of the lagged product $a_t e_{t+\ell}$. We will therefore test, on identical windows and the same CNN, a raw lagged-product map, a sliding-window cross-correlation map, and linear temporal-response-function weights fed to a small regressor. If the raw product map matches the cross field, the honest conclusion is that the gain comes from input-level coupling imaging, not from the angular encoding. The core claim survives that outcome, and the Gramian parameterization becomes an interchangeable choice, which we would report as such. The angular form can only claim credit if it beats the product map, plausibly because the $\sqrt{1-x^2}$ terms emphasize mid-range values and reduce sensitivity to EEG outliers.
Why not just use stronger late fusion or cross-attention between separately encoded streams? On paper these can learn the lag structure. But Combining Audio and Brain Activity for Predicting Speech Quality fused only scalar outputs. Learning delay-specific alignment from roughly 21 subjects × 36 stimuli is a hard sample-efficiency problem for attention, and EEG in that paper is clearly the weaker signal (RMSE 0.984 alone). The cross-attention and gradient-blending baselines above answer this directly. We will also run a learning curve over the number of training subjects, because our claim predicts the advantage is largest when data are scarce.
Why not just image the EEG on its own, as with stacked difference fields or spectrograms, and let fusion handle the audio? That is precisely the setting of Deep Neural Networks on EEG Signals to Predict Auditory Attention Score Using Gramian Angular Difference Field and of the time–frequency EEG branch in the speech-quality work. The per-stream-GAF ablation and the mismatched-stimulus control test it head-on. If matched and mismatched audio yield the same error, the cross field adds nothing and the proposal fails.
Proposed Experiments and Decisive Checks
PhySyQX is the primary benchmark. Its targets are mean opinion scores for 36 stimuli from human speakers and seven text-to-speech systems, rated by 21 listeners with synchronous EEG. We report RMSE and Pearson correlation under both subject-dependent and leave-one-subject-out splits, with five seeds each. Phyaat is the secondary benchmark, with attention-score MAE across its 25 subjects. Using it requires the speech stimuli to be recoverable for each trial; the research file does not confirm this, so it is the first thing we check, and if the stimuli are unavailable Phyaat drops to an EEG-only reference.
No reported result was found on Lacuna for any cross-subject evaluation on PhySyQX. Our leave-one-subject-out numbers for the reimplemented late-fusion model will therefore be the reference there.
Three outcomes decide the question. The first is whether the cross field beats reimplemented late fusion under leave-one-subject-out, with the gap larger than in the subject-dependent split. That would be evidence that relational inputs generalize across listeners. The second is whether the mismatched-stimulus control erases the gain, which confirms the model reads coupling. The third is whether the angular field beats the raw lagged-product map, which determines if the Gramian encoding specifically earns its place.
A further sanity check is that occluding lags between 50 and 300 ms should hurt more than occluding lags outside that range. This is where speech-tracking responses are typically expected; if attributions concentrate at implausible lags, we treat the result as suspect regardless of RMSE.
References
- Combining Audio and Brain Activity for Predicting Speech Quality
- Deep Neural Networks on EEG Signals to Predict Auditory Attention Score Using Gramian Angular Difference Field
- Imaging Time-Series to Improve Classification and Imputation
- An intelligent diagnosis method for PMSM radial misalignment based on current signal and Swin-BiGRU multimodal fusion
- Cross-Modal Knowledge Transfer in Time Series AI via Large Vision Models
- Gramian Multimodal Representation Learning and Alignment
- XSleepNet: Multi-View Sequential Model for Automatic Sleep Staging