Automatic scores for AI-generated music often disagree with listeners. Distribution metrics such as Fréchet Audio Distance (a distance between the statistics of generated and real audio embeddings) reward drum hits with realistic timbre even when they fall off the beat. Statistical measures also punish deliberate surprises, such as a jazz player's chromatic approach notes, that experts enjoy. This note proposes an evaluator built on a different idea. It does not ask how surprising the music is. It asks how often the music makes a confident structural promise and then breaks it. The music sets up a downbeat that never lands, a phrase that never closes, or a chorus that is implied but never returns.
The mechanism comes from anticipatory parsing, the study of how human and machine parsers project expected structure forward as input arrives. A forward-looking model listens left to right and keeps a "fringe" of open predictions about upcoming beats, harmony and section repetition at several time horizons. Coherence is then scored as the confidence-weighted rate at which those predictions are violated. Surprise that the music later resolves is tracked separately. The bet is that listeners penalize broken commitments, not surprise as such. Plain surprisal or perplexity mixes the two, which would explain part of the gap between objective metrics and human judgment.
Core Claim
Our claim is that total surprisal under a predictive music model can be split into two parts. One is surprisal at moments where the model was confident, which we call commitment violation. The other is surprisal where the model was uncertain or where the music later restores the expected structure. Only commitment violation, measured at bar-to-section horizons, should track human ratings of coherence and structural clarity. Resolved surprise should be neutral or even positive for memorability.
The language-side evidence for this split is that raw predictability is the wrong target. In Transformers Can Model Human Hyperprediction in Buzzer Quiz, GPT-2 surprisal predicts human reading times when quiz-takers anticipate whole clauses. But fine-tuning past roughly 1,500–2,000 sentences lowers perplexity while reducing fit to humans. A better predictor is not automatically a better model of the listener. On the music side, the jazz result in Improving Automatic Jazz Melody Generation by Transfer Learning Techniques shows that statistical outliers can be preferred by experts. The drum-loop analysis in A Benchmarking Initiative for Audio-domain Music Generation using the FreeSound Loop Dataset shows the reverse failure: realistic texture without meter scores well.
Plain surprisal cannot separate these cases. A decomposition into commitments and violations, borrowed from incremental parsing, can.
What Exists and What This Changes
Anticipation has so far served generation. Anticipatory Music Transformer interleaves future control events into the token stream with a lookahead offset, so that a symbolic model sees upcoming constraints within its recent context. It produces 15-second accompaniments that listeners could not reliably tell apart from human ones. ProGress generates music from a global hierarchical harmonic skeleton using graph diffusion. Both are symbolic generators. Neither scores arbitrary audio. ProGress's hierarchy is also computed globally rather than left to right. We reuse the lookahead idea as the evaluator's forecasting structure and invert its purpose, from steering generation to auditing it.
The incremental-parsing literature supplies the bookkeeping. PLTAG with role propagation (Incremental Semantic Role Labeling with Tree Adjoining Grammar) keeps explicit placeholders for heads and arguments not yet seen. It reaches 78.38 F1 against 52.50 for a non-predictive incremental parser. The same paper reports that it over-generates roles for frequent items, a warning that confident predictions must be calibrated before violations carry meaning. Brain-imaging work (Can fMRI reveal the representation of syntactic structure in the brain?) finds that representations of incomplete subtrees explain activity in language regions. That supports the premise that open predictions, not just completed structure, are psychologically real. We do not need a grammar of music. We need the parser's distinction between open slots and filled ones.
Learned evaluators are the competitors. CMI-RewardBench trains two-tower reward models on expert preferences. Its musicality head correlates about 0.85 with its prompt-alignment head, and it needs roughly 120 seconds of context before correlation with humans is good. MuseCritic writes a natural-language critique of the whole song before scoring. That is expensive, and it looks back over the song rather than following it incrementally. SongEval provides 2,399 full songs rated on five dimensions. It notes that these dimensions are entangled, which motivates a signal that targets structure specifically.
Research Program
We first build a structural forecaster. Audio is cut into beat-synchronous frames. At each frame we compute automatically extracted descriptors that stand in for musical structure. These are beat and downbeat position, a pitch-class (chroma) summary standing in for harmony, and a similarity code recording which earlier segment, if any, the current material repeats.
A causal transformer reads the descriptor history up to time $t$. Its frame encoder is a pretrained audio representation, held fixed. For each horizon $h$ in {1 beat, 1 bar, 4 bars, 8 bars, 32 bars}, the transformer outputs a distribution $p_{t,h}$ over the descriptor at $t+h$. This is the lookahead offset of the Anticipatory Music Transformer, turned into an explicit forecast target. Keeping horizons separate matters because short-horizon prediction is mostly meter, while long horizons are where macro-structure and the 120-second effect from CMI-RewardBench live. We train on unrated real music: rendered Lakh MIDI plus real audio outside the evaluation sets. We then recalibrate with temperature scaling so that a stated confidence matches observed accuracy on held-out real music. This step addresses the PLTAG over-generation problem.
Second, we score commitment. Let $s_{t,h} = -\log p_{t,h}(y_{t+h})$ be the surprisal of the observed descriptor $y_{t+h}$. Let $c_{t,h} = 1 - H(p_{t,h})/\log K$ be the forecast's confidence, where $H$ is entropy and $K$ is the number of descriptor classes. The song-level commitment-violation score at horizon $h$ is
$$V_h = \frac{1}{T}\sum_{t=1}^{T} c_{t,h}\, s_{t,h},$$
where $T$ is the number of frames. Confident wrong forecasts dominate this sum, and uncertain moments contribute little. A short-horizon surprise counts as resolved if the long-horizon forecasts made before it still come true afterward. That is the chromatic-enclosure pattern: a local deviation that lands where the structure promised. We report the rate $R$ of resolved surprises separately from $V_h$. The evaluator is the vector $(V_1,\dots,V_{32\text{ bars}}, R)$, mapped to human ratings with a small linear model fitted on training splits. Because the mapping is linear, every score can be traced back to specific horizons and timestamps. A listener can be pointed to "a section return expected at 1:42 that never arrived," which neither scalar regressors nor post-hoc critiques provide.
Third, we analyze when prediction stops tracking people. The hyperprediction result predicts that a forecaster trained longer, or a larger one, may become so accurate that broken commitments look ordinary to it. We therefore save forecaster checkpoints across training and plot correlation with human ratings against forecaster perplexity. The aim is to test whether music shows the same peak-then-decline pattern as reading times.
Baselines
The naive reference point is the distribution and embedding metrics the field already reports. These are Fréchet Audio Distance and CLAP text-audio similarity, computed per song against a matched real-music reference set. The drum-loop study above shows they reward texture over meter. We expect them to sit near zero correlation with SongEval Coherence. Any useful structural metric must clear them by a wide margin.
The key ablation, and the competitor we worry about most, is mean surprisal from the very same forecaster: $V_h$ with every $c_{t,h}$ fixed to one. It uses the same data, model, checkpoints and linear readout, so it isolates the single ingredient we claim matters, which is weighting surprise by commitment and splitting off resolution. A second ablation keeps only the 1-beat and 1-bar horizons. It tests whether long-horizon expectation, rather than meter tracking alone, carries the signal.
The strongest existing method on SongEval is the UTMOS-style regressor trained on SongEval, which reports a Pearson correlation of 0.912 with averaged human scores (SongEval). Audiobox-Aesthetic reaches 0.61–0.66 in the same paper. We retrain this regressor on our exact splits with the same hyperparameter search budget as our readout. We do not expect our unsupervised-plus-linear score to beat 0.912 on in-distribution overall quality, and we do not claim it will. Beating it means one of three things: higher correlation on the Coherence and Clarity of Structure dimensions specifically; a statistically significant partial correlation with those dimensions after controlling for the regressor's output; or larger, correctly signed responses to the structural perturbations described below, where the regressor's audio-fidelity shortcut should fail.
On pairwise preference, the strongest numbers are from CMI-RewardBench: CMI-RM reaches 78.20% musicality accuracy on CMI-Pref and 73.43% on Music Arena, against 65.80% for Gemini 3 Pro on CMI-Pref. MuseCritic reports 71.35% on Music Arena. We use released checkpoints where available and compare on identical pairs. Beating them standalone is not the bar. The bar is that adding our score to CMI-RM in a fused model improves accuracy on CMI-Pref and Music Arena, and that the gain grows for clips longer than 120 seconds.
Why not just…?
Why not just use the perplexity of a pretrained music language model as the coherence score? This is the most serious alternative, and it is exactly our mean-surprisal ablation. The evidence against it is indirect. Perplexity diverges from human predictive behavior in reading (Transformers Can Model Human Hyperprediction in Buzzer Quiz), and resolved surprises such as chromatic enclosures are preferred despite being improbable (Improving Automatic Jazz Melody Generation by Transfer Learning Techniques). Neither result is a direct music test. If plain mean surprisal matches the commitment score on SongEval Coherence and on the perturbation suite, the decomposition is unnecessary and we will report that.
Why not just fine-tune a scalar regressor on SongEval? It already reaches 0.912 correlation, so in-distribution it is hard to beat. Our argument is not accuracy on the training distribution. Such regressors may learn production polish, which is entangled with coherence according to SongEval itself, and MuseCritic reports that they cluster predictions near the mean. The perturbation suite settles this directly. If the regressor's scores drop as much as ours when bars are shuffled but fidelity is preserved, and stay as stable as ours under lo-fi degradation, the threat stands. Our contribution then reduces to interpretability and cheap streaming computation.
Why not just ask a multimodal LLM, or use MuseCritic? The Gemini 3 Pro figure of 65.80% on CMI-Pref suggests that general models judge musical structure poorly. MuseCritic is stronger but generates a long critique over the whole song before it scores. That makes it slow, and it cannot report where an expectation failed while the music is playing. We compare against it on Music Arena and report wall-clock cost per song. Our forecaster runs one causal pass, which makes it usable as a per-segment reward during decoding.
Proposed Experiments
The decisive test is a controlled perturbation suite. It is built from high-rated SongEval tracks, and from Lakh MIDI renders where structure is known exactly. Bar-shuffling and section deletion break structure while keeping fidelity and timbre. Low-bitrate re-encoding and added noise keep structure while degrading fidelity. This is the lo-fi demo case that SongEval describes as degraded but still coherent. Timing jitter breaks meter. Inserting chromatic approach notes that resolve to the expected chord tests resolved surprise. A small blind listening study on a subset of the perturbed tracks confirms that the perturbations shift human ratings in the intended direction.
The structural metric should satisfy three predictions. It should respond strongly to shuffling and jitter. It should barely respond to re-encoding. It should not penalize resolved approach notes. FAD, mean surprisal and the SongEval regressor should each fail at least one of these.
On SongEval, we report Spearman and Pearson correlation per dimension. We also report partial correlation of $V_h$ with Coherence and Clarity of Structure, controlling for the regressor's output and for the Naturalness rating. Splits hold out generators where provenance is available and otherwise hold out genres, so that the comparison tests transfer rather than memorization. On CMI-Pref and Music Arena, we report pairwise accuracy standalone and fused with CMI-RM, stratified by clip length. The checkpoint sweep reports correlation against forecaster perplexity.
The proposal fails cleanly in two cases. One is if commitment weighting adds nothing over mean surprisal. The other is if the partial correlation beyond the trained regressor is indistinguishable from zero. Either outcome would itself be informative about why objective and perceptual metrics diverge.
References
- Transformers Can Model Human Hyperprediction in Buzzer Quiz
- Improving Automatic Jazz Melody Generation by Transfer Learning Techniques
- A Benchmarking Initiative for Audio-domain Music Generation using the FreeSound Loop Dataset
- Anticipatory Music Transformer
- ProGress: Structured Music Generation via Graph Diffusion and Hierarchical Music Analysis
- Incremental Semantic Role Labeling with Tree Adjoining Grammar
- Can fMRI reveal the representation of syntactic structure in the brain?
- CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction
- MuseCritic: Learning Multi-Aspect Song Rewards through Natural-Language Aesthetic Critiques
- SongEval: A Benchmark Dataset for Song Aesthetics Evaluation