Keeping Rare Idioms Alive When Recurrent Music Models Are Compiled into N-Gram Tables

A small recurrent network, meaning a model that reads a sequence one step at a time while carrying a running memory, can learn the stylistic habits of jazz melody. One such habit is the chromatic enclosure: the line approaches a chord tone by stepping onto the notes a half-step above and below it before landing. Fast decoders, however, often cannot afford to query a neural network for every branch of a search. The standard workaround is to precompute the network's predictions offline into a static n-gram table, a lookup table giving the probability of the next token from the previous few tokens. This conversion is judged by whether the table keeps the network's average likelihood, or its BLEU score in translation. Those are the same kind of aggregate statistics that music research has repeatedly found to disagree with expert listeners.

This proposal argues that the two problems share one cause, and it tests that claim. Standard conversion chooses which contexts to store by how often they occur in the corpus. It therefore throws away the rare, long-range transitions that experts value, while held-out perplexity barely moves. We propose a conversion method that decides which n-grams to keep by where the recurrent model disagrees most with a count-based model. We then test, with jazz experts, whether this preserves perceived quality in cases where perplexity says two tables are equivalent.

Core Claim

Offline neural-to-n-gram conversion has so far been applied to words and characters, and validated only with automatic metrics. Converting Continuous-Space Language Models into N-Gram Language Models for Statistical Machine Translation rewrites the probabilities of n-grams that already exist in a back-off table, using a feedforward neural model restricted to a short-list of frequent words, and then renormalizes. On NTCIR-9 this came within 0.14 BLEU of two-pass neural reranking. The authors themselves note that the method cannot discover n-grams outside the existing table. Neural Network Language Model for Chinese Pinyin Input Method Engine uses static tables for a 100x speedup. Real-Time One-Pass Decoder for Speech Recognition Using LSTM Language Models uses lookahead tables built from count-based, not neural, n-grams. These three papers supply the method we reuse.

The motivating evidence comes from Improving Automatic Jazz Melody Generation by Transfer Learning Techniques. There, a recurrent generator with the better statistical overlap score lost to its rival among jazz experts, because expert-valued chromatic enclosures look like statistical outliers. Our claim has two parts. First, frequency-based support selection deletes exactly these outlier-but-idiomatic transitions, and held-out perplexity does not register the loss. Second, choosing support by neural-versus-count disagreement keeps them at the same table size, and experts can hear the difference.

Research Program

The source model is an autoregressive gated recurrent unit (GRU) language model trained on monophonic melodies from the TheoryTab and jazz (CY+R) corpora used by Hung et al. We use an event tokenization of pitch and duration. We deliberately use an autoregressive language model rather than their variational autoencoder, because conversion needs next-token conditionals.

Conversion then proceeds in three steps. The first step builds a candidate pool of order-2 to order-5 contexts. These contexts come from the training data and from contexts the GRU itself visits when sampling. The second step scores each candidate (context $h$, next token $w$) by how much probability mass the count-based model misassigns relative to the network, weighted by how often the network reaches that context:

$$s(h,w) = \pi_{\text{RNN}}(h)\; p_{\text{RNN}}(w\mid h)\,\log\frac{p_{\text{RNN}}(w\mid h)}{p_{\text{BO}}(w\mid h)}$$

In this score, $\pi_{\text{RNN}}(h)$ is the frequency of context $h$ under the network's own samples, $p_{\text{RNN}}$ is the network's next-token probability averaged over the histories that end in $h$, and $p_{\text{BO}}$ is the count-based back-off probability. The score is the contribution of each entry to the divergence between the network and the table, so keeping the top entries under a size budget greedily reduces that divergence. This is the reason for the form.

The third step writes the network's probabilities into the selected entries. It then recomputes the back-off weights exactly, so that each context sums to one. This replaces the ad hoc renormalization used in prior work.

Alongside conversion, we build an idiom audit. A rule-based detector finds enclosure setups in held-out jazz melodies, meaning a chromatic neighbor followed by the opposite neighbor of an upcoming chord tone. For each setup, the audit measures the probability the table assigns to the idiomatic resolution, relative to the probability the GRU assigns. The same audit covers a second idiom class, approach-note resolutions, so the result does not rest on one hand-picked pattern.

A secondary setting reuses the recipe on the Performance-Timed Music Tokens of Agogic: Performance-Timed Music Tokens for LLM-Native Text-to-Symbolic-Music Generation. That 609-symbol vocabulary uses 10 ms timing bins, which makes n-gram sparsity much harsher. This setting tests whether divergence-guided support still helps when the vocabulary is fine-grained. It is exploratory. The Agogic results are from Transformers, and no reported result was found on Lacuna for a recurrent model on MidiCaps.

Baselines

The naive reference point is a count-based Kneser-Ney style n-gram trained directly on the same melodies, with no neural information. This is what the lookahead tables of Jorge et al. are built from, and it sets the floor.

The strongest existing conversion is Wang et al.'s rewriting method, reimplemented for the GRU. It rewrites all n-grams already in the count table, with the same order and the same entry budget, over the full melody vocabulary, since that vocabulary is small enough that no short-list is needed. The only reported number for this method is on translation: 33.22 first-pass BLEU against 33.36 for two-pass reranking on NTCIR-9, from Converting Continuous-Space Language Models into N-Gram Language Models for Statistical Machine Translation. No reported result was found on Lacuna for neural-to-n-gram conversion in music. It is the strongest competitor because it is the only published conversion that matched live neural reranking.

The upper reference is the unconverted GRU sampled directly. For perceptual calibration against a published system, we also include Hung et al.'s fine-tuned recurrent model where it can be reproduced. That model received a mean expert rating of 2.85/5, against 2.52/5 for the multitask model, whose overlap score was higher at 0.7536 against 0.7490.

A Monte Carlo baseline samples a large corpus from the GRU and trains a count n-gram on it, using the same GPU budget our scoring step consumes.

The key ablation keeps our exact renormalization and candidate pool but selects entries by context frequency instead of by $s(h,w)$. This isolates support selection as the active ingredient.

All tables share the same entry budgets, which we sweep over four sizes, the same maximum order, and the same held-out split. Beating the baselines means the following. At matched table size and matched held-out perplexity (within 1%), our table must keep a significantly higher share of the GRU's enclosure-resolution probability. Experts must also rate its outputs closer to the GRU's and higher than Wang-style rewriting.

Why not just…?

Why not just keep running the GRU? For plain sampling of four-bar melodies, a small GRU is cheap, and this proposal does not claim otherwise. The table matters where Jorge et al. needed one: constrained or beam search with many live hypotheses, and lookahead costs queried at every branch. It also matters for on-device interactive accompaniment. Separately from speed, the table makes the model's local grammar inspectable, and that is what lets the audit locate the lost idioms. We report wall-clock cost for constrained generation in both setups.

Why not just sample a big corpus from the GRU and count n-grams? This is the most serious threat. It does discover n-grams outside the original table. However, rare idioms appear rarely in samples, so their estimates stay noisy unless the number of samples grows large. Our scoring queries the network's exact probabilities at candidate contexts instead of waiting for the idioms to be sampled. The matched-compute Monte Carlo baseline settles the question. If it equals our method on idiom retention and expert ratings, we will report that simple sampling suffices and that frequency-based support was the only real defect.

Why not just interpolate the count n-gram with a sharpened or tempered neural distribution? Interpolation and temperature change how probability is spread across entries the table already holds. They cannot restore an entry the table never stored. We include a tuned interpolation of the count table with the Wang-rewritten table as an extra point on the size sweep.

Why not just score outputs with a learned aesthetic evaluator instead of experts? The best-calibrated evaluator in the file, the UTMOS-based model in SongEval: A Benchmark Dataset for Song Aesthetics Evaluation, reaches r = 0.912. But it was trained on full-length rendered songs, not four-bar jazz lines. Hung et al.'s split between expert and casual listeners suggests that non-specialist judgments would miss the very effect under study. We therefore treat any automatic evaluator only as a secondary signal.

Proposed Experiments

The first check is diagnostic. Across the four table sizes, we plot held-out perplexity against idiom retention for frequency-selected and divergence-selected tables. The claim predicts that perplexity curves nearly coincide while idiom-retention curves separate. If both curves separate together, perplexity was an adequate proxy after all, and the perceptual angle collapses.

The second check is decisive. We pick the pair of tables with the closest perplexity but the largest retention gap. We generate matched sets of melodies from each, using the same seeds and the same sampling settings, plus GRU samples and the count baseline. A blind listening test then collects ratings from jazz musicians and from casual listeners, following Hung et al.'s protocol. The claim is supported only if experts, but not necessarily casual listeners, prefer the divergence-selected table and place it closer to the GRU.

The third check examines metric divergence directly. We compute the overlap-area statistics of Hung et al. for every system and test whether they rank the systems in expert order. We expect overlap area to be blind to the difference, which would reproduce the published divergence under controlled conditions.

The fourth check repeats the diagnostic on the performance-timed tokens, measuring only perplexity, idiom-proxy retention and Fréchet Music Distance. It reports whether the gain survives a fine-grained vocabulary.