Music generators that write guitar tablature, a notation that says which string and fret to press for every note, are judged almost entirely by people who listen to synthesized playback or by statistics computed over tokens. Neither judge holds a guitar. A software synthesizer renders a fret-2-to-fret-19 jump inside a thirty-second note just as cleanly as a comfortable phrase. A listener therefore rates it as good music, and a guitarist cannot play it. This proposal measures that gap directly. Guitarists attempt to play generated excerpts and mark the exact measures that defeat them, and separate listeners rate the same excerpts by ear. We then test whether cheap, interpretable physical scores can find the defects the players found.
The concrete deliverable has three parts. The first is a measure-level benchmark of playability defects in generated tablature, the musical analogue of image-defect benchmarks that mark where a generated hand has six fingers. The second is a small, explicit hand model whose cost weights are fitted to performer marks rather than set by hand. The third is a test of whether reranking generator samples with this score improves performer-rated playability without lowering listener-rated quality. The headline question is empirical and falsifiable. On generated music, do listener opinion and hand-crafted playability costs predict what performers actually find unplayable, and where?
Core Claim
Playability metrics have their own objective–subjective gap, and it is worst on generated content. Existing physical models of performance were built and validated on human-written scores. These include hidden-Markov fingering decoders (Automatic Decision of Piano Fingering Based on a Hidden Markov Models), dynamic-programming finger-velocity features (Score Difficulty Analysis for Piano Performance Education based on Fingering), and learned fingering embeddings (Combining piano performance dimensions for score difficulty classification). They predict a piece-level difficulty grade, not where a passage becomes impossible. Generated tablature fails in ways human scores rarely do. ShredGP reports impossible fret jumps, and Automatic Composition of Guitar Tabs by Transformers and Groove Modeling reports low-register notes leaking onto the wrong strings. We claim two things. Listener ratings will be largely uninformative about these failures. A hand model with a handful of cost weights fitted to performer marks will localize them substantially better than the same model with hand-set weights, including on generators it was not fitted on.
The closest work is the piano difficulty classifier of Ramoneda et al. It showed that adding physical-gesture features brings difficulty prediction closer to expert grades. However, it is discriminative, works only on existing classical scores, and its authors say it cannot pinpoint the hardest passage. We differ in three ways. We work on generated tablature, we supervise and evaluate at the measure level with performer annotations, and we put the score into the generation loop. Chen et al. ran a listening test, but it collected only a holistic 1–5 opinion score on 16-bar clips. No performer was ever asked to play anything.
Research Program
Benchmark. We will train a Transformer-XL tablature generator in the style of ShredGP on the DadaGP rock subset and on SoloGP. We will also retrain the fingerstyle model of Chen et al. on its 333-tab corpus. This gives two generators with different failure styles, one built around fast solos and one around fingerstyle chords. From each we sample 8-bar excerpts. We mix in matched human excerpts from the same corpora as controls, targeting roughly 600 excerpts in total.
Intermediate and advanced guitarists receive the tab at the notated tempo. Each guitarist may slow the tempo down if needed and record the slowest tempo that worked. For every measure, they mark it as playable, hard, or impossible, and they choose a short reason from a fixed menu: stretch, shift distance, shift speed, string crossing, or finger collision. Each excerpt gets at least three performers so that we can report agreement between performers. In parallel, listeners who are not asked to play rate the synthesized audio of the same excerpts on the Chen et al. opinion scale. Having both judgments on identical items is what lets us measure the gap between hearing and playing instead of asserting it.
Hand model. For each tab event, the fretting hand is a latent state: a fret position plus an assignment of fingers to the pressed strings. The fret geometry follows the voicing-constraint logic of Chord Recognition using Instrument Voicing Constraints. Transitions between states are scored by a weighted sum of interpretable features $\phi$: fret stretch within a chord, shift distance, shift distance divided by the time between notes, reuse of the same finger on different frets, barre requirements, and string crossings for the picking hand. We decode the cheapest hand trajectory with the Viterbi algorithm, as in Yonebayashi et al. For measure $m$ we then predict
$$P(\text{defect in } m) = \sigma\big(a + b \max_{t \in m} w^\top \phi_t\big),$$
where $\phi_t$ is the feature vector of the transition into event $t$ along the decoded path, $w$ are the cost weights, $a,b$ are link parameters, and $\sigma$ is the logistic function. The max encodes the observation from the piano difficulty work that one bottleneck passage dominates. We fit $w$, $a$ and $b$ to performer marks. Because there are only around fifteen parameters, the fit stays interpretable and is unlikely to memorize one generator's quirks.
Closing the loop. We use the fitted score in two ways. The first draws $N$ samples per prompt and keeps the one with the lowest predicted defect mass. The second masks next tokens whose cheapest hand transition exceeds a threshold during decoding. New performers and new listeners then evaluate the outputs. We also track the style-attribution accuracy of SoloGPBERT, so that we catch a playability gain bought by flattening style into safe, boring licks.
Baselines
The naive reference point is a pair of measures. One is the unfiltered generator. The other is the listener opinion score itself, used as a predictor of performer playability. For orientation, Chen et al. report a listener score of 3.43 for their generator against 3.48 for human compositions. If listener scores track performer marks well, most of this proposal's motivation collapses, so we treat this as a real baseline and not a strawman.
The simplest physical baseline is a static rule filter. It flags any measure whose simultaneous fret span exceeds four or five frets, or whose same-string fret jump exceeds a fixed distance. It is the cheapest thing a practitioner would ship. It is run on identical excerpts and reranking budgets.
The strongest competitor is the same Viterbi hand model with hand-set, uniform or expert-chosen weights. This is a guitar analogue of the uncalibrated Pianoplayer and HMM costs. On piano these costs are already strong. Ramoneda et al. report that a finger-velocity feature reaches a Spearman correlation of 0.86 ± 0.05 with Henle difficulty rankings on Mikrokosmos, with 68 ± 10% three-class accuracy. That is a piece-level piano result. For measure-level playability localization on generated tablature, we found no reported result on Lacuna, so our own evaluation of this baseline defines the bar. The comparison isolates the key ingredient, which is calibration to performers. Both versions share the decoder, the features and the inference cost, and differ only in where $w$ comes from.
A learned black-box critic is the fourth baseline. We take a small Transformer, fine-tuned on DadaGP tokens with the same performer labels and an equal tuning budget, and give it access to as many annotated excerpts as the calibrated model. Ablations remove the max-pooling, which leaves mean pooling, and remove the speed features, which leaves static geometry only.
Beating the strongest baseline means a statistically higher measure-level area under the ROC curve than the uncalibrated hand model on a held-out generator. Concretely, we fit on solo-generator excerpts and test on fingerstyle-generator excerpts, and vice versa. In the loop, success means higher performer-rated playability at matched listener score and matched style accuracy.
Why not just…?
Why not just filter with a hand-span rule? On piano, whether a stretch is playable depends on the preceding hand position and the tempo, which is why velocity-based features carry the signal in Ramoneda et al. In orchestration, METEOR found that static pitch-range limits miss breathing and fatigue constraints. Guitar adds its own complications: the same pitch can sit at several string–fret positions, and barres change what counts as reachable. Still, if most generated defects turn out to be gross jumps, a rule may catch them. The rule baseline and the performers' reason tags will show what share of defects are of each kind. If the rule matches the calibrated model, we will report that and recommend the rule.
Why not just use the hand-crafted cost without performer data? This is the most serious threat. The decisive comparison is the held-out-generator evaluation. If uncalibrated weights localize defects as well as fitted ones, the contribution shrinks to the benchmark and the gap measurement. We consider those still worth having, because no measure-level playability benchmark exists. We would report it that way.
Why not just rely on listening tests, or a learned preference model? Listeners hear audio that a synthesizer renders perfectly. The 3.43 versus 3.48 near-tie in Chen et al. coexisted with documented string leakage, and MuPT's preference tests (MuPT) ask only which track a listener prefers. Image work shows that scalar preference models such as Human Preference Score do not localize small structural defects. In AGIDefect-4K, over 85% of defects covered less than 5% of the image, and zero-shot multimodal models scored AUC 0.29–0.38 while a supervised localizer reached 0.65. Our paired listener–performer design tests this directly rather than assuming the image result transfers to music.
Why not just scale the generator? Scaling cannot teach what the data and the loss never penalize. Corpora mix arrangements and reductions, and cross-entropy weighs an impossible fret token the same as any other token. We test this cheaply by including excerpts from two model sizes in the benchmark and checking whether the defect rate falls with size.
Proposed Experiments and Decisive Checks
The first check measures the hearing–playing gap. We compute the rank correlation between per-excerpt listener opinion scores and performer playability, and between listener scores and the share of measures performers marked impossible, separately for human and generated excerpts. Our hypothesis is that the correlation is weak on generated excerpts. A strong correlation would refute the premise.
The second check is localization. For each candidate scorer we compute the measure-level ROC area against the majority performer label and compare it with the agreement between pairs of performers, which acts as a ceiling. The scorers are the listener score, the rule, the uncalibrated hand model, the calibrated hand model, and the black-box critic. All are evaluated in the cross-generator split.
The third check is the generation loop. We rerank or constrain both generators at matched sample budgets and collect fresh performer ratings, fresh listener scores, SoloGPBERT style accuracy, and the token-statistic metrics of MusPy to detect collapse. The proposal succeeds if the calibrated score reduces performer-marked impossible measures more than the rule and the uncalibrated cost, while listener score and style accuracy do not drop. It fails informatively if every gain in playability costs listener preference, because that would show the two axes trade off rather than merely diverge.
References
- Automatic Decision of Piano Fingering Based on a Hidden Markov Models
- Score difficulty analysis for piano performance education based on fingering
- Combining piano performance dimensions for score difficulty classification
- ShredGP: Guitarist Style-Conditioned Tablature Generation
- Automatic Composition of Guitar Tabs by Transformers and Groove Modeling
- DadaGP: A Dataset of Tokenized GuitarPro Songs for Sequence Models
- Chord Recognition using Instrument Voicing Constraints
- METEOR: Melody-aware Texture-controllable Symbolic Music Re-Orchestration via Transformer VAE
- MuPT: A Generative Symbolic Music Pretrained Transformer
- Human Preference Score: Better Aligning Text-to-image Models with Human Preference
- AGIDefect-4K: A Richly Annotated Dataset for AI-Generated Image Defect Detection, Localization and Explanation
- MusPy: A Toolkit for Symbolic Music Generation