Systems that name the composer of a piece of music are usually tested on a dozen composers. In that setting a classifier can succeed by recognizing who is playing, or which recording setup was used, instead of who wrote the notes, and nobody would notice. Two recent results disagree in a way that points at this problem. A composer classifier trained on piano performances got better when note loudness was removed from its input (Deep Composer Classification Using Symbolic Representation, F1 rising from 0.8333 to 0.8525 on 13 composers). A music generator got far closer to real music when it kept loudness and fine timing (Agogic, Fréchet Music Distance 159 versus 272–286 for a beat-grid model). Our reading is that expressive timing and dynamics mostly carry the performer's identity. To a composer classifier they act as a confound. To a listener they are much of what makes music sound real.
We propose to split a symbolic piano performance into two streams. The first is the quantized notes, which carry the composition. The second is the residual deviations in timing and loudness, which carry the performance. We train a separate identity-embedding model on each stream and evaluate them on test splits that deliberately cross composer with performer. These splits are the music analogue of the "topic confusion" test from text stylometry. We then use the two embeddings as per-sample, reference-free scores that ask whether a generated piece carries the composer or performer identity it was asked for. Distribution metrics such as Fréchet Music Distance cannot answer that question. We check the scores against human listeners to find out whether they track something people actually hear.
Core Claim
Composer attribution accuracy reported on standard random splits is inflated by performer and recording confounds. We expect the inflation to grow as the candidate pool grows, because performers repeat across composers and confounds accumulate. A model that factorizes notes from deviations should keep composer accuracy under splits that cross composer with performer, where single-stream models should drop. The same factorization should recover performer identity from the deviation stream alone. Finally, an identity score built on these embeddings should agree with listeners' judgments of "sounds like X" better than distributional metrics do. If the composer-stream advantage disappears once the crossed split is applied to a simple binarized baseline, then the composer half of this claim reduces to Kim et al.'s ablation, and the contribution narrows to performer attribution and the metric. We state that outcome in advance.
Positioning
Text stylometry supplies the diagnosis and the evaluation design. Luyckx et al. showed that small candidate sets overestimate accuracy and the stability of the features a model relies on, and Azarbonyad et al. and Sage et al. describe how styles crowd together as the number of candidates grows. The Topic Confusion Task flips author–topic pairings between training and test, and it found that pretrained language models fall to about 50–60% while syntactic features reach 83.3% on cross-topic Guardian. We reuse that flipping design, with performer and musical form playing the role of "topic." Authorship Attribution with Author-aware Topic Models separates content topics from author topics with two disjoint latent spaces. Our two-stream split has the same intent. We implement it with a deterministic signal decomposition rather than Gibbs sampling, which the file notes scales poorly. The move from N-way softmax to embedding comparison follows the large-N text systems, for example Cracking Double-Blind Review at 2,070 authors, and the open-world concern in Open-World Authorship Attribution.
Two music papers sit closest to this proposal. Deep Composer Classification Using Symbolic Representation feeds piano rolls to a ResNet-50 on MAESTRO for 13 composers. It never controls for performer, never scales the candidate pool, and never asks whether its features are audible. Agogic supplies the expressive tokenization and the "imprinting" failure, in which generators reproduce their training distribution and ignore the conditioning. It does not study identity at all. We take its performance-timed token format as the input for the deviation stream. MAESTRO supplies tightly aligned performance MIDI. Its competition setting, where many performers play overlapping repertoire, is what makes crossed splits possible.
On the metric side, Fréchet Audio Distance, the FreeSound loop benchmark, SongEval and CMI-RewardBench together establish three points. Distribution metrics favor texture over musical structure. They cannot score a single sample. Learned evaluators measure aesthetic quality and adherence to the prompt, but not identity. These papers motivate the gap. We do not compete with them on aesthetics.
Research Program
Decomposition. We beat-track each MAESTRO performance and snap its notes to a metrical grid, which produces the content stream. The deviation stream is each note's timing offset from its grid position, its duration ratio, and its velocity relative to a local running mean. Removing the local mean means the stream records shaping and not overall loudness.
Encoders. Each stream gets a transformer encoder over its token sequence. Both are trained with a supervised contrastive loss, which pulls together segments that share an identity and pushes apart segments that do not. The content encoder uses composer labels and the deviation encoder uses performer labels. A gradient-reversal head on each encoder penalizes it for predicting the other stream's label.
Attribution. We compare embeddings to identity centroids instead of training an N-way output layer. New identities can then be added without retraining, and a distance threshold gives open-world rejection.
Scaling sweep. We sweep the candidate pool size N from the 13 composers used by Kim et al. up to every composer and every performer present in MAESTRO. We report the full curve rather than one operating point. Going beyond MAESTRO depends on whether the Agogic 86.6k aligned corpus or the 6.25M captioned corpus carries recoverable artist metadata. The file does not say, so this extension is a stated risk and not a promise.
Generated-music score. For generated music, the identity score is the cosine similarity between a sample's stream embedding and the requested identity's centroid. Its margin over the nearest other identity is a per-sample measure of imprinting.
Baselines
The naive reference point is chance accuracy at each N, together with a nearest-centroid classifier on raw pitch-class and interval histograms. Any learned model has to beat these by a clear margin before its other numbers mean much.
The strongest existing method for this question is Kim et al.'s ResNet-50 over two-channel piano rolls with majority voting across 20-second segments (Deep Composer Classification). The reported results are weighted F1 0.8333 with velocity and 0.8525 with binarized frames, both on 13 composers from MAESTRO v2.0.0. We will reimplement both variants and confirm that we reproduce these numbers on their setting. We then run both variants on our crossed and scaled splits, using the same segments, the same voting scheme, and a hyperparameter budget matched to our models. No reported result was found on Lacuna for composer attribution under performer-crossed splits, for performer attribution on MAESTRO, or for any N beyond 13. Beating the binarized ResNet means two things on the crossed split: higher composer F1 at every N above 13, and a smaller drop from the random split to the crossed split.
A single-stream transformer on the full performance-timed tokens, trained with the same contrastive loss, is the key ablation. It has the same capacity and data as our model but no decomposition and no reversal heads. If it matches the two-stream model on the crossed split, the decomposition is not doing the work. A second ablation removes only the gradient-reversal heads.
For the metric, the comparison is Fréchet Music Distance computed per identity, as used in Agogic, and audio-embedding distance from a VGGish-style Fréchet Audio Distance on rendered audio (Fréchet Audio Distance). Both are scored against the same listener judgments. Beating them means significantly higher rank correlation with listeners' identity judgments at the level of individual samples.
Why not just…?
Why not just binarize velocity, quantize timing, and attribute composers as Kim et al. did? This is the most serious threat to the composer half of the proposal. Kim et al.'s ablation already suggests that stripping performance detail helps. Our content stream is close to this baseline, so on composer attribution alone the gain may be small. What binarization cannot do is identify performers, or score whether a generator reproduces a requested performance style, because it discards exactly the signal that Agogic found matters for realism. The crossed-split experiment will measure whether residual performer confounds survive binarization, for example through pedalling and note-density habits. If they do not, we report the composer result as a confirmation of Kim et al. and rest the contribution on the performer stream and the metric.
Why not just train one classifier on expressive tokens and use its confidence as a style score, in the way Inception Score uses a classifier? It is cheap, and on random splits it will probably look excellent. Our hypothesis is that it fails because it is confounded, not because it lacks capacity. On a crossed split, a classifier that learned "this performer usually plays Chopin" will misattribute. That is the topic-sponge behavior The Topic Confusion Task documented for BERT and RoBERTa. The single-stream ablation is exactly this alternative, run under identical conditions, so the experiments settle the question directly.
Why not just compare against each author's corpus with Fréchet distances or audio embeddings? These distances need batches of samples, so they cannot score one piece (CMI-RewardBench). Audio embeddings also cluster by timbre and recording conditions rather than musical grammar (FreeSound loop benchmark). We include both as metric baselines rather than dismissing them.
Why not just ask a multimodal LLM? As a music critic, Gemini 3 Pro reaches only 65.80% agreement with experts (CMI-RewardBench). Prompt-based attribution in text degrades beyond roughly 10–20 candidates (Can Large Language Models Identify Authorship?). We run it once, at small N only, as a reference point.
Proposed Experiments and Decisive Checks
The first experiment produces attribution curves over N on MAESTRO under three splits: random segments, held-out pieces, and held-out composer–performer pairs. A second confusion split flips musical form within each composer, for example training on sonatas and testing on nocturnes. Every model is scored by macro-F1 and by top-5 accuracy. The decisive check is the gap between the random split and the crossed split. We predict that this gap widens with N for the full-roll baselines and stays roughly flat for the content stream. We also predict that the deviation stream identifies performers well above chance, while the content stream does not.
The second experiment tests sensitivity to perturbations. We render the same MIDI through different piano sample sets, and we add timing jitter and velocity jitter whose audibility is calibrated in a pilot listening test. A model whose decisions flip under changes that listeners cannot hear is relying on artifacts. We expect identity scores to stay stable under inaudible changes and to move under audible style edits, such as transplanting one performer's deviations onto another performer's notes.
The third experiment is a listener study with trained pianists. Each trial presents pairs of rendered excerpts and asks "same composer?" and "same performer?" We then compute rank correlation between each metric and the listeners' answers. This directly tests whether attribution features are perceptible, which is the point where the two research directions meet.
The fourth experiment conditions an expressive-token generator in the style of Agogic on composer or style prompts. We measure how often each sample's identity margin favors the requested identity, and we compare those per-sample scores against listener judgments and against Fréchet Music Distance. The result we look for is that the identity score exposes imprinting that Fréchet Music Distance misses. This would show up as samples with good Fréchet Music Distance whose identity margin points to the training distribution's dominant identity, with listeners agreeing.
References
- Deep Composer Classification Using Symbolic Representation
- Agogic: Performance-Timed Music Tokens for LLM-Native Text-to-Symbolic-Music Generation
- Authorship Attribution and Verification with Many Authors and Limited Data
- Time-Aware Authorship Attribution for Short Text Streams
- Investigating the Influence of Selected Linguistic Features on Authorship Attribution using German News Articles
- The Topic Confusion Task: A Novel Evaluation Scenario for Authorship Attribution
- Authorship Attribution with Author-aware Topic Models
- Cracking double-blind review: Authorship attribution with deep learning
- Open-World Authorship Attribution
- Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset
- Fréchet Audio Distance: A Reference-Free Metric for Evaluating Music Enhancement Algorithms
- A Benchmarking Initiative for Audio-domain Music Generation using the FreeSound Loop Dataset
- SongEval: A Benchmark Dataset for Song Aesthetics Evaluation
- CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction
- Can Large Language Models Identify Authorship?