The Room as a Second Voice

A single singer, hummer or violinist recorded in a real room does not produce a single-voiced signal. Each note's reverberant tail is still ringing when the next note starts, so the spectrogram, the time-frequency picture that transcription models read, shows two or more overlapping harmonic series even though only one person is playing. Pitch trackers built on the assumption that exactly one fundamental frequency sounds at a time (the "monophonic assumption") misread this room-made overlap as extra notes, octave jumps or smeared offsets. Polyphonic transcribers go wrong in a different way: they faithfully report the tails as real notes. This note proposes treating the room as an explicit second voice. The transcriber explains each recording as one dry melodic line convolved with a room response, and it fits that response to the specific recording at test time.

The second half of the idea concerns what kind of room the model should expect. Most cheap reverberation simulators assume a "shoebox", a rectangular box with parallel walls. This is the acoustic version of the Manhattan world assumption from 3D vision, which holds that all surfaces align with three perpendicular axes. Parallel walls produce periodic reflection trains, and periodic echoes carve an evenly spaced comb into the spectrum that looks like a harmonic series. We therefore expect the geometric prior used to model the room to matter specifically for pitch, not only for generic robustness. The note proposes to measure this effect and to build a transcriber whose room model relaxes from shoebox to "Atlanta world" (a vertical gravity axis with free horizontal wall directions) to arbitrary scanned geometry.

Core Claim

When the source really is monophonic, the monophonic assumption stops being a limitation and becomes the constraint that makes blind room estimation solvable. If at most one dry fundamental is active per frame, any additional harmonic energy must come from the room, so the room's contribution can be separated from the performance without a probe signal. We model the observed magnitude spectrogram $X(t,f)$ at frame $t$ and frequency $f$ as

$$ X(t,f) \approx S(t,f) + \sum_{k=1}^{K} h_k(f)\, S(t-k,f), $$

where $S$ is the dry spectrogram of a single melodic line (at most one fundamental and its harmonics per frame), $h_k(f)$ is a non-negative per-frequency decay filter over the previous $K$ frames standing in for the room, and the sum is the "second voice".

A shoebox with parallel walls at distance $d$ adds a strong reflection train with period $2d/c$ ($c$ is the speed of sound). That train imposes comb teeth spaced $c/(2d)$ Hz apart, which is roughly 57 Hz for $d = 3$ m. This is why we expect Manhattan-type rooms to inject pseudo-harmonic structure that competes with the true fundamental. Non-Manhattan rooms break the periodicity and spread reflections more evenly. The prediction that makes this testable is that trackers trained with shoebox reverberation will be miscalibrated in both directions. They will expect comb artifacts that irregular rooms do not produce, and they will fail to handle the diffuse, aperiodic tails that irregular rooms do produce.

Relation to Prior Work

Monophonic transcription systems motivate the problem because they assume a single clean periodicity. Signal-to-Score Music Transcription using Graphical Models maps solo singing to a score with a graphical model and notes that moving to multiple simultaneous pitches causes a combinatorial explosion. MF-PAM gets its pitch accuracy by exploiting clean periodicity patterns. Dynamic HumTrans pairs a network that extracts acoustic features with dynamic-programming decoding for humming. None of the three models the room, and each treats overlap as something that does not occur in solo input. We keep their one-voice prior but apply it only to the dry line $S$.

Polyphonic methods supply the machinery for handling overlap, and they also constrain the design. MT3 transcribes overlapping instruments as a token sequence using a 60M-parameter T5 transformer, and it operates without any room context. Jointist shows that transcription can guide separation through shared bins. It also shows that joint training from scratch collapses because early noisy transcriptions misallocate energy. DGMO argues that ratio masks cannot divide energy within shared bins. That argument is why we fit an additive generative room term rather than masking the spectrogram. Our room term is small and parametric and starts from a tracker trained on dry audio, which is designed to sidestep the Jointist-style cold start.

Room-acoustics work supplies the geometric priors and the inversion idea. Deep Neural Room Acoustics Primitive learns room responses self-supervised, by convolving a known source with a predicted response and penalising the mismatch with the recording. We borrow that inversion loss but replace the known source with an unknown source that is constrained to be monophonic. 3D Room Geometry Inference from Multichannel RIR recovers non-shoebox walls from clean 32-channel impulse responses. Research on Prediction Method of Objective Assessment of Building Acoustics trains on 800 shoebox halls and reports failure on non-parallel walls. That failure is the direct motivation for varying the geometric prior. Neither of these two papers touches music. On the vision side, AtlasGS and GeoLayout show that relaxing Manhattan to Atlanta-world or to free plane parameters improves reconstruction of non-cuboid rooms. We carry that same relaxation ladder into the room model used for acoustics.

Research Program

The first component is a controlled testbed. Dry monophonic material with note labels comes from three sources. The first is the individual instrument tracks of URMP, which are real recordings with aligned notes. The second is the monophonic stems of Slakh2100 (dataset), mainly bass and lead lines. The third is sung melodies from GTSinger. Each clip is convolved with room responses from three families. The shoebox family uses image-source simulation over a range of dimensions and absorptions. The non-Manhattan family uses SoundSpaces 2.0 path-traced responses in Matterport3D and Replica meshes. The real family uses the measured MeshRIR room.

The Matterport rooms are stratified by how far they depart from a box. LayoutMP3D provides Manhattan layouts with 4, 6, 8 and 10+ corners. The residual between each room's mesh and its best Manhattan layout gives a continuous "Manhattan-ness" score. Some of these rooms are not closed polygons and leak energy through scan holes, which SoundSpaces 2.0 itself warns about. We will exclude those rooms using the platform's own diagnostics.

The second component is the model. A convolutional frame encoder predicts, for each frame, a single fundamental (or silence) and a harmonic amplitude envelope, and a dynamic-programming decoder in the style of Dynamic HumTrans turns these into notes. A room head predicts the decay filters $h_k(f)$ under one of three parameterisations. The shoebox parameterisation constrains them to a periodic reflection train plus an exponential tail. The Atlanta parameterisation allows independent horizontal reflection delays while keeping a floor-ceiling periodicity. The free parameterisation places no constraint beyond non-negativity and decay. Training reconstructs $X$ through the equation above and supervises the notes. At test time we optimise only $h$, and optionally the harmonic envelopes, on the single input recording while the note decoder stays fixed. The output is the transcription together with an interpretable room estimate.

The third component is the analysis. We measure how often each tracker places spurious fundamentals near multiples of $c/(2d)$ in shoebox rooms, and whether that error rate falls as rooms become less Manhattan. This tests the comb-filter mechanism directly rather than inferring it from aggregate scores.

Baselines

The naive reference is a monophonic tracker trained only on dry audio and run on reverberant audio: the MF-PAM and Dynamic HumTrans architectures, retrained on our dry training set. It measures how much the room alone damages transcription. The same models evaluated on dry audio give the ceiling.

The strongest competitor is data augmentation. The same encoder and decoder are trained on the same dry clips, convolved on the fly with the full pool of training room responses from all three families, but with no room head and no test-time fitting. This is the method a practitioner would actually use. It sees exactly the room responses our model sees, so it will be given the same number of optimisation steps, the same hyperparameter-search budget and the same parameter count; the room head's parameters will be added to its encoder width. Beating it means a better onset-plus-offset note F1 on held-out non-Manhattan and real rooms, with the gain concentrated in offsets and repeated same-pitch notes. That is where tails do the most damage.

The second strong competitor is MT3 fine-tuned on the same reverberant data with a "highest-salience note" reduction to one voice. It tests whether a general polyphonic transcriber, told nothing about monophony, already handles room-made overlap.

The ablation that isolates the key ingredient keeps the room head but removes the monophonic constraint on $S$, allowing multiple fundamentals. If identifiability comes from monophony, this variant should fit the spectrogram equally well but assign tail energy to spurious dry notes. A second ablation turns off test-time fitting, and a third swaps the room parameterisation along the shoebox, Atlanta and free ladder.

No reported result was found on Lacuna for monophonic note transcription under reverberation on any of these datasets. The nearest published anchors are dry. High-Resolution Piano Transcription reports 96.72% note F1 on MAESTRO for solo piano, and MT3 reports a 263% relative Onset+Offset F1 gain on URMP from mixture training. We will therefore establish all reverberant baseline numbers ourselves under the matched protocol above.

Why not just…?

Why not just augment with realistic room responses and let the network learn to ignore tails? This is the real threat, and it may well close most of the gap on frame-wise pitch accuracy. The argument against it is specific. An augmented model learns a room-averaged correction and cannot adapt to the one room in front of it. Kong et al. show that frame classifiers degrade sharply when the timing of the evidence shifts: note F1 fell from 93.92% to 76.52% under a 50 ms label shift in High-Resolution Piano Transcription. Reverberant tails shift apparent offsets in a room-dependent way. Manilow et al. find in Cutting Music Source Separation Some Slakh that the realism of training audio matters more than its quantity. That makes the choice between shoebox and mesh responses a live question even for augmentation. The matched augmentation baseline settles the question directly. If it equals our model on held-out geometry classes, the modelling contribution fails. What would remain is the testbed and the Manhattan-prior finding, and we would report the result that way.

Why not dereverberate first and then run an ordinary pitch tracker? A cascade using a dereverberation model in the spirit of AV-RIR was designed for speech, where the target is intelligibility rather than exact harmonic amplitudes and offsets. Masking-based dereverberation hits the within-bin ceiling described in DGMO, and its residual errors propagate into the tracker. We will run this cascade with the same training rooms. The cascade can also only mimic our use of the monophonic constraint indirectly, because the dereverberator never knows that the dry signal has one voice.

Why not just run a polyphonic transcriber and keep the top note? The MT3 baseline tests exactly this. Our expectation is that it confuses a sustained tail with a held note, because nothing in its objective distinguishes the two. The monophony-free ablation of our own model is the controlled version of the same question.

Why not just assume a shoebox room, since most rooms are roughly rectangular? This alternative is itself the scientific question. It is plausible for small treated studios. Wan et al. nonetheless report that shoebox-trained acoustic predictors break on irregular walls, and the comb-spacing argument predicts pitch-specific errors. Training every model separately on shoebox-only and mesh-derived responses answers it.

Proposed Experiments

The central experiment trains every system on URMP, Slakh-monophonic and GTSinger training splits convolved with training rooms. Evaluation uses four held-out conditions: dry audio, unseen shoebox rooms, unseen Matterport rooms binned by LayoutMP3D corner count and Manhattan residual, and the real MeshRIR room. Metrics are note onset F1 and onset-plus-offset F1 under the standard 50 ms onset tolerance, raw pitch accuracy, octave-error rate, and false-positive notes per minute occurring inside tail regions.

The first decisive check is the geometry generalisation matrix. We train on shoebox-only rooms and test on non-Manhattan rooms, and the reverse. The hypothesis holds if shoebox-trained systems lose accuracy as the Manhattan residual grows, and if our free or Atlanta room head shrinks that slope more than matched augmentation does.

The second check is the mechanism test. The hypothesis requires spurious fundamentals in shoebox rooms to cluster near multiples of $c/(2d)$ for the simulated wall spacings, and the clustering to disappear in non-Manhattan rooms. If there is no clustering, the comb-filter explanation is wrong, whatever the aggregate scores show.

The third check is identifiability. On simulated rooms, where the true response is known, the fitted $h_k(f)$ should correlate with the true per-band decay under the monophonic constraint and fail to do so without it.

The fourth check tests whether test-time fitting explains the gain. Performance should improve with the number of fitting steps on a single recording, and it should improve most on the real MeshRIR room, which appears in no training data.

The proposal succeeds if the room-as-second-voice model beats matched augmentation on onset-plus-offset F1 in held-out non-Manhattan and real rooms, and the monophony ablation shows that the constraint is what buys the gain. If augmentation ties, the published outcome is a benchmark, a measured effect of the shoebox prior on pitch tracking, and a clear negative result for explicit room modelling.