When two violins play at once, much of their sound energy lands on the same frequencies at the same moments. A separation model that only hears the mixture often cannot say which player produced which part. Spatial position helps when the seating is known, but it is static. The players' bodies are not. Each violinist's bow reverses direction, and each left hand lands on the fingerboard, at slightly different moments, and those moments line up with the onsets and swells in that player's own sound. This proposal turns each player's limb trajectories into a predicted "activity map": a curve over time and frequency band saying when, and roughly where in the spectrum, that player should be producing energy. The map is then used to break the tie inside an existing separator.
The predictor is built from reservoir computing. A reservoir, or echo state network, is a recurrent network whose internal weights are random and never trained. Only a linear output layer is fit, by ridge regression in closed form. We run one small reservoir per limb, one for the bowing arm and one for the fingering hand, and fit the output layer per performer from a minute or two of solo "enrollment" playing, the kind of material a soundcheck provides. The claim is specific. Fast, gradient-free calibration of each player's own motion-to-sound mapping, computed separately per limb, resolves same-instrument overlap better than static position cues or hand-crafted bow features. The point of comparison is the regime where paired motion and audio are scarce, and there the claim should hold at a fraction of the cost of a trained motion encoder.
Core Claim and Where It Sits
Three prior results motivate the idea. Position also matters! separates string-quartet stems with a conditional U-Net and shows that timbre conditioning alone fails on Violin 1 versus Violin 2, while static seating-position embeddings succeed. Its cue cannot change over time, so it says nothing about who is playing at a given instant. Motion informed audio source separation couples non-negative matrix factorization (NMF) activations to hand-crafted bow inclination and velocity through a penalty. It improves separation on the Ensemble Expressive Performance (EEP) string data. However, its coupling weight needs per-recording calibration, and it fails on viola in piece P2 because the viola's bow descriptors collide with those of the violin and cello. Audio Matters Too! runs the coupling in the opposite direction, using a pitch tracker to correct fingertip pose. It collapses under polyphony, which shows that the motion-to-sound link has not yet been exploited to resolve polyphony itself.
One prior result supplies the method. WALKING WALKing walking (ConvESN) feeds each of five body parts into its own fixed echo state network and decodes the result for action classification, training about 7 times faster than a hierarchical LSTM on HDM05. We reuse its per-part reservoir decomposition. We change three things. The target is a continuous per-band acoustic activity map rather than a class label. The readout is fit per performer. The limb split is defined by sound production, separating the bowing arm from the fingering hand, rather than by gross anatomy. The reason to decompose by limb comes from the joint-level modeling literature. In a single whole-body vector, large slow motions such as torso sway mask small fast ones such as a wrist flick (Huang et al.; Karanam et al.). In a string player, the small fast motions are exactly the ones that predict note onsets.
Three lines of work constrain the design. The fading-memory theory of reservoirs (Gonon et al.; Gallicchio et al.) bounds how far back a reservoir can remember. This suits our target, since the lag between a bow change and its acoustic consequence is short, but it rules out using the reservoir for phrase-level structure. Jointist shows that guiding a separator with an estimated time-frequency activity map works, reporting a 5 dB SDR gain. It also shows that such guidance destabilizes training when the guide is learned jointly from scratch. Our guide is a frozen reservoir with a closed-form readout, fit before the separator is trained, so the guide cannot drift during that training.
Research Program
Each performer $p$ contributes limb trajectories $u^{(l)}_t$ for each limb $l$: joint positions, velocities, and accelerations of the right arm, wrist, and bow, and of the left hand and fingers. Each limb drives a leaky reservoir:
$$x^{(l)}_t = (1-\alpha)\,x^{(l)}_{t-1} + \alpha \tanh\!\big(W^{(l)}_{in} u^{(l)}_t + W^{(l)} x^{(l)}_{t-1}\big).$$
In this equation, $x^{(l)}_t$ is the reservoir state at frame $t$ and $\alpha$ is the leak rate. $W^{(l)}_{in}$ and $W^{(l)}$ are fixed random matrices, with $W^{(l)}$ scaled to a spectral radius near one, so the reservoir sits close to the stability boundary where its memory is largest.
The readout predicts the performer's activity map $\hat{E}^{(p)}_t \in \mathbb{R}^{K}$, the log energy in $K$ non-uniform frequency bands. The bands are narrow at low frequencies, following Music Source Separation with Band-split RNN. The readout is the ridge solution
$$W^{(p)}_{out} = E^{(p)} X^{(p)\top}\big(X^{(p)} X^{(p)\top} + \lambda I\big)^{-1},$$
where $X^{(p)}$ stacks the concatenated limb states over the enrollment frames and $E^{(p)}$ holds the measured band energies of that player's solo audio. Reservoir states are computed once and cached. Refitting for a new performer therefore takes one small matrix inversion.
We integrate the predicted maps in two ways. The first is the lighter test: drop-in replacement of the hand-crafted motion activations in the motion-informed NMF objective with $\hat{E}^{(p)}$. The second is the main system. In the conditional U-Net of Position also matters!, we replace the static position embedding with the time-varying $\hat{E}^{(p)}$, injected through the same gated modulation layers. The separator is trained on mixtures with the reservoirs and readouts frozen.
The analysis has three parts. First, we inspect which limb's readout carries the source-specific signal on overlapping frames; we expect bow-arm onsets and left-hand landings to contribute differently. Second, we sweep spectral radius and leak rate to locate the memory window the task actually needs. Third, we check whether per-limb reservoirs recover the P2 viola case where whole-instrument bow descriptors collided.
Baselines
The do-nothing reference is the same U-Net with timbral conditioning only. Position also matters! reports that this setting cannot separate identical instruments, so any motion-conditioned system must clear it by a wide margin on same-instrument segments.
The first strong competitor is the static-position U-Net from the same paper. It is the most direct existing answer to same-instrument ambiguity, and it matches large models with 46M parameters. We train it on identical mixtures, with the same backbone, epochs, and hyperparameter trial budget. Beating it means higher per-source SDR on same-instrument mixtures. The decisive measure is a lower rate of assignment swaps: segments where Violin 1's material lands in Violin 2's output.
The second strong competitor is motion-informed NMF with bow inclination and velocity (Motion informed audio source separation), run on EEP with its coupling weight tuned per recording as the paper requires. Our version gets one global setting. The file reports only that this method wins "often by a large margin" on EEP; no numeric SDR for EEP was found on Lacuna. We will rerun it. Beating it means higher SDR on Set 1 trios and Set 3 same-instrument duos without per-track tuning, and success on P2 viola.
The third competitor is video-based separation with iQuery (iQuery, code at JiabenChen/iQuery). It reports 11.17 dB SDR on the MUSIC benchmark, against 7.74 dB for CCoL. We compare on MUSIC duets, with poses extracted from the video by a markerless estimator. Matching iQuery there, rather than beating it, would already be meaningful given the compute gap. The real test is same-instrument pairs, where iQuery's appearance-based queries have no visual feature that distinguishes the two players.
Two ablations isolate the key ingredients. The first is a single whole-body reservoir with the same total state size, which tests the per-limb decomposition. The second replaces the reservoir with a memoryless random projection of the current frame's kinematics, which tests whether fading memory matters.
Why not just…?
Why not just train a small GRU on the joint trajectories, or put a linear probe on a pretrained motion encoder? This is the real threat. A pretrained encoder with a per-performer ridge probe has exactly the same calibration cost as our readout. We include both. The GRU is trained end to end on the same enrollment data, and the encoder is pretrained on String Performance Dataset motion-audio pairs and probed per performer. We expect the end-to-end GRU to overfit at one to two minutes of enrollment. If the pretrained probe matches the reservoir, the honest conclusion is that the contribution is the per-limb activity-map interface and per-performer calibration, not the reservoir as such. The remaining advantage would be that the reservoir needs no pretraining corpus and is not tied to that corpus's instruments. We will test this directly on cello performers held out from pretraining.
Why not just use static position embeddings? Position disambiguates only as well as the seating is known. When two violinists sit close together, or a recording uses an unseen arrangement, nothing in the cue changes from moment to moment. The comparison above settles whether time-varying motion adds information beyond seating. We measure it separately on frames where the two violins overlap in unison or octaves, identified from the ground-truth stems, since that is where any gain should concentrate.
Why not just use bow velocity and inclination as the NMF paper did? Those descriptors summarize the whole instrument and miss the left hand. The paper's own P2 failure shows they collide across players. Running our reservoir on only those descriptors, then on full per-limb kinematics, separates two questions: whether the gain comes from richer inputs or from the reservoir's temporal memory.
Why not just use audio-only separation or text-queried diffusion? A text prompt cannot name "the second violin" in a way that differs from the first. Fast accuracy estimation of deep learning based multi-class musical source separation argues that magnitude masking has a ceiling when sources share harmonics. The do-nothing baseline above quantifies this directly on our mixtures. We do not run diffusion separators, and say so as a scope limit.
Proposed Experiments
The primary testbed is EEP, which pairs optical motion capture with multitrack string audio and includes same-instrument duos. The second is synthetic same-instrument duos built from the String Performance Dataset, by mixing two different violinists, or two different cellists, playing different pieces. Its 3D body, hand, and bow annotations let us compare ground-truth kinematics against markerless estimates and so measure how pose noise degrades the method. MUSIC provides the video-only comparison against iQuery.
Enrollment uses the first one to two minutes of each performer's solo material, and evaluation uses disjoint pieces. Performers in the test set never appear in the separator's training set. Metrics are per-source SDR and SI-SDR, reported overall and on overlap frames, plus the assignment-swap rate.
Four checks are decisive. First, on same-instrument mixtures, the motion-conditioned U-Net should lower the swap rate relative to the static-position U-Net. Second, the per-limb reservoir should beat the whole-body reservoir of equal size on overlap frames. Third, an enrollment curve from 10 seconds to 5 minutes should show the reservoir readout equal to or better than the end-to-end GRU at short enrollment. Fourth, the P2 viola case should no longer fail.
If the first check fails, motion adds nothing beyond seating, and the idea is wrong. If only the third check fails against the pretrained probe, we report that the per-limb activity map, not the reservoir, is what carries the result.
References
- Position also matters! Separating Same Instruments in String Quartet using Timbral and Positional Cues
- Motion informed audio source separation
- Audio Matters Too! Enhancing Markerless Motion Capture with Audio Signals for String Performance Capture
- WALKING WALKing walking: Action Recognition from Action Echoes
- CoMo: Controllable Motion Generation Through Language Guided Pose Code Editing
- Learning Local Recurrent Models for Human Mesh Recovery
- Risk Bounds for Reservoir Computing
- Randomized Machine Learning Approaches: Recent Developments and Challenges
- Jointist: Simultaneous Improvement of Multi-instrument Transcription and Music Source Separation via Joint Training
- Music Source Separation With Band-Split RNN
- iQuery: Instruments as Queries for Audio-Visual Sound Separation
- Fast accuracy estimation of deep learning based multi-class musical source separation