Listening for the Hand Strike: Audio-Timed Force Estimation for Upper-Limb Weight Bearing

When a wheelchair user slides onto a bench, or a gymnast blocks off a vault, most of the mechanical stress on the wrist, elbow and shoulder arrives in a few tens of milliseconds after the hand hits a surface. Cameras are the cheapest way to measure movement outside a laboratory, but at ordinary frame rates such a contact window spans one or two frames. The usual trick for turning video into force is to double-differentiate estimated body position (force equals mass times acceleration, plus gravity), and that trick smooths these brief peaks away. A microphone has the opposite profile. It records nothing about posture, but it timestamps the moment of a hard contact to within a fraction of a millisecond, and how long the sound lasts tells us how long the contact lasted. This proposal fuses the two. Sound fixes when contact begins and how long the force takes to rise. Video supplies how much momentum the body loses during that time. A small camera frame budget is concentrated in the windows the audio marks.

The motivation is clinical as well as technical. Manual wheelchair users load their shoulders through their hands thousands of times a day. Work on the Swiss spinal cord injury cohort shows that the subacromial space visibly narrows under high-load lifts. That work also shows fatigue concentrating in only a few shoulder muscles, which creates an imbalance between them. Neither effect can currently be tracked outside a laboratory full of force plates and instrumented beams. A tool that measures, from a phone-grade camera and microphone, how hard each hand strikes and how unevenly the two arms share the load would let researchers study whether fatigue and muscle imbalance shift load onto one shoulder in daily life.

Core Claim

For upper-limb contact events, the dominant error in video-based force estimation is uncertainty about contact timing, not poor posture estimates. With a 30 fps camera, a contact onset is known only to within about ±17 ms. That error goes straight into the estimated rate of force rise, and the rise itself happens over roughly 20–40 ms. Our claim is that an acoustic onset and an acoustic contact duration, combined with momentum change measured from video, recover peak hand force and the rate at which joint moments rise much more accurately than vision-only pipelines with the same number of frames. The core relation is the impulse–momentum balance,

$$ F_{\text{peak}} \approx k \, \frac{m\,\Delta v}{\tau}, $$

where $m$ is the effective mass moving through the contact (estimated from body size and the share of weight carried by the arms), $\Delta v$ is that mass's change in velocity across the contact (taken from video frames placed just before and just after it), $\tau$ is the contact rise duration measured from audio, and $k$ is a pulse-shape factor (π/2 for an ideal half-sine pulse) that is learned per surface type. The relation shows where vision fails. $\Delta v$ is reasonably well observed by a camera, but $\tau$ is not. A vision-only pipeline that blurs $\tau$ from 25 ms to 60 ms underestimates peak force by more than half, even when its posture estimates are perfect.

Relation to Prior Work

The closest prior work estimates ground reaction forces from video or sparse signals, and all of it assumes the feet carry the load. GRF-MV predicts ground reaction forces from monocular video. It adds a foot-ground contact loss to a 3D body estimator, uses physics-based inverse kinematics to prevent floating, and derives force from $ma+mg$ averaged over five frames. Its authors report that it fails when weight is borne away from the feet. Its error also rises on fast motions: on GroundLink, MSE is 0.051 for a soccer kick and 0.023 for a tennis serve, compared with 0.003 for the steady "tree_arms" pose. That paper motivates our work, because the five-frame window is exactly what erases impact peaks. Physics-Informed Ground Reaction Dynamics trains a temporal convolutional network on dense motion-capture trajectories with a physics loss from a simulated controller. It reports an average 18% improvement in vertical ground reaction force over the GroundLink baseline, but it covers only the lower limbs and needs dense marker data. Predicting Ground Reaction Force from Inertial Sensors shows that linear regression on a low-rank decomposition of 500 Hz body-worn motion sensor signals reaches about 0.12–0.13 body weights of error in running. We treat it as the wearable competitor, and it also shows that cheap linear models can be enough when the input signal is sampled densely in time. Our proposal differs from all three in three ways: it models multi-point hand contact, it uses audio, and it targets brief impacts rather than gait.

Two lines of work supply method rather than motivation. DenseWarper staggers the shutters of multiple cameras so that four 50 fps cameras yield 200 fps 3D poses. It reaches 21.3 mm joint error on Human3.6M, but it estimates posture only and it loses continuity when motion is fast relative to the stagger interval. We reuse its staggering idea, but schedule frames around audio-detected contacts instead of on a fixed grid. Keyframe selectors such as Adaptive Keyframe Sampling, FOCUS and VideoEspresso show that keeping a few well-chosen frames can beat dense input. They score frames by semantic relevance to a text query, however, and nothing in them would favor the 20 ms in which a hand meets a bench. Temporal Segment Networks sample one frame per fixed segment, which on average misses brief events.

On the audio side, HEAR documents the "deaf response": video dialogue models score about the same with or without audio, yet lose up to 38.2% BLEU-4 on questions that depend on sound. CA2ST fuses frozen spatial, temporal and audio encoders through small cross-attention bottlenecks. Both papers treat audio as a carrier of meaning, not of physical timing. We take a constraint from HEAR: a learned fusion model trained on force targets may also learn to ignore audio. For that reason the audio enters our design as an explicit timing variable in the force solver, not only as tokens to attend over.

The biomechanics literature sets the application regime. Sitting-pivot transfers produce hand forces above 29% of body weight per limb (custom transfer measurement system). Technique scores on the Transfer Assessment Instrument explain 27–79% of the variance in upper-limb joint moments and in how fast they rise (transfer skills and kinetics). The fatigue protocol produces selective over-activation of the pectoralis and upper trapezius (neuromuscular changes during the propulsion fatigue protocol). In the same cohort, fatigue did not change subacromial space (p = 0.570), but high-load lifts did (fatigue, weight relief lifts and subacromial space). This tension is why load magnitude and asymmetry, rather than posture alone, are the quantities worth measuring. Lower-limb fatigue studies show the same pattern of load shifting to other joints (quadriceps fatigue in the scissor jump; ankle compensation). The contracture literature (myostatin-reduced dystrophic dogs; post-polio deformity) motivates why chronic loading asymmetry matters. We do not claim to measure bone remodeling. We aim to provide the field-deployable asymmetry measurement that longitudinal studies of remodeling would need.

Research Program

The system has three parts.

The first is an acoustic contact detector. It takes audio from a room microphone and, separately, from a cheap contact microphone taped under the transfer surface. For each hand strike it outputs an onset time, a rise duration $\tau$, and a short spectral embedding. The detector is trained against force-onset and force-peak times from the instrumented beam, so audio is calibrated against true contact rather than heard as a sound category.

The second is a frame scheduler. Given a budget of $K$ frames per transfer, it spends a fixed fraction uniformly across the motion to track posture. It concentrates the rest in brackets just before and just after each acoustic contact, so that $\Delta v$ is measured from frames that bracket the contact rather than straddle it. With multiple cameras, it staggers their shutters inside those brackets, following DenseWarper.

The third is the force solver. It extends GRF-MV's body model so that the hands, buttocks and feet can all be contacts, and it assigns body weight among them. It then estimates per-hand force in two ways: a physics branch that applies the impulse–momentum relation with the audio $\tau$, and a learned residual branch, a small temporal network that corrects for pulse shape and surface compliance. Joint moments at the wrist, elbow and shoulder come from standard inverse dynamics on the estimated forces and poses.

The outputs are per-hand peak force, impulse, rate of force rise, peak shoulder and elbow moment, moment rate of rise, and a leading-versus-trailing-arm asymmetry index.

There is no public dataset with synchronized audio, video and upper-limb force. We will therefore collect one, following the established transfer protocol: force plates under the chair, bench and feet, a 6-component load-sensing transfer beam at 360 Hz, and Vicon motion capture at 60 Hz. To this we add four consumer cameras recording at 240 fps, so that any lower rate can be simulated by dropping frames, plus a room microphone and contact microphones. Participants will be manual wheelchair users, plus able-bodied people performing depression transfers and weight relief lifts as a secondary group. Each person performs at least five transfers per condition, the minimum the reliability study recommends. Conditions cross three bench surfaces (hard wood, padded vinyl, transfer board) with the session before and after the 15-minute Figure-8 fatigue protocol. The surface factor is deliberate: padded surfaces are where audio should help least, and we want to measure that rather than avoid it.

Baselines

The naive reference splits each person's body weight between the two hands using the average leading/trailing share measured on training participants, and spreads it over a template force curve aligned to the motion. Its errors show how much of the target is explained by body mass alone. Any method that does not clearly beat it on rate of rise is not measuring the impact.

The strongest existing vision method is GRF-MV, adapted fairly. We give it the same multi-contact body model (hands, buttocks, feet), retrain it on our training participants, and supply it with the same $K$ frames placed uniformly. We run two versions, one with its original five-frame smoothing and one with smoothing tuned on validation data, and give each the same hyperparameter search budget as our method. Its reported numbers come from foot contact on GroundLink (MSE 0.051 for soccer kick, 0.023 for tennis serve). For upper-limb contact, no reported result was found on Lacuna, so its performance here will be measured, not quoted. Beating it means at least a 30% reduction in peak-force error and in moment-rate-of-rise error, on held-out participants, at equal frame budget.

The wearable reference is the low-rank linear regression method from Predicting Ground Reaction Force from Inertial Sensors, applied to motion sensors at 500 Hz or more on the wrist and upper arm. Its reported 0.12–0.13 body-weight error is for running vertical force on the Pac-12 dataset, so it does not transfer directly. We expect matching it on hand peak force to be hard, and treat parity as a success, given that our method requires nothing worn by the participant.

The ablation that isolates the key ingredient replaces the acoustic onset with a visual one: the frame where the estimated hand velocity toward the surface crosses zero, refined by interpolation. This variant gets the same scheduler, solver and frame budget. If audio does not beat it, the audio channel is not needed. A second ablation feeds audio as tokens into a CA2ST-style cross-attention fusion with no explicit $\tau$ term. This tests whether a learned fusion uses the sound or develops HEAR's deaf response. The upper reference is dense 240 fps vision-only processing with the same solver. It shows how much of the audio's benefit is simply a substitute for frames.

Why not just…?

Why not just strap motion sensors to the wrists? For laboratory studies with cooperative participants, sensors may be the better instrument, and we include them as a reference rather than dismissing them. The cases against them are specific. Sensors must be fitted to people with impaired hands, which can change how they grip the bench. They measure the acceleration of the segment they sit on, not where or how the hand meets the surface. And extreme impacts, above 13 g on the rings according to the rings review, risk saturating them. The experiment reports both methods' errors on the same transfers, so the size of this trade-off becomes a measured number.

Why not just record at 240 fps and skip both the sparse sampling and the audio? This is the most serious threat, because high-frame-rate phone video is cheap. Two things limit it. At indoor clinical lighting, high frame rates force short exposures, which add noise. Processing every frame with a 3D body estimator costs roughly eight times the compute of 30 fps, which matters for all-day monitoring. The dense 240 fps condition is in the experiments for this reason. If it matches the audio-timed method at equal or acceptable cost, we will report that audio's contribution is efficiency at low frame rates, not accuracy, and narrow the claim to that.

Why not just detect contact visually from the hand stopping? The hand decelerates before it touches the surface, and its velocity is a second-order quantity estimated from noisy keypoints at low frame rate. Its zero crossing is therefore uncertain by about a frame, and often more under motion blur. The visual-onset ablation measures this directly. Audio wins only if its timing error against force-beam onset is much smaller than a frame interval and if that advantage carries through to the rate-of-rise estimate.

Why not just regress force from the audio alone? Sound loudness depends on microphone distance, room acoustics and surface material far more than on force. Audio alone should give good timing but poor magnitude, and that division of labor is the reason for the design. We run an audio-only regressor to confirm it. If audio alone predicts peak force well within one surface type, that would itself be a useful and cheaper result.

Proposed Experiments and Decisive Checks

The main evaluation holds out entire participants. It reports per-hand peak-force error in body weights, impulse error, rate-of-force-rise error, and error in shoulder and elbow moment rate of rise. Moment rate of rise is the quantity the Transfer Assessment Instrument relates to technique, so we also check that our estimates reproduce the reported relationship between instrument scores and moments. We sweep the frame budget $K$ from uniform 30 fps equivalents down to a few percent of 240 fps.

The decisive check for the main claim is the comparison between acoustic and visual onset at equal frame budget, broken down by surface. The claim predicts three things: a large audio advantage on hard surfaces, a smaller one on padded vinyl, and close to none on the transfer board. It also predicts that the advantage shrinks as the frame rate rises toward 240 fps. A uniform gain across all surfaces would suggest the model is using sound as a general cue rather than as contact timing, so we would treat that result with suspicion, not as a success.

A second check tests whether the force estimate's errors are explained by timing. We will deliberately corrupt the audio onset by known offsets and confirm that peak-force error changes as $1/\tau$ predicts.

A third, application-level check asks whether the camera-and-microphone system detects the change in leading-versus-trailing arm asymmetry before versus after the fatigue protocol with the same sign and significance as the force beam does. This is the measurement that would let fatigue-driven load shifting be tracked outside the lab. If the beam shows no fatigue-related asymmetry change, consistent with the null result for subacromial space, we report that as a biomechanical finding and judge the tool by its agreement with the beam.

Finally, we run the vision-only version of the solver on GroundLink's foot-contact motions, which are not reported to include audio. This confirms that extending the body model to hand contacts does not degrade the foot case relative to GRF-MV's reported soccer-kick and tennis-serve errors.

References

  1. GRF-MV: Ground Reaction Force Estimation from Monocular Video
  2. Physics-Informed Ground Reaction Dynamics from Human Motion Capture
  3. Predicting Ground Reaction Force from Inertial Sensors
  4. From Sparse to Dense: Spatio-Temporal Fusion for Multi-View 3D Human Pose Estimation with DenseWarper
  5. Adaptive Keyframe Sampling for Long Video Understanding
  6. FOCUS: Efficient Keyframe Selection for Long Video Understanding
  7. VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection
  8. Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
  9. HEAR: Hearing Enhanced Audio Response for Video-grounded Dialogue
  10. ${\text{CA}^{2}\text{ST}}$: Cross-Attention in Audio, Space, and Time for Holistic Video Recognition
  11. Development of custom measurement system for biomechanical evaluation of independent wheelchair transfers
  12. The Relationship between Independent Transfer Skills and Upper Limb Kinetics in Wheelchair Users
  13. Changes in neuromuscular activation, heart rate and rate of perceived exertion over the course of a wheelchair propulsion fatigue protocol
  14. Effect of Fatiguing Wheelchair Propulsion and Weight Relief Lifts on Subacromial Space in Wheelchair Users
  15. Analysis of Quadriceps Fatigue Effects on Lower Extremity Injury Risks During Landing Phases in Badminton Scissor Jump
  16. Impact of Quadriceps Muscle Fatigue on Ankle Joint Compensation Strategies During Single-Leg Vertical Jump Landing
  17. Dystrophin-deficient dogs with reduced myostatin have unequal muscle growth and greater joint contractures
  18. Management of Rigid Post-poliomyelitis Equinocavovarus Deformities: A Brief Review
  19. Technical elements on the rings in men's artistic gymnastics—a systematic review