Measuring how many calories a person burns still means strapping a mask to their face and connecting it to a metabolic analyzer. This method, called indirect calorimetry, reads oxygen use from breath. Video models promise the same number with nothing but a camera, but on the one benchmark with real calorimeter labels, a model that sees only the skeleton is off by about 29%. That is worse than a smartwatch. We argue that two specific mistakes cause much of this gap, and that both can be tested.
The first mistake concerns where coordinates are measured from. Pose pipelines crop each person into a box and normalize the joints inside it. This throws away the quantities that set mechanical work: how far the center of mass rises against gravity, and how far the limbs reach relative to the body. The second mistake is timing. The calorimeter does not report instantaneous effort. Oxygen uptake trails exertion by tens of seconds to minutes, yet video models regress each window directly onto this lagged signal. We propose a convolutional model over pose heatmaps whose added coordinate channels are measured in a gravity-aligned, body-height-normalized frame anchored at the hip and the ground, rather than in image or crop pixels. Its output is a latent "metabolic demand" that passes through a learned first-order oxygen-kinetics filter before being compared with calorimeter readings. In two sentences: give the convolution coordinates that mean something physically, and stop asking it to predict a delayed breath measurement as if it were instantaneous.
Core Claim
Coordinate-aware convolution, which appends coordinate maps as extra input channels so filters can respond to where a feature sits, helps on tasks where absolute position carries meaning, and gives essentially nothing on tasks where it does not. An intriguing failing of convolutional neural networks and the CoordConv solution introduced it and showed both outcomes: perfect held-out-quadrant generalization on its synthetic Not-so-Clevr task, and a +0.04% top-1 gain on ImageNet. Our claim is that energy estimation from pose is a position-meaningful task only when the coordinates are expressed in a physical frame.
If coordinates are attached in crop space, CoordConv only re-supplies information that a coordinate-token Transformer already has. If they are attached in a body-anchored frame (height above the ground contact line and horizontal offset from the hip, both divided by the person's standing height), the network can read vertical excursion and limb extension directly. Separately, we claim that a calorimeter-aware output layer removes a systematic error caused by lag, and that this error is largest exactly where wrist devices are known to fail: interval training and rapid intensity changes (Accuracy of Wrist-Worn Activity Monitors During Common Daily Physical Activities and Types of Structured Exercise: Evaluation Study).
Relation to Prior Work
Three works motivate the gap. E3V-K5: An Authentic Benchmark for Redefining Video-Based Energy Expenditure Estimation and Vision-based multimodal energy expenditure estimation for aerobic exercise in adults recorded video alongside COSMED K5 calorimetry. They built E3SFormer, a two-branch Transformer over RTMPose joint coordinates, and reported 28.81% mean relative error (MRE) from skeleton alone. Their attention maps point to the hip, the approximate center of mass, as the most informative joint. That finding is direct evidence that center-of-mass motion matters, but their model sees coordinates only as tokens and performs window-to-label regression with no account of oxygen kinetics.
Should I take a walk? Estimating Energy Expenditure from Video Data showed that holistic 3D convolutional networks on its Vid2Burn benchmark degrade from about 34.5 to about 159.5 kcal/hr on unseen activities. The authors attribute this to memorized scene context. We do not claim that coordinate channels fix scene memorization in RGB models. We work on pose heatmaps, which have no background, and test generalization to held-out activities there.
Two works supply the method. Revisiting Skeleton-based Action Recognition introduced PoseConv3D, a 3D convolutional network over stacked Gaussian joint heatmaps. It is robust to dropped keypoints (under 1% accuracy loss versus over 14% for graph networks), but its kernels are translation-invariant inside a subject-centered crop. We reuse its representation and add body-anchored coordinate channels using the uber-research/coordconv layer. Location Matters: Harnessing Spatial Information to Enhance the Segmentation of the Inferior Alveolar Canal in CBCTs showed that cropped 3D patches lose context and that injecting absolute position recovers it (Dice 0.810 to 0.824, with run-to-run variance halved). That paper restores position in a fixed scanner frame. Our setting differs because the frame has to be built per person, from body height and ground contact.
One work supplies the dynamics. From Lab to Wrist: Bridging Metabolic Monitoring and Consumer Wearables for Heart Rate and Oxygen Consumption Modeling modeled oxygen uptake as a continuous-time process with Neural ODEs and a learned Kalman filter on wearable signals, reaching 11.4–13% MAPE against COSMED K5. We borrow its premise that oxygen uptake follows a dynamic response to load, but use a much simpler first-order kernel, driven by video instead of heart rate.
Sensing leg movement enhances wearable monitoring of energy expenditure constrains the design. Two leg IMUs reached 13% cumulative error, which shows that lower-limb mechanics carry most of the signal and sets a bar that a contactless method should eventually approach.
Research Program
The input is a PoseConv3D-style heatmap volume of $K$ joints over $T$ frames on an $H \times W$ grid. For each clip we estimate the ground line $y_g$ (the lowest ankle position over a gait cycle), the hip trajectory $x_h(t)$, and standing height $h$ (median head-to-ankle extent). We then append two dense channels at every pixel and frame:
$$u = \frac{x - x_h(t)}{h}, \qquad v = \frac{y_g - y}{h}.$$
Here $x, y$ are the pixel coordinates, $u$ is the horizontal offset from the hip, and $v$ is the height above the ground, both in body heights. Because these channels are computed before cropping and rescaling, vertical excursion survives normalization of the crop. Because they are divided by $h$, camera distance cancels to first order. An optional third channel holds the frame-to-frame change in hip height, to separate motion against gravity from posture.
The backbone outputs a per-window demand $D(t)$. Calorimeter oxygen uptake is then predicted as
$$\widehat{VO_2}(t) = VO_{2,\text{rest}} + \int_0^t \frac{1}{\tau}\, e^{-(t-s)/\tau}\, D(s)\, ds,$$
with a learned time constant $\tau$, optionally conditioned on activity. The exponential form comes from the standard first-order description of oxygen-uptake kinetics. The loss is applied to $\widehat{VO_2}$, and evaluation reports both the filtered prediction against K5 and the cumulative energy, which the lag does not change.
This filter needs clips ordered in time within each session. A first step is to confirm from the E3V repository (zsxm1998/E3V) that session timestamps are available. If they are not, the kinetics component is tested only on the synchronized runner dataset of From Lab to Wrist, conditional on access, and E3V-K5 results report the coordinate-channel contribution alone.
Baselines
The naive reference is a per-activity, per-intensity mean predictor fit on the training subjects. This is effectively a MET-table lookup scaled by body mass, and it shows how much of the signal is just activity identity. A second reference is the commercial smartwatch on the same E3V-K5 test cohort, which Vision-based multimodal energy expenditure estimation for aerobic exercise in adults reports at 18.10% MRE.
The strongest skeleton-only competitor is E3SFormer, at a reported 28.81% MRE on skeleton-only input (E3V-K5). With heart rate and physical attributes added, it is reported at 15.32% MRE in one paper and 18.01% MRE with PCC 0.9157 in the other. We will rerun the released code on our splits rather than quote either multimodal number, because the two disagree. E3SFormer is the strongest competitor because it already consumes explicit coordinates and was designed on this exact data.
Plain PoseConv3D on E3V-K5 is the architectural parent; no reported result was found on Lacuna for PoseConv3D on E3V-K5, so we will produce it ourselves. All models get the same RTMPose keypoints, the same subject-disjoint splits, the same per-model hyperparameter search budget (equal number of trials), and comparable parameter counts.
The ablations that isolate the key ingredient are PoseConv3D with standard CoordConv in crop coordinates and PoseConv3D with CoordConv in uncropped image coordinates. These are compared against body-anchored channels, each with and without the kinetics filter. Beating the baselines means a skeleton-only MRE clearly below E3SFormer's rerun value across five seeds, together with a body-anchored-over-crop-CoordConv gain larger than seed variance. Without the second condition, any win would be attributable to the heatmap backbone rather than the coordinate frame.
Why not just…?
Why not just compute hand-crafted kinematic features (hip vertical velocity, limb extension, cadence) from keypoints and feed them to E3SFormer or a gradient-boosted regressor? This is the most serious threat, because body-anchored channels encode roughly these quantities. If a feature-augmented E3SFormer matches our model, the honest conclusion is that the physical frame matters and the convolution does not. We will run this baseline with the same features computed from the same keypoints. The case for heatmap convolution rests on its robustness to keypoint dropout (Revisiting Skeleton-based Action Recognition), so we will also compare both under synthetic keypoint dropout and jitter. That is where hand-crafted derivatives of noisy keypoints should break first.
Why not just add heart rate, since multimodal E3SFormer already beats the smartwatch? Heart rate needs a worn sensor, which defeats a contactless system, and it inherits the cardiovascular lag that hurts wrist devices. We will report multimodal variants of our model for completeness. The headline comparison stays skeleton-only, where the 28.81% gap is largest.
Why not just shift labels by a fixed delay instead of learning a kinetics filter? A fixed shift cannot represent the asymmetric rise and fall of oxygen uptake. We include a tuned-shift baseline. If it matches the learned filter on interval sessions, we will drop the filter in favor of the shift.
Why not just use uncropped frames so zero-padding gives position? The CoordConv paper shows that padding-based localization is sample-inefficient and fails on held-out regions. Uncropped image coordinates also vary with camera placement. The uncropped-CoordConv ablation tests this directly, and E3V-K5's multiple viewpoints let us hold out a camera view.
Proposed Experiments
The primary experiment is subject-disjoint evaluation on E3V-K5, reporting MRE, PCC, and cumulative-energy error per session. The decisive checks follow from the claims.
First, leave-one-activity-out across E3V-K5's six exercise types tests whether physically framed coordinates transfer to an unseen movement. Running and rope skipping, with large vertical excursion, should benefit most. Cycling and elliptical, which have little center-of-mass motion, may not, and we will report that openly as a limit of the approach. We also run the same protocol on Vid2Burn-Diverse to probe the documented unseen-activity collapse. Because Vid2Burn labels come from MET tables rather than calorimetry, the kinetics filter is disabled there.
Second, leave-one-viewpoint-out tests whether height normalization actually cancels camera geometry.
Third, we compare per-segment error on HIIT and intensity-transition segments against steady-state segments. The kinetics filter should shrink transition error while leaving cumulative error nearly unchanged; a drop in transition error without that pattern would suggest the filter is fitting noise.
Fourth, as a sanity check, we render synthetic skeletons whose vertical excursion is varied independently of posture. Body-anchored models should regress excursion linearly, and crop-CoordConv models should not, mirroring the quadrant-split diagnostic of the CoordConv paper.
Finally, we report the known blind spot explicitly. Load or incline that changes effort without changing visible motion should leave every vision model wrong. We will quantify how much of the residual error sits in E3V-K5's intensity levels that differ mainly in resistance.
References
- An intriguing failing of convolutional neural networks and the CoordConv solution
- Accuracy of Wrist-Worn Activity Monitors During Common Daily Physical Activities and Types of Structured Exercise: Evaluation Study
- E3V-K5: An Authentic Benchmark for Redefining Video-Based Energy Expenditure Estimation
- Vision-based multimodal energy expenditure estimation for aerobic exercise in adults
- Should I take a walk? Estimating Energy Expenditure from Video Data
- Revisiting Skeleton-based Action Recognition
- Location Matters: Harnessing Spatial Information to Enhance the Segmentation of the Inferior Alveolar Canal in CBCTs
- From Lab to Wrist: Bridging Metabolic Monitoring and Consumer Wearables for Heart Rate and Oxygen Consumption Modeling
- Sensing leg movement enhances wearable monitoring of energy expenditure