Many systems that recognize human actions from depth sensors or point clouds first flatten the 3D data onto a 2D grid. That way, fast image networks can be used. The usual worry about this flattening is geometric: two things stacked along the collapsed axis merge into one pixel. We argue that, for action recognition, the more damaging loss is about motion. When a point cloud sequence is projected onto a plane, any movement along the plane's normal, the direction that gets collapsed, disappears from the frame-to-frame differences that "motion maps" are built from. Body shape, by contrast, survives. Punching toward the camera, pushing, or kicking forward then look almost static in a front-view motion map. The network falls back on appearance, which is exactly the shortcut that motion-centric recognition is meant to avoid.
This note proposes to measure that loss class by class, attribute it using gradient-based saliency, and repair it cheaply. The repair keeps a single projected view and adds a signed "out-of-plane velocity" channel, computed at several time scales, which a multi-scale dilated temporal convolution then reads. The central prediction can be checked before any model is trained. On NTU RGB+D, the share of each action class's skeleton motion that lies along the projection normal should predict how much accuracy that class loses when a native-3D model is replaced by a projected one. It should also predict how much of that loss the new channel recovers.
Core Claim
The claim has three parts.
First, projection-based recognizers are appearance-biased for a specific, measurable reason: they lose the velocity component along the collapsed axis. This offers one explanation for the weak motion branch that PointMapNet reports. That paper splits point sequences into an appearance map and a motion map, built from inter-frame coordinate differences, and its authors concede that the motion map is "insufficient" next to the appearance map. PointMapNet's maps are learned features rather than a fixed geometric plane, so we treat it as motivating evidence, not a direct test case. Our controlled experiments use explicit planar projections.
Second, the loss is concentrated, not uniform. It should scale with how much a class's discriminative motion lies along the normal, so classes with mostly lateral motion should barely suffer.
Third, adding back one channel of signed normal velocity, at several temporal strides, should recover most of the gap to native-3D models such as PSTNet. It should do so at close to the cost of a single 2D view.
What is new is treating projection loss as a loss of motion, predicting its per-class size from motion statistics computed independently of any model, and verifying with attributions and interventions that the repaired model actually uses the recovered motion.
Research Program
Inputs and controlled projections. We take the NTU RGB+D 60 and MSR-Action3D depth sequences as point clouds. From these we build controlled projected inputs onto three planes: the sensor front view, a top view, and a side view. Each frame becomes a 2D occupancy-and-depth image. This lets us vary which axis is collapsed while holding data, labels, and backbone fixed.
Out-of-plane velocity channel. For each occupied cell, we estimate the signed velocity along the normal. We match each point to its nearest neighbor in frame $t+d$ within a small radius, then average the normal component of the displacement in each cell. Here $d$ is a temporal stride. Nearest-neighbor matching avoids needing true point correspondences, which PSTNet notes are absent in point cloud sequences. We compute the channel at strides $d \in \{1, 2, 4\}$. Small strides capture fast motion. Large strides average out the sensor jitter that makes single-step differences noisy, a problem the file attributes to coarse motion vectors and frame differences.
Multi-scale dilated temporal convolution. Dilated convolution means spacing a kernel's taps apart so it covers a wider window with the same number of weights. Differencing at stride $d$ is itself a dilated temporal derivative. We therefore feed the multi-stride channels into a temporal block whose dilation rates vary cyclically across channels, in the style of PSConv. PSConv mixes dilation rates within one layer at the parameter count of a standard convolution and reported an ImageNet top-1 error of 21.1% versus 22.8% for ResNet-50. It was evaluated only on images, so here it supplies the technique.
Two papers constrain the design. PDAN found that stacking more than five dilated blocks hurt short actions. TFE-DCN found an optimal depth of three dilated layers and warns that wide temporal windows let background bleed into boundaries. Both worked on 1D snippet features with no spatial projection. Following them, we cap temporal depth at three dilated layers and report results split by action duration.
Attribution as a measurement tool. We use gradient-based saliency, meaning derivatives of the class score with respect to inputs. Exploring Explainability in Video Action Recognition showed that Grad-CAM heatmaps in video models often highlight static background (the "background trap"). For that reason we do not rely on spatial heatmaps. Instead we use Integrated Gradients, which accumulates gradients along a path from a neutral baseline to avoid the saturation problem of plain gradients. We use it to split each prediction's attribution across three input groups: appearance (occupancy and depth), in-plane motion, and out-of-plane motion.
From this we define a per-class motion reliance ratio: the share of attribution falling on motion channels. Because attributions can mislead, every attribution claim is paired with an intervention. We zero out or shuffle the channel group the attribution highlights and measure the resulting accuracy drop. The test we borrow from On the Integration of Optical Flow and Action Recognition serves as a causal check of motion reliance. That paper showed flow streams hold near 84% on UCF101 under color permutation while RGB collapses from 85% to 34%. Our version perturbs appearance by rescaling subjects' body proportions and shuffling static frames.
Model-independent predictor. For each NTU class, we compute the out-of-plane motion fraction for a given plane. This is the share of total joint-velocity energy, taken from the provided skeletons, that lies along that plane's normal. Because it uses no learned model, it is a clean prior for the per-class predictions.
Baselines
The naive reference point is a single front-view projected recognizer with ordinary frame-difference motion maps. It uses the same 2D backbone, trained with the same schedule. Every gain is reported relative to this model.
The strongest native-3D competitor is PSTNet. It convolves directly on point sequences, using "point tubes" that follow local neighborhoods through time, and so never collapses an axis. It reports 91.20% accuracy on MSR-Action3D. The file gives no absolute NTU RGB+D 60 or 120 number for it; no reported result found on Lacuna for PSTNet NTU absolute accuracy. We will retrain it on our splits. PSTNet is the right ceiling because it represents what projection gives up. Our method beats it in the relevant sense if it comes within one point of PSTNet's accuracy at a substantially lower latency and memory footprint, or exceeds it on the high out-of-plane classes at matched compute.
The strongest projection-style competitor is PointMapNet. It reports 89.4% cross-subject and 96.7% cross-view accuracy on NTU RGB+D 60, and 91.91% on MSR-Action3D. Beating it means exceeding 89.4% cross-subject on NTU RGB+D 60 under the same evaluation protocol. Beyond the headline number, the gain should come mainly from classes with a high out-of-plane motion fraction.
All retrained models get the same frame count, the same augmentation, and an equal hyperparameter search budget. Two reference points get separate treatment. Published numbers from the original papers are quoted beside our retrained results rather than substituted for them. For Complex-YOLO, we borrow only its design, which keeps a maximum-height channel to soften vertical collapse.
Three ablations isolate the key ingredient. In the first, the out-of-plane channel is replaced by a per-cell difference of depth values across frames, which is a cheap proxy. In the second, the channel is kept but uses a single stride with a non-dilated temporal convolution. In the third, the same parameters are given to extra in-plane channels instead, so the comparison checks that capacity alone does not explain any gain.
Why not just…?
Why not just use PSTNet and skip projection? If compute is no object, this may be the right answer, and we take it seriously. The case for projection is efficiency and reuse of 2D hardware. The file's evidence for that case is indirect: voxel approaches such as VoxelNet pay heavily in 3D-convolution memory, while BEV projection runs at 50.4 FPS in Complex-YOLO. We will report an accuracy-versus-latency curve. If PSTNet dominates that curve, the proposal's contribution narrows to the diagnosis: explaining where and why projected models fail.
Why not just project onto three orthogonal planes? Multiple views do restore the motion lost along any single normal, so this is a real threat. Our argument is cost: one extra channel versus three full views. The experiment compares accuracy against FLOPs. If three views match our single-view model at similar cost, we will say so. The per-class correlation analysis would still identify which classes need which view.
Why not just difference the depth channel over time? This proxy is the first ablation, and it may come close. It conflates a cell becoming occupied or empty with true motion along the normal, and it has no multi-stride smoothing. We expect it to fail mainly on slow or noisy movements. If it matches the full method, we will adopt the simpler version, and the motion-loss claim stands unchanged.
Proposed Experiments
The first decisive check runs before any new model is trained. Across NTU RGB+D 60 classes, take each class's accuracy drop from PSTNet to the naive front-view model and compute the Spearman correlation with that class's out-of-plane motion fraction. We predict a clearly positive correlation. We further predict that the correlation should move when the plane changes: top-view projection should hurt vertically moving classes such as sitting down and standing up instead. If the correlation is near zero, the motion-loss hypothesis is wrong, and we would revisit the idea.
The second check tests whether the repair recovers the gap selectively. We expect larger per-class recovery on high out-of-plane classes than on lateral ones, along with a rise in the attribution-based motion reliance ratio. Zeroing the new channel should remove the recovered accuracy; if it does not, the gain came from capacity rather than motion.
The third check is appearance robustness. Under body-proportion rescaling and static-frame shuffling, the repaired model should degrade less than the naive projected model and PointMapNet. This would show the recovered motion reduces appearance reliance, as Sevilla-Lara and colleagues found for optical flow.
Finally, we test on MSR-Action3D and UTD-MHAD, where PointMapNet reports 91.61% on UTD-MHAD. These checks confirm that the effect is not specific to NTU. We report results by action duration so that any harm to short actions from dilation is visible.
References
- PointMapNet: Point Cloud Feature Map Network for 3D Human Action Recognition
- PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences
- PSConv: Squeezing Feature Pyramid into One Compact Poly-Scale Convolutional Layer
- PDAN: Pyramid Dilated Attention Network for Action Detection
- Temporal Feature Enhancement Dilated Convolution Network for Weakly-Supervised Temporal Action Localization
- Exploring Explainability in Video Action Recognition
- On the Integration of Optical Flow and Action Recognition
- Complex-YOLO: An Euler-Region-Proposal for Real-Time 3D Object Detection on Point Clouds
- VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection