Protecting Rare Poses from Common-Pose Gradients

Human pose estimators, which locate body joints in an image, work well on people standing or walking and fail on gymnasts, sprinters and anyone lying down. The usual explanation is data imbalance. Roughly 85% of COCO people are upright (Exploring Rare Pose in Human Pose Estimation), and the standard remedies follow from that diagnosis: show rare poses more often, weight their loss more heavily, or synthesize more of them.

This proposal tests a different, sharper hypothesis. During training, the gradient from common poses may point against the gradient from rare poses in the shared network, so each step toward upright accuracy actively undoes progress on the tail. Reweighting only changes how much each gradient counts. It cannot remove a component that points the wrong way.

If this conflict exists, a method from multi-task learning should help. Multi-task learning trains one shared network on several objectives at once, and "gradient surgery" removes the conflicting part of one objective's gradient before the objectives are combined. The idea here is to treat "common poses" and "rare poses" as two tasks inside ordinary pose estimation and apply that surgery. The result is a single model with zero inference overhead, and the first step is a cheap measurement that could kill the idea outright.

Core Claim

The claim has two parts, and the first gates the second. First, in a standard heatmap pose network trained on COCO and MPII, the gradients of the loss on head-cluster (common) poses and on tail-cluster (rare) poses have negative cosine similarity on a substantial fraction of steps. This should hold in the later shared layers, which is where the upright bias should live. Second, if the conflict is real, removing the head-gradient component that opposes the tail gradient should improve rare-pose accuracy more than rarity reweighting with matched gradient magnitudes. It should do so without the head-set loss that reweighting typically causes.

The mechanism motivating this comes from Ukita. That work found that upright structural priors override local image evidence on inverted and athletic bodies, pulling predictions toward standing templates. A shared representation shaped mostly by head-pose gradients is a natural place for such a template to form.

Relation to Prior Work

Two papers are the closest, and both leave this experiment undone.

TS-MOF is the only work in the file that brings gradient surgery (a deterministic PCGrad variant) to long-tailed learning. It uses that surgery for class-balanced image classification, and its "tasks" are several classifier heads, each trained with a different long-tail loss on top of a frozen backbone. We differ in three ways:

  • our tasks are strata of the data distribution, not alternative losses;
  • surgery acts on the backbone itself, end to end;
  • the output is structured spatial regression, not a class label.

Because TS-MOF freezes the backbone, it implicitly bets that imbalanced pretraining yields features adequate for the tail. We run that bet directly as a baseline.

Tang et al. frame pose estimation as multi-task learning over body parts. They show that unrelated parts suffer negative transfer and fix this with a static, dataset-wide branching network. Their tasks are joints; ours are pose configurations. Their fix is architectural and fixed; ours is an optimization change that adds no parameters.

The file also contains work that supplies our method:

  • PCGrad contributes the projection rule and the observation that magnitude rebalancing does not resolve directional conflict.
  • RotoGrad handles conflict by rotating the shared features per task instead of editing gradients. We keep it as a secondary variant.
  • Exploring Rare Pose provides the rarity definition: K-means clustering of normalized 2D skeletons, with rarity measured as distance to the nearest cluster center relative to a threshold.

Research Program

We cluster training skeletons with the Hwang et al. procedure and label each training instance as head or tail. Rarity is computed from ground truth, so it is used only at training time. Every mini-batch has a fixed tail quota, and every method we compare sees exactly the same batches. This isolates how gradients are combined from how data is sampled.

At each step we compute the head-stratum gradient $g_h$ and the tail-stratum gradient $g_t$ with respect to the shared parameters, using two backward passes. If they conflict ($g_h \cdot g_t < 0$), we apply a tail-protecting, asymmetric projection. Unlike symmetric PCGrad, only the head gradient is edited:

$$\tilde g_h = g_h - \frac{g_h \cdot g_t}{\|g_t\|^2}\, g_t, \qquad g = \tilde g_h + g_t .$$

Here $\tilde g_h$ is the head gradient with its tail-opposing component removed, and $g$ is the applied update. The two-task setting keeps the known memory cost of gradient surgery modest. We also test restricting surgery to the last backbone stages, where we expect the conflict to concentrate.

The first deliverable is diagnostic. We log, per layer and over training:

  • the head–tail gradient cosine;
  • the fraction of conflicting steps;
  • an "upright collapse rate," meaning the fraction of tail instances whose predicted skeleton falls in an upright cluster.

The second deliverable is the controlled comparison below. Secondary extensions, attempted only if the core result holds, are a K-way version over multiple rarity tiers and a RotoGrad-style feature rotation for comparison.

Baselines

The naive reference is plain fine-tuning of the same backbone with the same tail-quota batches and an unweighted loss. It shows what the batch composition alone buys.

The strongest established competitor is oversampling of rare poses (DRP) from Exploring Rare Pose. That paper reports +6.09% PCKh on MPII extreme outliers (distance threshold > 1.2) and up to 13.5 mAP improvement on the rarest COCO subsets. The file reports these only as gains, so we will reproduce DRP ourselves under our backbone, schedule and tuning budget. Its sibling, the cluster-distance weighted loss (WLCD), is run the same way. Each baseline gets the same number of hyperparameter trials as our method.

Beating these baselines means a statistically reliable gain on the MPII outlier and rarest-COCO subsets over the best-tuned of DRP and WLCD, across at least three seeds, with head-subset accuracy no worse than plain fine-tuning.

The key ablation is magnitude-matched reweighting. We rescale $g_h$ at every step so its norm equals that of $\tilde g_h$, but we keep its direction. This spends the same "tail emphasis" as the projection without removing the conflicting direction. If it matches our method, the story is magnitude, not direction.

A second control applies the same projection between two random halves of the batch. This checks whether any gain comes merely from the regularizing noise of surgery.

We also run a TS-MOF-style decoupled baseline: freeze the backbone after standard training, then train head and tail heads with deterministic projection, following TS-MOF. This tests whether end-to-end surgery is needed at all.

Finally, the two-model generalist/specialist ensemble (DACP) from the same Hwang et al. paper is an accuracy reference point at twice the inference cost. For absolute context on MPII, the part-branching network of Tang et al. reports 92.7% [email protected] on the test set.

Why not just…?

Why not just oversample rare poses? Oversampling already appears in our design, since all methods share a fixed tail quota. DRP tuned more aggressively is therefore the natural competitor. It can raise the tail's share of the loss, but if head and tail gradients point in opposite directions, more tail samples mean more cancellation, not less. Hwang et al. also flag the overfitting risk of repeated samples. If our diagnostic finds little conflict, DRP is the right answer and we will say so.

Why not just weight the loss by rarity? This is the most serious threat, because in a two-task setting reweighting and projection can look similar. TS-MOF documents the seesaw effect, where gains in tail accuracy cost head accuracy. PCGrad argues that weights cannot fix direction. Neither paper shows this for pose. The magnitude-matched ablation settles the question directly: identical norms, different directions.

Why not just train a specialist model for rare poses? DACP works, but Hwang et al. name its ensemble overhead as a limitation, and it needs a test-time selector. We report it as an accuracy reference point. A single model that closes most of the gap to DACP would be the practical win.

Why not just use a huge pretrained backbone like Sapiens? Sapiens gains +7.6 mAP on whole-body pose from 300M-image pretraining, but it never reports rare-pose subsets. Its fine-tuning is still driven by imbalanced labels. If compute allows, we will repeat the core comparison on a pretrained ViT to check that the conflict persists at scale. If it vanishes, that is an important negative result.

Proposed Experiments

Training uses COCO 2017 and MPII with a standard heatmap backbone. Evaluation covers:

  • COCO rarity-stratified mAP;
  • MPII PCKh on the distance-threshold > 1.2 outliers and on the head subset;
  • Leeds Sports Pose, zero-shot, as held-out sports data;
  • 2D keypoints on AthletePose3D, both zero-shot and after fine-tuning. The best reported 2D result there is 95.7 PDJ (MogaNet).

The first decisive check is the gradient-conflict diagnostic, run before any method comparison. If the conflicting-step fraction in shared layers is near what the random-split control shows, we stop. We then report that rare-pose failure in these models is a magnitude problem best handled by DRP or WLCD.

The second check is the magnitude-matched ablation on the tail subsets.

The third check measures head-subset accuracy and upright collapse rate. The prediction is that projection reduces collapse on tail instances while leaving head accuracy flat, and that reweighting reduces collapse only by trading away head accuracy.

The fourth check compares against the frozen-backbone decoupled baseline. If end-to-end surgery clearly beats it, that answers, for structured pose regression, the question TS-MOF's design leaves open: whether a backbone trained on imbalanced data can support the tail with only its heads adapted.