Per-Keypoint Prompt Dropout for Category-Agnostic Pose Estimation

To find the keypoints of an object category a model has never seen, such as the corners of an unfamiliar cabinet or the joints of a rare animal, today's methods ask the user for one kind of hint. Some ask for an annotated example image. Others ask for a written description of each point ("top left side of the backrest"). In practice users mix the two. They click a few points on an example photo and describe the rest in words, because some points are easy to click and others are hard to see but easy to name. We propose to train a single keypoint localizer that accepts, for each keypoint separately, whichever hints the user happened to give: a visual example, a text description, both, or neither, plus an optional skeleton linking the points. The training trick comes from multimodal learning. During training we randomly delete an entire hint type for each keypoint, and sometimes the whole skeleton, so the model cannot come to depend on any one of them.

The bet is that this per-keypoint deletion does more than make the model tolerate missing inputs. Every keypoint is processed jointly with the others, and they share information along the skeleton. A point described only in words can therefore be placed using its visually anchored neighbors, and a visually anchored point can borrow the left/right meaning of a neighbor's description. If that is right, one jointly trained model should beat the obvious alternative: running an image-prompted model on the clicked points, a text-prompted model on the described points, and stitching the results together.

Background and the gap

The pose estimation literature began with category-specific specialists trained for one class and a fixed set of joints, for example human body models (Shang Liu et al.) and rigid-object pose (Enric Corona et al.). These fail outright on new categories. They also require large annotation efforts per class (Yuan Liu et al.; Yann Labbé et al.). That cost motivates category-agnostic pose estimation (CAPE): localizing keypoints on an unseen category given only a few "support" prompts.

The CAPE line has split by prompt type.

Image prompts. POMNet defined the MP-100 benchmark and matches annotated support-image features to the query image. SCAPE simplifies this to one-stage attention and direct coordinate regression, with a refiner that infers occluded points from visible ones.

Image prompts plus a skeleton. GraphCape and EdgeCape add skeleton message passing, and EdgeCape also learns edge weights. GenCape infers the skeleton from the support image itself.

Text prompts. CapeX replaces support images with a graph of text descriptions. CapeLLM feeds query tokens and detailed descriptions to a large language model that writes coordinates as text.

These papers motivate our work in two ways. First, each prompt type has a documented weakness. CapeX reports confusion over which "left" is meant and pastes known layouts onto unrelated shapes. The image-matching models degrade under self-occlusion and on visually identical symmetric parts. Second, no model in this line can take more than one prompt type at a time, let alone a different type per keypoint.

On the modality-dropout side, several papers supply the method we reuse. M3 zeroes whole modalities in sentiment analysis. It shows that deleting a modality entirely breaks reliance on the dominant stream, while ordinary feature dropout and partial masking do not. HeMIS and mmFormer train segmentation networks on random subsets of MRI sequences so they degrade gracefully when a sequence is missing. UMDF adds distillation between two differently masked views of the same sample. All of these drop a modality for a whole sample.

Our setting differs structurally. The prediction is a set of interacting outputs (the keypoints), and the relevant missingness is per output, not per sample.

Two papers constrain the design. Wasserstein Modality Alignment finds that forcing modality representations to coincide hurts performance. We therefore fuse image and text embeddings additively with learned "absent" tokens rather than aligning them. Compositional Failure in Audio-Visual LLMs finds that asymmetric modality dropout shifted surface answer bias in audio-visual language models without improving conflict resolution; the base model stayed at 51.7% on the matching task. That is a warning that dropout alone may not make a model actually combine evidence. We treat it as a hypothesis to test, not an assumption to dismiss.

Core Claim

A CAPE model whose keypoint tokens are built from independently droppable image, text, and skeleton prompts, and trained with per-keypoint modality dropout, should do two things:

  1. Match or beat single-prompt specialists when all keypoints share one prompt type.
  2. Clearly beat per-keypoint stitching of specialists in the mixed-prompt regime.

The advantage in the mixed regime should concentrate on text-only keypoints that have visually anchored skeleton neighbors, and on left/right symmetric pairs.

Research Program

Architecture. We build on CapeX's text-graph pipeline because its encoder already handles text keypoint prompts and its graph decoder already handles skeletons. We add a support-image branch that pools backbone features at annotated support locations, as in GraphCape and SCAPE.

For keypoint $k$, the prompt token is

$$t_k = m^{I}_k\, e^{I}_k + (1-m^{I}_k)\, a^{I} + m^{T}_k\, e^{T}_k + (1-m^{T}_k)\, a^{T},$$

where:

  • $e^{I}_k$ is the pooled support-image feature;
  • $e^{T}_k$ is the text embedding of the description;
  • $m^{I}_k, m^{T}_k \in \{0,1\}$ indicate which prompts are present;
  • $a^{I}, a^{T}$ are learned "absent" vectors.

We use learned absent vectors instead of zeros to avoid the out-of-distribution zero-padding problem that the modality-dropout literature flags for normalization layers. The skeleton enters as the decoder's graph. When it is dropped, the decoder falls back to plain self-attention.

Training. Each episode first samples a regime: all-image, all-text, all-both, or mixed. In the mixed regime, each keypoint independently draws its mask, with the constraint that at least one prompt is present per keypoint. The skeleton is dropped with probability $p_G$.

We also test an optional consistency term that pulls predictions from a masked view toward those from the complete view of the same episode, following UMDF. It is ablated rather than assumed.

Analyses. Following M3's finding that upweighting the dominant modality's drop rate helps, we sweep the image-drop probability; with strong backbones, image matching is likely the dominant prompt. We then record attention from text-only keypoint tokens to image-anchored neighbors. This tests whether cross-keypoint propagation actually carries the gain.

Baselines

Naive reference. The same architecture trained only on complete prompts (image, text, and skeleton for every keypoint). At test time, missing prompts are filled with the absent vectors. This isolates whether dropout training matters at all.

Strongest existing single-prompt methods. These are retrained on our backbone and splits. For text, CapeX reports 88.81% [email protected] one-shot on MP-100 (CapeX). For image plus skeleton, GraphCape reports 87.23% one-shot and 91.16% five-shot (Pose Anything). SCAPE reports 91.95 PCK one-shot with a DINOv2 ViT-B backbone (SCAPE). CapeLLM reports 92.60% one-shot under its support-query-pair protocol (CapeLLM); that protocol and model scale differ from ours, so we cite it as context rather than as a head-to-head comparison.

These are the strongest competitors because each is the best published model for its prompt type. Our claim fails if combining prompts cannot beat them. For fairness, we rerun CapeX and GraphCape with the same backbone (SwinV2, and separately DINOv2 ViT-B), the same five MP-100 splits, identical epochs, and the same hyperparameter search budget as our model. We will count it as beating them if our single model is within 0.5 [email protected] of each specialist in that specialist's own regime and strictly better in the image-plus-text regime.

Per-keypoint stitching. This is the strongest competitor for the mixed regime. We run retrained GraphCape on the image-prompted keypoints and CapeX on the text-prompted keypoints, then merge the outputs; where a keypoint has both prompts, we average the two predictions. For the mixed regime, no reported result was found on Lacuna, because no prior work evaluates it. Beating stitching means higher mean [email protected] across mask fractions, with a margin larger than the across-split standard deviation.

Ablations. Whole-sample modality dropout in the style of M3 (same expected drop rate, but no per-keypoint mixing) isolates the key ingredient. EdgeCape-style within-modality keypoint masking checks whether masking inside one modality already suffices. Further ablations remove skeleton dropout and remove the consistency term.

Why not just…?

Why not just stitch together specialist models per keypoint? This is the real threat. If keypoints were localized independently, stitching would be optimal, and our model would add only complexity. The evidence that they are not independent comes from SCAPE's refiner, which infers occluded points from visible ones, and from GraphCape's gains from skeleton message passing. Both suggest that anchored points should help unanchored ones. We will not argue the point; the mixed-regime curve settles it. If stitching matches us, especially on text-only keypoints with image-anchored neighbors, the central claim is false.

Why not just train on complete prompts and fill in blanks at test time? Fusion layers trained only on complete inputs see unfamiliar inputs when a prompt is missing. M3 shows that models trained this way rely on the dominant stream. The naive reference baseline measures exactly this gap.

Why not just use feature dropout or EdgeCape's keypoint masking? M3 found that partial masking leaves enough of the dominant modality for the model to ignore the others. EdgeCape's masking stays within the visual prompt and never trains the model to work without the image or the skeleton. Both appear in our ablations.

Why not just convert everything to text, as CapeLLM does? CapeLLM needs detailed, spatially explicit descriptions and runs an autoregressive large language model. SCAPE, by contrast, runs near 30 FPS on an RTX 3090. The audio-visual LLM study also shows that language models can fall back on internal priors when inputs conflict. We still treat CapeLLM's 92.60% as a ceiling to discuss honestly. Our contribution is prompt flexibility at feed-forward cost, not raw one-shot accuracy under fully detailed text.

Proposed Experiments

All experiments use MP-100, one-shot and five-shot, over its five category-disjoint splits. We report [email protected] and [email protected], with AP-10K as a secondary cross-species test using EdgeCape's protocol.

Mixed-regime curve (decisive). For each test episode we vary the fraction of keypoints that have an image prompt from 0 to 1, in steps of 0.25; the rest are text-only. We plot PCK for our model, stitching, and whole-sample dropout.

Symmetry test. We measure the rate at which left/right keypoint pairs are swapped, in each prompt regime. The prediction is that image-plus-text beats either prompt alone, because the image fixes the frame and the text names the side.

Skeleton removal. We measure the PCK drop when the skeleton is withheld at test time, with and without skeleton dropout in training.

Conflict probe. Motivated by the audio-visual LLM result, we pair a support image with a deliberately swapped left/right text description. We then measure which prompt the model follows, to see whether dropout produced real evidence combination or merely a bias shift.

The proposal is falsified if either of the following holds:

  • the mixed-regime advantage over stitching is within split noise;
  • the image-plus-text regime fails to beat the better single-prompt specialist.