Remembering Which Leg Is Which: Recurrent Part Memories for Unsupervised Part Discovery in Video

Methods that find object parts without labels, such as the beak, wings and legs of a bird, usually run several attention "heads" in parallel over a single image. Each head outputs a soft mask saying where it is looking. These methods work well on still photos. But they break exactly where video gets hard. When a horse gallops, its legs cross. When a tiger sits still, its limbs stop moving. A system that looks at each frame on its own cannot tell which leg was which a moment ago. We propose to keep parallel attention as the part finder, re-run from scratch on every frame, and to add a small recurrent network per part, a network that carries a running memory from frame to frame. This memory decides only which discovered mask belongs to which part identity. The memory never draws the masks itself, so it cannot slowly drift and collapse the parts together. It only resolves ambiguity when appearance and motion in the current frame are not enough.

The central bet is narrow and testable. We claim recurrence earns its place in part discovery by preventing identity swaps at crossings and during stillness. We do not expect it to improve average mask quality on ordinary frames. A feed-forward model trained with motion cues may match it on average. It should fail when two legs overlap and then separate. The experiments are built around measuring exactly that event.

Where the directions stand

Parallel spatial attention supplies the part finder. Part-aligned person re-identification uses K parallel 1×1-convolution mask heads over a shared feature map, trained without part labels (Deeply-Learned Part-Aligned Representations for Person Re-identification). Fine-grained recognition does something similar with parallel excitation branches (Multi-Attention Multi-Class Constraint for Fine-grained Image Recognition). Both papers show a real constraint on our design: parallel heads collapse onto the same dominant region unless something pushes them apart. The attention-based ensemble paper shows this directly by ablating its divergence loss (Attention-Based Ensemble for Deep Metric Learning). We reuse that kind of per-frame divergence penalty.

The best static part discoverer in the file uses three training signals: contrastive matching across instances, a "visual consistency" loss that asks each part to have uniform color, and equivariance under warps (Unsupervised Part Discovery from Contrastive Reconstruction). It reports NMI 46.0 on CUB-200-2011. Its authors name the uniform-color assumption as a failure mode on patterned animals. That is our motivation for replacing it with a motion cue.

Four papers come closest at the intersection, and each leaves the target open.

Recurrent Spatial Transformer Networks drives one glimpse at a time with a recurrent network, specifically a GRU. It does so on cluttered MNIST digits, with 1.5% error versus 2.0% for the feed-forward version. It is sequential, uses one crop per step, and works on toy data. It never decomposes an object into parts.

RVOS tracks whole objects in video with convolutional LSTMs. It is trained with ground-truth masks and finds instances one after another rather than in parallel.

PARTS is the most important neighbor. It is an unsupervised video model that keeps one LSTM state per slot, where a slot is a latent vector meant to capture one entity. It works on synthetic scenes, reaching ARI-F 0.7349 on 3D Playroom. It reports that letting the previous frame's latent act as the prior makes slots drift and merge. It therefore re-discovers entities from a fixed prior every frame, and it concedes this costs temporal smoothness. Our design takes this negative result as a constraint. We keep PARTS-style per-frame re-discovery for the masks and confine recurrence to identity assignment. We also target articulated sub-parts of real animals rather than whole synthetic objects.

The closest part-level video work is Discovering the Physical Parts of an Articulated Object Class from Multiple Videos. It groups superpixels by shared motion across many videos of tigers and horses in a conditional random field, reaching part IoU of 0.33 to 0.35 on tigers. It is not a learned deep model. The authors report that it swaps labels between crossing legs during galloping. That failure is the one our method is designed to fix.

A related static-image line binds objects through recurrent phase synchronization, SynCx. There, recurrence means iterative refinement within one image, not memory across time, so it is a methodological cousin rather than a competitor.

Core Claim

Each frame produces a new, unordered set of K part masks from parallel attention, with no temporal prior. K persistent recurrent part memories then claim those masks through a soft one-to-one matching. We claim this combination avoids the drift that PARTS observed while keeping part identity stable across leg crossings and still periods. Feed-forward part discovery cannot do this, even when trained with motion cues and linked by overlap. Autoregressive recurrent slot models cannot do it either.

Research Program

Per-frame part finder. A shared backbone processes each frame. K parallel mask heads plus one background head produce soft masks through a softmax across heads. We add the divergence penalty from the attention-based ensemble paper and the contrastive and equivariance losses from the contrastive-reconstruction paper. The masks carry no identity by themselves.

Motion training signal. Instead of asking each part to have uniform color, we ask each part to move as one rigid-ish piece, a "common fate" cue. This is a learned analogue of Del Pero et al.'s motion distance. The loss is

$$ L_{\text{fate}}=\sum_{k}\min_{A_k}\sum_{p} m_k(p)\,\big\| f(p) - A_k\,[p;1]\big\|^2 . $$

Here $m_k(p)$ is mask $k$'s weight at pixel $p$, $f(p)$ is the optical flow at $p$ from an off-the-shelf estimator, and $A_k$ is a 2×3 affine motion fitted in closed form by weighted least squares. The flow is used only at training time.

Recurrent part memories. Each of K GRU memories $h_k$ reads the pooled feature, mask centroid and mask shape of the part it was matched to at the previous step. From these it forecasts that part's next location, shape and appearance. At frame $t$, we score every memory against every fresh mask by forecast agreement. A Sinkhorn step, a differentiable way to turn the score table into an approximately one-to-one assignment, then assigns masks to memories. Because the memory only chooses among masks that were found bottom-up, it cannot pull a mask onto the wrong region.

Temporal consistency losses. A forecast loss makes each memory's predicted mask match the mask it is assigned at $t+1$. A forward-backward cycle loss makes the assignment computed forward in time agree with the one computed backward. These losses supervise identity only.

Cross-video training. Training runs over many unlabeled clips of one animal class. This way, memory $k$ comes to mean the same anatomical part in every video, which is the cross-video linking Del Pero et al. showed matters. K is fixed at the 11-part budget of the TigDog annotation. The file names fixed part count as an open limitation, and we do not solve it. We only report sensitivity to K.

Baselines

Naive reference points. The first is rigid spatial bins over the object box, applied frame by frame. The part-aligned re-identification paper already shows fixed stripes lose to learned masks under pose change. The second is single-video motion segmentation, which Del Pero et al. report at roughly 0.18 part IoU on tigers. Beating these is necessary but uninformative.

Strongest existing method on the exact task. This is Del Pero et al.'s cross-video CRF, reported at 0.33 to 0.35 average part IoU on the 32-video TigDog/YouTube-Objects subset with 11 parts. We evaluate on the same videos and the same 11 labels and quote their reported number. If a public implementation is not available, we reimplement it only for the swap-rate analysis, since they report no swap metric. Beating it means exceeding 0.35 IoU and showing a lower swap rate. IoU alone could improve simply from deep features.

Strongest neural competitor. This is the contrastive-reconstruction part discoverer run per frame, with masks linked across frames by Hungarian matching on mask overlap. The file contains no reported result for it on TigDog, so we will train it on the same unlabeled clips, with the same backbone, same K and same hyperparameter search budget.

Ablation of motion as a training signal alone. The same per-frame finder is trained with $L_{\text{fate}}$ but without memories, then linked by overlap. This ablation isolates whether recurrence adds anything beyond using motion during training.

Autoregressive variant. Here the memories themselves generate the masks, which is the design PARTS warns against.

PARTS-style fixed-prior slot model. This is a slot model with slot-wise LSTMs retrained on the TigDog clips. The file has no reported result for it on TigDog.

Fairness rules. All neural methods share the backbone, input resolution, training clips, number of epochs and number of tuning trials.

Why not just…?

Why not just run a static part discoverer per frame and link masks by overlap? This is the most natural competitor. It should do well on frames where parts are well separated. It has no way to decide which of two overlapping legs is which once they separate: overlap is identical for both assignments at the crossing. Del Pero et al. document exactly this swap failure for a per-frame-plus-motion approach. The per-frame baseline, measured on annotated crossing events, settles the question directly.

Why not just train a feed-forward model with the motion loss and skip recurrence? This is a genuine threat. The motion cue may deliver most of the gain in average IoU. That possibility is why we do not claim average IoU as the payoff of recurrence. The claim is confined to swap rate at crossings and part persistence through still segments. If the feed-forward-plus-motion ablation matches the full model on those two measures, recurrence adds nothing here and we will say so.

Why not just cluster features from a pretrained network? The contrastive-reconstruction paper reports that clustering pretrained convolutional features gives blurry, blob-like parts. It improves CUB NMI only by adding dedicated losses: 46.0 versus 39.1 for SCOPS. The file has no evidence on modern self-supervised transformer features. We therefore include K-means over frozen backbone features plus overlap linking as a cheap reference. It inherits the same identity problem as any per-frame method.

Why not just use PARTS? PARTS re-discovers slots every frame with no identity-carrying memory beyond its LSTM dynamics. It was shown on whole synthetic objects. Its own authors report the smoothness cost of re-discovery. The retrained PARTS baseline tests whether its slots reach part granularity on real animals and whether they keep limb identity.

Proposed Experiments

Primary benchmark. The main evaluation uses the TigDog/YouTube-Objects tiger and horse subset with 11 annotated parts. We report mean part IoU against the 0.33 to 0.35 reported by Del Pero et al. The contrastive-reconstruction paper warns that a single metric can be gamed, so we add NMI and ARI over the part segmentation.

Decisive measurement. We will mark crossing events in the evaluation videos from the existing part labels: frames where two leg masks overlap and then separate. The swap rate is the fraction of such events after which a predicted part identity lands on the other leg. A second measure is persistence on still segments, where flow is near zero: the IoU of each part in those segments relative to its IoU in moving segments.

Predictions that would falsify the claim. The central claim fails if either of two things happens. The first is that the full model's swap rate is not clearly below that of the feed-forward-plus-motion ablation. The second is that the autoregressive variant matches it without drifting over long clips. We track drift as mask overlap between memories as a function of clip length.

Secondary checks. We test static transfer by applying the trained per-frame finder to PASCAL-Part horse images. No reported result was found on Lacuna for unsupervised part discovery on PASCAL-Part, so we compare against the contrastive-reconstruction model trained on the same data. We also sweep K to expose the fixed-cardinality limitation honestly.