When a video model learns to localize actions from video-level labels alone, it tends to find only the single most telling moment. For a high jump, that moment is the athlete clearing the bar. It then reports a time interval far shorter than the real action and drops the approach run and the landing. This failure is called discriminative bias. We argue that a large part of it comes from a specific, fixable flaw in how these models are trained. The approach run looks like the run-up in long-jump videos, and the model is told that long-jump videos contain no high jump. So training actively teaches the model that running is not high jump. Phases shared across classes get pushed into "background," even though they are part of the action.
We propose to represent video snippets in hyperbolic space, a geometry in which volume grows exponentially with distance from the origin, so tree-shaped hierarchies fit with little distortion. Each action class becomes a cone nested inside the cone of a learned, more general parent (for example "run-up"). Training applies negative evidence only to a class's own cone, never to its parents. At test time an action's extent is the contiguous stretch of snippets that fall inside that class's cone or its ancestors' cones, anchored on a confident peak. It is not the stretch where a class score exceeds a threshold. In two sentences: shared phases are treated as general ancestors of an action rather than as its negatives, and localization asks "is this snippet contained by the action or its parent?" rather than "is this snippet distinctive of the action?"
Core Claim
Our claim is that peak collapse in weakly supervised temporal action localization (WTAL, where only video-level labels are available) is driven substantially by penalizing shared sub-action phases through negative videos. We further claim that hyperbolic entailment cones are an appropriate structure for this problem, because they can learn which phases are shared without a given hierarchy. Two parts are new: the asymmetric training objective that applies negatives only at the leaf level, and decoding boundaries by cone containment.
The claim is falsifiable at two levels, and we design the experiments to separate them. The loss asymmetry might explain most of the gain on its own, even in flat Euclidean space. Or the geometry might be needed so that a latent hierarchy can form without labels. If the first holds, the hyperbolic part is decoration, and the proposal says so.
Relation to Prior Work
The motivating evidence comes from WTAL. Completeness Modeling and Context Separation describes the "smoking gun" behavior. It counters it with parallel branches forced to activate on different snippets, plus hard negatives mined from low-motion snippets. EM-MIL replaces soft attention with thresholded pseudo-labels, but its authors report that the expansion heuristic overestimates boundaries into adjacent background. That result constrains us: any rule that widens predicted extents must be anchored, or it will bleed.
AUMN is the closest competitor in spirit. It builds actions from shared "action units" stored as flat learnable templates, using the same running-shared-by-high-jump-and-long-jump example. We differ in two ways. Our shared parts are nested containers with a direction, not flat templates matched by attention. And the negative-class loss is explicitly kept off the shared parts. FTCL compares temporal structure across videos with differentiable sequence alignment, and SSN adds a separate completeness classifier over start, course, and end phases, which needs boundary supervision. Both model action structure in Euclidean space without any notion of which phases are shared across classes.
The method comes from hyperbolic learning. Hyperbolic Entailment Cones defines cones in the Poincaré ball whose width shrinks with distance from the origin, which encodes directed "is-a" relations. It reports hypernymy F1 of 80.1% in 5 dimensions against roughly 70% for the best baselines on WordNet. We reuse this cone and its violation energy directly. Learning the Predictability of the Future shows that video embeddings trained contrastively in the Poincaré ball move uncertain predictions toward the origin, which corresponds to more general categories. This supports our expectation that generic phases will settle nearer the origin without being told to. PE-HCM learns fine-grained structure from coarse labels in hyperbolic space, but only on static images.
The work closest to ours in setting is HLFormer. It uses Lorentz-model attention and a cone-based loss requiring a text query to be contained by its untrimmed video, for partially relevant video retrieval. By its own design it uses no temporal boundaries and produces none. We move cones from a cross-modal text-to-video relation to a within-video relation between snippets and actions, and use them to produce boundaries.
Method
Snippet features from a frozen I3D backbone (the features used by Completeness Modeling and Context Separation) pass through a small temporal encoder. They are then mapped into the Lorentz model of hyperbolic space for numerical stability, following HLFormer, and converted to Poincaré coordinates only to compute cones. Each class $c$ has a learned apex point $a_c$, and there are $K$ learned ancestor points. Each class is softly assigned a parent $\pi(c)$ among them.
Let $E(x, y)$ be the cone-violation energy of Hyperbolic Entailment Cones. It is zero when point $x$ lies inside the cone rooted at $y$, and grows with the angular excess otherwise. With snippet embeddings $z_t$ and the label set $Y$ of a video, the training loss is
$$\mathcal{L} = \sum_{c \in Y} \frac{1}{k}\sum_{t \in \mathrm{top}_k(c)} E(z_t, a_c) \;+\; \sum_{c \notin Y} \max\!\big(0,\, m - \min_t E(z_t, a_c)\big) \;+\; \lambda \sum_c E(a_c, a_{\pi(c)}).$$
Here $\mathrm{top}_k(c)$ are the $k$ snippets least in violation of class $c$'s cone, $m$ is a margin, and $\lambda$ weights the term requiring each class cone to sit inside its parent's cone.
The key point is what the second term leaves out. It pushes all snippets of negative videos out of the leaf cone $a_c$, but never out of the parent cone $a_{\pi(c)}$. A run-up snippet in a long-jump video may therefore sit inside the shared "run-up" cone without being penalized as not-high-jump. A background cone, trained on the lowest-scoring snippets of each video, gives the model a place to put scene context.
At test time we find peaks inside $a_c$'s cone. From each peak we grow the interval in both directions for as long as snippets stay inside the cone of $a_c$ or of $a_{\pi(c)}$, and outside the background cone. Growth stops at the first snippet that fails. This rule is anchored on a peak and requires contiguity, which is our direct response to the overestimation reported for EM-MIL.
Research Program
The work has three stages.
The first is a controlled study on FineGym, described in FineGym: A Hierarchical Video Dataset for Fine-Grained Action Understanding. Its three-level label hierarchy (event, set, element) and two-level temporal annotations let us train with element labels at the routine level only, then check two things: whether the learned ancestors recover set-level groupings, and whether element boundaries improve. Because the true hierarchy is known here, we can compare the latent hierarchy against a Euclidean model given that hierarchy directly.
The second stage moves to THUMOS'14 (THUMOS challenge) and ActivityNet v1.2 and v1.3, where no hierarchy is given and the latent ancestors must be discovered.
The third is diagnostic. We break localization errors into three kinds: incompleteness (predicted duration shorter than the true duration), fragmentation (more than one prediction per true instance), and bleed (predicted snippets outside the true interval). We add the duration-distribution Jensen-Shannon distance proposed by HVQ. The proposal should reduce incompleteness and fragmentation without raising bleed, and the diagnostics show whether that tradeoff actually happened.
Baselines
The naive reference is a standard single-branch multi-instance learning model on the same I3D features, with a class activation sequence (a per-snippet class score over time) and thresholded decoding. It shows how much of any gain is just the encoder.
The strongest existing method is AUMN. It reports THUMOS'14 average mAP of 52.1% and ActivityNet average mAP of 25.5% (v1.2) and 23.5% (v1.3). It is the strongest competitor because it targets the same shared-sub-action intuition with flat memory templates. FTCL, with 43.6% on THUMOS'14 and 24.8% on ActivityNet v1.3, and Completeness Modeling and Context Separation, with 40.9% on THUMOS'14, represent the alignment and diversity families. MLLM4WTAL reports only relative gains (+1.8 average mAP over DELU on THUMOS'14). We found no absolute figure for it on Lacuna, so it enters as an add-on comparison.
The threshold ranges used for "average mAP" may differ across these papers. We will therefore rerun AUMN, FTCL, and the diversity model from public code on identical features and splits, with the same encoder size and the same hyperparameter search budget. We report both our reruns and the published figures. For FineGym weakly supervised element localization we found no reported result on Lacuna, so every FineGym number will come from our own reruns.
Two ablations isolate the key ingredient. The first, a Euclidean asymmetric model, keeps our two-level loss with negatives applied only to leaves and uses the same containment decoding. It replaces the hyperbolic cones with Euclidean order-embedding containment, the family Hyperbolic Entailment Cones outperformed on WordNet. It is given up to four times the embedding dimension. The second, a hyperbolic symmetric model, keeps the geometry but applies negatives to parent cones as well. The first ablation tests whether geometry matters, and the second tests whether the asymmetry matters.
Beating AUMN means a higher average mAP on THUMOS'14 under the shared rerun protocol. The margin must exceed seed variance over five seeds, and the incompleteness and fragmentation diagnostics must improve without bleed getting worse.
Why not just…?
Why not just scale up the Euclidean embedding? Learning the Predictability of the Future found on FineGym that a 64-dimensional hyperbolic embedding beat a 256-dimensional Euclidean one: 13.37% against 10.08% in prediction accuracy, and 66.64% against 52.00% in hierarchical accuracy. That was a different task, so we test the question directly. The Euclidean asymmetric ablation is swept over dimensions 64 to 512. If the widest Euclidean variant matches the hyperbolic model, the geometry claim fails.
Why not just use a Euclidean hierarchical classifier with leaf-only negatives? This is the most serious threat. On FineGym, where the hierarchy is given, a two-level Euclidean multi-instance model with set-level and element-level heads, negatives only on elements, and decoding by growing element peaks through set-level activations may do as well as our method. We will run it with the ground-truth hierarchy as an upper bound for the "loss alone" explanation. The hyperbolic model's remaining case is that it reaches this bound without the hierarchy on FineGym, and that on THUMOS and ActivityNet, where no hierarchy exists, it beats the Euclidean asymmetric ablation that must learn its parents. If it does neither, the contribution reduces to the loss asymmetry, and we will report it that way.
Why not just erase peaks or force branch diversity? Hide-and-seek erasing (by Krishna Kumar Singh) and the orthogonal-branch loss of Completeness Modeling and Context Separation make the model look beyond the peak, but they keep the negative-class penalty on shared phases. Our prediction is therefore specific: their gains should concentrate on classes without shared phases, while ours concentrate on classes whose non-peak phases also appear in other classes. We test this by splitting THUMOS'14 classes according to whether their learned ancestor is shared.
Why not just distill from a multimodal language model? MLLM4WTAL adds textual phase descriptions and gains 1.8 points over DELU. It is complementary rather than competing, since its guidance could be attached to our encoder. We will report our model with and without it, so that any gain from geometry is not confused with gain from language priors.
Proposed Experiments and Decisive Checks
The main table reports mAP at tIoU thresholds 0.1 to 0.7 on THUMOS'14 and 0.5 to 0.95 on ActivityNet v1.2 and v1.3. It covers the naive model, AUMN, FTCL, the diversity model, both ablations, and our model, all from the shared rerun protocol. On FineGym we report element-boundary mAP under routine-level weak labels, together with the purity of learned ancestors with respect to set labels.
Three results decide the question. First, the hyperbolic model must beat the Euclidean asymmetric ablation at matched and at quadrupled dimension on THUMOS'14; otherwise the geometry is not needed. Second, the hyperbolic symmetric ablation must fall back toward the naive model on incompleteness; otherwise the asymmetry is not the mechanism. Third, the snippets our model recovers beyond the Euclidean peaks must lie closer to the origin, and inside a parent cone shared by at least two classes, much more often than chance; otherwise the explanation that shared phases are general ancestors is wrong even if mAP rises.
A known risk is static context such as pool tables and diving blocks. Nothing in the cone structure guarantees that context falls in the background cone rather than a parent cone, so we report bleed separately on the context-heavy THUMOS classes identified by Completeness Modeling and Context Separation.
References
- Completeness Modeling and Context Separation for Weakly Supervised Temporal Action Localization
- Weakly-Supervised Action Localization with Expectation-Maximization Multi-Instance Learning
- Action Unit Memory Network for Weakly Supervised Temporal Action Localization
- Fine-grained Temporal Contrastive Learning for Weakly-supervised Temporal Action Localization
- Temporal Action Detection with Structured Segment Networks
- Hyperbolic Entailment Cones for Learning Hierarchical Embeddings
- Learning the Predictability of the Future
- Hyperbolic Space with Hierarchical Margin Boosts Fine-Grained Learning from Coarse Labels
- Enhancing Partially Relevant Video Retrieval with Hyperbolic Learning
- FineGym: A Hierarchical Video Dataset for Fine-Grained Action Understanding
- The THUMOS Challenge on Action Recognition for Videos "in the Wild"
- Hierarchical Vector Quantization for Unsupervised Action Segmentation
- Weakly Supervised Temporal Action Localization via Dual-Prior Collaborative Learning Guided by Multimodal Large Language Models