Open-World Task Masks for Instructional Video: Letting the Whole Constrain the Parts When the Whole Is New

Systems that recognize the steps in an instructional video ("loosen the lug nuts", "pour the batter") work much better when they first guess the overall task ("changing a tire") and then suppress step predictions that make no sense for that task. The trouble is that this top-down mask is built from a fixed list of tasks and the steps each one allows. When a video shows a task the system has never seen, it is forced into the nearest known task. Its genuine steps are then masked out as "out of context." The more useful the filter is on familiar data, the more damage it does on unfamiliar data.

We propose to replace the fixed task-to-step mask with a growing one, using a Dirichlet process. A Dirichlet process is a prior over clusterings that leaves room for an unbounded number of clusters and opens a new one when no existing cluster explains the data. In our model, each task owns a set of step clusters drawn from a pool that all tasks share. A test video can join a known task, or it can open a new task that reuses known steps and adds new ones. Using a step outside a task's current set is penalized rather than forbidden. The new claim is about what happens over time. After the first few videos of a novel task, the model has built a mask for that task, so later videos of the same task benefit from top-down filtering they could never get before. Simply switching the filter off when the model is unsure cannot produce this effect.

Core Claim

Hierarchical filtering and nonparametric clustering each fix the other's main weakness. Task-consistency filtering has three known problems: it is closed-set, it is one-directional, and its errors propagate downward. Dirichlet-process clustering has no notion of whole-to-part structure and tends to spawn many redundant clusters. If the task-to-step mask becomes the cluster-usage structure of a hierarchical Dirichlet process (HDP), solved quickly with small-variance asymptotics, three things should follow. Known-task step localization should stay about as good as with a hard mask. Novel tasks and steps should be discovered as structured clusters rather than lumped into one "unknown" bucket. Filtering quality on a novel task should improve as more of its videos arrive.

How It Relates to Prior Work

The filtering side supplies the motivation and the target. COIN (Tang et al., 2019) and its extended version introduced the Task-Consistency module. It pools proposal scores across a whole video to infer the task, then suppresses steps that are incompatible with it. This raised step localization from 8.12% to 9.05% [email protected], and adding ordering dependencies raised RGB [email protected] from 19.39% to 23.23%. Two related works use the same top-down principle. Wang et al. force segment labels to lie inside a video-level candidate set on YouTube-8M. Xu and Corso add a video-level soft potential to a pixel-level conditional random field on A2D. All three assume a fixed vocabulary of tasks and steps. We keep their mechanism but let the vocabulary grow.

The nonparametric side supplies the method. Cao et al. use an HDP for open-set recognition: known classes and the test batch are groups that share mixture components, so new subclasses appear without hand-set rejection thresholds. We borrow the idea of groups sharing components. Their groups are flat, though, and inference uses Gibbs sampling, a slow repeated-resampling procedure. They also need an ad hoc pruning threshold, as does Ghourchian et al. for HDP mobility topics. Wang and Zhu supply the inference tool. They take the zero-noise limit of a Dirichlet-process subspace model and obtain DP-space, a deterministic penalized objective. It matches sparse subspace clustering on Hopkins 155 at roughly 3.3% error while taking about 2 seconds instead of more than 500. We derive the analogous limit for a task-structured HDP.

OpenTAL constrains our evaluation. It detects unknown actions in untrimmed video (78.33% AUROC on THUMOS14), but it uses a fixed-size Dirichlet over class probabilities and treats unknowns as a single reject class with no task context. Our contrast with it is structure: discovered steps are grouped into clusters and attached to tasks. CN-DPM and NOMAD show that Dirichlet-process expansion works in streams. NOMAD's authors note that it struggles where semantic cues are weak, and neither system has a whole-to-part structure. That missing structure is what we add.

Research Program

Segment features come from a frozen proposal or frame backbone of the kind used in the COIN baselines. A small embedding head is trained on known steps so that each known step has a prototype and classification is by distance to prototypes. Each known task $t$ starts with a step set $S_t$ read from the training annotations.

At test time, videos arrive as a stream. For each video $v$, the model jointly chooses a task assignment $c_v$ and a step assignment $z_s$ for each segment $s$, by minimizing the following objective:

$$ \sum_{s} \lVert x_s - \mu_{z_s} \rVert^2 \;+\; \lambda_{\text{step}} K \;+\; \lambda_{\text{task}} T \;+\; \lambda_{\text{use}} \sum_{t} \lvert S_t \rvert $$

Here $x_s$ is a segment embedding and $\mu_k$ is the prototype of step cluster $k$. $K$ is the number of step clusters and $T$ is the number of tasks. $S_t$ is the set of steps task $t$ uses, which grows when a video assigned to $t$ needs a step not yet in it. The objective is a small-variance limit of an HDP in the style of DP-space, with each penalty $\lambda$ coming from a concentration parameter.

The $\lambda_{\text{use}}$ term is the top-down filter, made soft. A segment pays an extra cost for any step outside its task's current set. Setting $\lambda_{\text{use}}$ very large recovers the hard Task-Consistency mask. Setting it to zero removes filtering. Opening a new task costs $\lambda_{\text{task}}$ and starts with an empty set, so a novel video is never forced into a wrong mask. It pays its way by building a new step set instead.

The usage penalty also works against cluster proliferation. A redundant new step cluster has to be "bought" for every task that uses it, so near-duplicate clusters are discouraged without a separate pruning pass. Inference alternates hard assignments and prototype updates, like k-means, once per arriving video. We will set the three penalties on a validation split of known tasks with simulated holdouts, never on the novel test tasks.

We will also analyze when the new-task option fires and how sensitive results are to the three penalties. This addresses the known sensitivity of Dirichlet-process models to their concentration parameter. Boundary errors, the HASR limitation identified by Ahn and Lee, are out of scope. We take proposal boundaries as given.

Baselines

The naive reference uses the same backbone and prototype classifier with no top-down filter, and unknown segments are simply assigned to the nearest known step. This tells us how much filtering helps at all on this split.

The strongest existing method is the Task-Consistency module of Tang et al., reimplemented on our backbone. On the standard closed-set COIN protocol it reports 9.05% [email protected] with SSN-Fusion. The extended paper reports 23.23% RGB [email protected] with ordering plus task consistency. For any open-set COIN split, we found no reported result on Lacuna. We will therefore first reproduce the closed-set setting within a point of those figures, then run every method on the identical held-out-task split. Beating Task-Consistency means two things: known-task [email protected] within about half a point of it, and clearly higher mAP on novel-task videos.

We include two further strong baselines. The first is a soft, confidence-gated version of Task-Consistency: it marginalizes the mask over the task posterior and disables the mask entirely when the maximum task probability falls below a validation-tuned threshold. The second is an OpenTAL-style pipeline. It applies OpenTAL evidential rejection followed by flat DP-means clustering of the rejected segments. This is our reading of Cao et al. without the task layer. Since OpenTAL's 78.33% AUROC is on THUMOS14, we will not transfer that number, only the method.

Every method gets the same frozen features, the same proposal set, and the same number of validation trials for its thresholds or penalties.

Three ablations isolate the key ingredient. The first removes the new-task option, so that $\lambda_{\text{task}}$ is effectively infinite. The second sets $\lambda_{\text{use}}=0$, which gives an HDP with no top-down filter. The third freezes each $S_t$ after training, so that step sets cannot grow.

Why not just…?

Why not just soften the mask by averaging it over the task posterior? Averaging fixes errors between known tasks. If a video sits between two known tasks, both masks contribute. It cannot help a genuinely novel task, though, because every mixture component still suppresses the novel steps. The held-out-task split separates the two cases directly, and the ablation without a new-task option is exactly this baseline.

Why not just turn the filter off when the task classifier is unsure? This is the real threat. On the first video of a novel task, gating and our method should behave about the same. The difference appears afterward. Gating stays unfiltered for every later video of that task, whereas our method has built a step set for it. If novel-task mAP for gating matches ours across the stream, then the hierarchical nonparametric machinery is not worth its cost, and we will report that.

Why not just reject unknowns and cluster them afterward? Flat post-hoc clustering ignores the fact that novel tasks reuse known steps. A new recipe still involves "stir," for example. It also has no way to filter later videos. COIN's taxonomy shares steps across tasks, so if sharing matters, it should show up as higher clustering agreement on novel steps and fewer redundant clusters for our method. If it does not, the task layer is not earning its place.

Proposed Experiments

On COIN (180 tasks, 778 steps), we hold out roughly 15% of tasks entirely, which also removes the steps unique to them. We stream test videos in random order, with known and novel tasks interleaved. On Breakfast (10 tasks, 48 actions), we run leave-one-task-out, with segmental edit score reported alongside. HASR's 70.8% edit score on MS-TCN gives the closed-set reference point.

The metrics are as follows. For known-task steps we report step [email protected] and @0.1. For novel videos we report step mAP after optimal matching between discovered and true steps. We report novel-step detection AUROC, and the adjusted Rand index between discovered and true novel steps and tasks. We also report the number of redundant clusters and wall-clock time per video, comparing our deterministic solver against Gibbs-sampled HDP inference.

The decisive check is the stream curve. We plot novel-task step mAP against how many prior videos of that task have been seen. Our method should rise and confidence-gated filtering should stay flat. Known-task mAP should stay close to hard Task-Consistency. A flat curve, or a large loss on known tasks, would falsify the proposal.