GAIN: On the Generalization of Instructional Action Understanding

Junlong Li, Guangyi Chen, Yansong Tang, Jinan Bao, Kun Zhang, Jie Zhou, Jiwen LuICLR 2023 · OpenReview

Introduction

Deep learning has driven significant progress in Instructional Action Understanding (IAU)—the task of identifying and localizing steps in "how-to" videos (e.g., "first, crack the egg; then, whisk it"). However, while current models excel on their training distributions, they struggle to generalize to unseen, out-of-distribution (OOD) tasks, even when those new tasks share fundamental steps with the training data.

The core issue is contextual bias. Models often use the "temporal flow" of a specific task as a cheat sheet. If a model always sees "inflating a tire" in the context of a bicycle, it might fail to recognize the same action in a car-repair video. It has "short-cut" its learning to rely on the surrounding context (like the presence of a bicycle or the preceding steps) rather than the visual features of the action itself.

To address this, researchers introduce GAIN, a new benchmark designed to evaluate how well instructional video models generalize to unseen tasks, alongside a novel causal inference approach to mitigate this bias.

The GAIN Benchmark

To measure this generalization gap, the authors created the GAIN dataset by reassembling action steps from existing datasets into entirely new task categories.

For example, if the training set contains "Make Chocolate" (which includes a "put into molds" step) and "Make Pizza" (which includes a "bake" step), the authors construct a new OOD task like "Make Jelly" or "Roast Chicken" using those pre-existing step categories.

Illustration of how the GAIN benchmark reassembles action steps from existing training sets into entirely new out-of-distribution tasks.

The resulting benchmark contains 1,231 videos across 116 OOD tasks and 230 step categories. It is split into GAIN-C (derived from the COIN dataset) and GAIN-B (derived from the Breakfast dataset).

Mitigating Bias with Causal Inference

To reduce the model's dependency on biased temporal contexts, the authors apply the Back-Door Criterion from causal inference.

They treat the surrounding video context as a confounder that creates spurious correlations between a specific step and the model's prediction. By performing a "causal intervention," they mathematically cut the link between the step and its specific training context.

Since calculating the true causal effect is computationally difficult, they use a Monte Carlo-based "step pool." They disassemble training videos into individual steps and reassemble them into randomized, "intervened" sequences. This acts as a form of data augmentation, forcing the model to learn the appearance of a step regardless of what actions precede or follow it.

Key Findings and Results

  • The Generalization Gap is Real: Models that achieve high accuracy on standard benchmarks see a massive performance drop when tested on GAIN. On the GAIN-B split, baseline accuracy dropped by approximately 60%, proving that models over-rely on contextual crutches.
  • Improved Action Segmentation: The proposed causal approach consistently improved performance across four major baselines (LSTM, ED-TCN, TResNet, and MS-TCN++). For instance, Causal MS-TCN++ improved frame accuracy by +8.1% on GAIN-C compared to the standard version.
  • Improved Action Detection: In the task of localizing steps (mAP@mid), the causal method improved the LSTM baseline from 6.0 to 8.0 on GAIN-C and from 8.5 to 13.2 on GAIN-B—relative improvements of over 33% and 55%, respectively.
  • Zero Performance Cost: Importantly, the causal intervention improved OOD generalization without sacrificing performance on the original in-distribution test sets.

Limitations and Future Work

Constructing the GAIN dataset required significant manual effort to filter logical task candidates and annotate fine-grained temporal boundaries. Furthermore, while GAIN is a significant step forward, it cannot cover every possible real-world scenario.

While causal inference shows great promise for breaking spurious temporal correlations, designing algorithms for universal generalization in dynamic environments remains an open challenge. Future work will need to focus on even more complex environmental shifts beyond just task-category changes.

Summary updated

Abstract

Despite the great success achieved in instructional action understanding by deep learning and mountainous data, deploying trained models to the unseen environment still remains a great challenge, since it requires strong generalizability of models from in-distribution training data to out-of-distribution (OOD) data. In this paper, we introduce a benchmark, named GAIN, to analyze the GeneralizAbility of INstructional action understanding models. In GAIN, we reassemble steps of existing instructional video training datasets to construct the OOD tasks and then collect the corresponding videos. We evaluate the generalizability of models trained on in-distribution datasets with the performance on OOD videos and observe a significant performance drop. We further propose a simple yet effective approach, which cuts off the excessive contextual dependency of action steps by performing causal inference, to provide a potential direction for enhancing the OOD generalizability. In the experiments, we show that this simple approach can improve several baselines on both instructional action segmentation and detection tasks. We expect the introduction of the GAIN dataset will promote future in-depth research on the generalization of instructional video understanding.
BibTeX
@inproceedings{li2023gain,
  author = {Junlong Li and Guangyi Chen and Yansong Tang and Jinan Bao and Kun Zhang and Jie Zhou and Jiwen Lu},
  title = {{GAIN: On the Generalization of Instructional Action Understanding}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=RlPmWBiyp6w},
}
Figure