HA-ViD: A Human Assembly Video Dataset for Comprehensive Assembly Knowledge Understanding

Hao Zheng, Regina Lee, Yuqian LuNeurIPS 2023

While artificial intelligence has made significant strides in recognizing simple, everyday actions (like "drinking water" or "walking"), it consistently struggles with the complex, multi-step procedures found in real-world manufacturing. Existing datasets often fall short because they are either too application-specific, lack representative industrial tools, or ignore the cognitive process of learning a task.

Furthermore, there is a persistent "language gap" between how humans describe actions and how robots need to understand them. To build "ultra-intelligent" factories where humans and robots collaborate seamlessly, we need datasets that break down actions into granular, transferable components. Enter HA-ViD, a massive new multi-modal video dataset designed to help AI understand the nuances of industrial assembly by capturing humans learning to build a complex product.

Methods & Data Collection

To create a representative benchmark, the researchers developed the Generic Assembly Box (GAB), a standardized product containing 35 parts (11 standard, 24 non-standard) and requiring 4 different tools.

They recorded 30 participants assembling this box using a unique three-stage progressive learning setup designed to capture natural human behavior, including pauses, errors, and alternative routes:

  1. Discovery: Participants receive minimal instructions, forcing them to explore the "how-to" and problem-solve.
  2. Instruction: Participants follow detailed, step-by-step guides.
  3. Practice: Participants perform the assembly task entirely from memory.
Overview of the HA-ViD dataset, illustrating the Generic Assembly Box, multi-view data collection, and annotation taxonomy.

Annotation and Taxonomy

To annotate this complex data, the team utilized the Human-Robot Shared Assembly Taxonomy (HR-SAT). This protocol decomposes every action into five critical elements: Subject, Action Verb, Manipulated Object, Target Object, and Tool. This structured approach ensures the resulting data is technically useful for training robotic systems, bridging the comprehension gap between human action and robot execution.

The HA-ViD Dataset

The primary contribution of this research is the HA-ViD dataset itself, which stands out in both scale and detail:

  • Scale: 3,222 multi-view videos captured from side, front, and top angles, featuring multi-modality data (RGB, Depth, and Skeleton).
  • Labels: The dataset encompasses 1.5 million frames, featuring 96,000 temporal labels and over 2 million spatial (bounding box) labels.
  • Comparison: Unlike predecessors such as Assembly101 or IKEA-FA, HA-ViD systematically tracks varying efficiency, alternative assembly routes, human errors, and pauses within a consistent shared taxonomy.
Dataset statistics highlighting the distribution of left/right-hand annotations and the prevalence of small objects.

Key Insights and Challenges

The dataset reveals several critical blind spots in current foundational video understanding techniques:

  • The "Null" Action Challenge: One of the most frequent "actions" in the dataset is actually a pause (null). Current models are surprisingly bad at identifying these moments because they are typically trained on datasets where continuous action is the norm.
  • Compositional Complexity: Recognizing an industrial action isn't just about the motion; it's about the objects involved. Researchers found that models struggle significantly more with identifying the interacted objects and tools than the action verbs themselves.
  • Under-Segmentation: Current action segmentation models tend to "under-segment," grouping several distinct assembly steps into one giant block. To quantify this specific failure, the authors propose a new metric called Segmentation Adequacy (SA).
  • The Two-Hand Problem: Most video models rely on "global features" that analyze the whole frame. HA-ViD reveals that these models fail to independently track what the left and right hands are doing simultaneously—a critical requirement for parallel industrial tasks.

Key Results

The authors benchmarked several State-of-the-Art (SOTA) models against the HA-ViD dataset:

  • Action Recognition: The UniFormerV2 model achieved the best performance, reaching a 62.4% Top-1 accuracy on primitive tasks.
  • Object Detection: The DINO model reached 35.5 Average Precision (AP). However, performance on "small objects"—which make up a staggering 46.6% of the dataset—remains a major bottleneck, scoring only 27.4 AP.
  • Multi-Object Tracking (MOT): While models performed well when given ground-truth bounding boxes, their performance plummeted when relying on automated detections. This highlights the extreme difficulty of tracking small, frequently occluded assembly parts in real-time.

Limitations and Future Work

The authors acknowledge that HA-ViD currently lacks direct force and torque data, which are vital for certain robotic "skill learning" applications. Additionally, while object poses can be inferred, they are not yet explicitly labeled in 3D space.

Future iterations of the project aim to expand the dataset's utility by including 3D and pixel-wise annotations (moving beyond simple bounding boxes to capture precise 3D hand-object interactions). The team also plans to introduce diverse environments with varying lighting conditions and industrial settings, and to leverage trajectory data to predict human intention—shifting the focus from what a human is doing to what they intend to do next.

Summary updated

BibTeX
@inproceedings{zheng2023ha,
  author = {Hao Zheng and Regina Lee and Yuqian Lu},
  title = {{HA-ViD: A Human Assembly Video Dataset for Comprehensive Assembly Knowledge Understanding}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
}
Figure