CLUE-AD: A Context-Based Method for Labeling Unobserved Entities in Autonomous Driving Data

Ruwan Wickramarachchi, Cory A. Henson, Amit P. ShethAAAI 2023 · DOI

Abstract

High-quality training data is the lifeblood of autonomous driving (AD), yet even the most advanced datasets contain "blind spots" where objects go unlabeled due to occlusion, sensor failures, or low resolution. CLUE-AD is a dataset-agnostic framework that automatically identifies and labels these "missing" objects by analyzing surrounding video frames. By leveraging the psychological principle of object continuity, it recovers labels for unobserved entities (UEs), providing a significantly richer ground truth for training safer perception systems.

The Challenge of Unobserved Entities

Standard computer vision approaches often treat each frame in isolation. If an object is not immediately visible—for instance, if a truck blocks a pedestrian or a person is sitting inside a vehicle—it typically remains unlabeled. This creates a gap in "object permanence," the understanding that an object continues to exist even when it moves out of sight. For a self-driving car, forgetting a pedestrian exists simply because they stepped behind a van is a critical safety risk.

The CLUE-AD framework addresses common AD challenges like occlusion and sensor failure by using a temporal workflow to label unobserved entities.

The CLUE-AD Framework

The authors introduce a multi-step workflow designed to be compatible with any existing AD dataset:

  1. Scene Processing: Raw dataset sequences are standardized into ordered <scene:labels> pairs to ensure temporal consistency.
  2. Object Continuity Assumption: Grounded in developmental psychology, this core logic assumes that if an object is observed in the immediate past or future of a target scene, it still exists within that scene, even if sensors cannot currently detect it.
  3. Scene Windowing: Users can configure a temporal "window" (backward, forward, or bidirectional) to look for missing entities.
  4. Inference Rule: Any entity found within the surrounding window that is missing from the target frame is flagged as a UE and added to the augmented labels.
  5. Interactive Interface: A Python-based tool allows researchers to explore augmented scenes and visualize inferred labels.

Key Insights and Results

One of the most significant applications of CLUE-AD is the "nested object" solution. While a standard bounding box might only label a "Car," CLUE-AD can infer and label a "Person Sitting" inside that car based on frames where they were previously visible (e.g., getting into the vehicle).

An example of a 'nested object' where CLUE-AD identifies a person sitting inside a car, a detail often missed in standard bounding box annotations.

The researchers evaluated CLUE-AD using PandaSet, featuring complex urban driving scenes. Key findings include:

  • Significant Augmentation: Using a bidirectional window of 19 frames, CLUE-AD found that 12.49% of all scenes contained at least one unobserved object missed by original annotators.
  • Label Density: In augmented scenes, the framework added an average of 1.34 new labels per frame.
  • Qualitative Success: The system successfully identified complex UEs such as "PersonSitting" and "PersonWithObject" in scenarios where traditional methods only accounted for the containing vehicle.

Conclusion and Future Work

CLUE-AD shifts the focus of AD perception from pure detection to contextual inference. While the current framework does not explicitly account for "false positives" (objects leaving the scene within the window), it provides a robust foundation for enriching datasets. Future work involves using this data to improve Knowledge-based Entity Prediction (KEP) models, which aim to predict entity movement using semantic reasoning—a critical step toward human-like reasoning in autonomous vehicles.

Summary updated

BibTeX
@inproceedings{wickramarachchi2023clue,
  author = {Ruwan Wickramarachchi and Cory A. Henson and Amit P. Sheth},
  title = {{CLUE-AD: A Context-Based Method for Labeling Unobserved Entities in Autonomous Driving Data}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.27089},
}
Figure