tinyHulk: Lightweight Annotation for Wearable Cognitive Assistance

Chanh Nguyen, Roger Iyengar, Qifei Dong, Jim Blakley, Mahadev SatyanarayananResearch Square 2024 · DOI

TL;DR

tinyHulk is a lightweight, automated tool designed to accelerate the creation of training datasets for Wearable Cognitive Assistance (WCA). By leveraging "green screen" recording and chroma keying, it reduces human annotation time by 4x while producing models that outperform those trained on manually labeled or purely synthetic data.

What Problem Does This Paper Address?

Wearable Cognitive Assistance (WCA) systems—think smart glasses that guide you through assembling furniture or repairing an engine—rely on Deep Neural Networks (DNNs) to recognize objects and task states in real-time. However, every new WCA application requires a custom-trained model, which in turn requires thousands of annotated images.

Currently, this is a massive bottleneck. Manual annotation using tools like CVAT is "painful" and labor-intensive; for instance, labeling a dataset for a single IKEA cart assembly app can take over 50 person-hours. While synthetic data (generated via 3D models) is an alternative, it requires high-fidelity 3D assets and specialized rendering skills that many developers lack. tinyHulk aims to provide a "middle way": high-quality, automated labeling using real-world video but minimal human effort.

Methods

The tinyHulk workflow transforms a raw video into a fully labeled, augmented dataset through a four-stage pipeline:

  1. Video Parsing & Duplicate Removal: To prevent the model from learning biases from redundant frames (e.g., when a camera moves slowly), the tool uses perceptual hashing. This converts frames into 64-bit integers and uses Hamming distance to filter out near-duplicates.
  2. Automatic Bounding-Box Drawing: The user records the object of interest on a uniform, high-contrast background (typically green). tinyHulk uses adaptive thresholding and contour detection to automatically wrap a 2D bounding box around the object in every frame.
  3. Interactive Inspection & Anomaly Detection: A human supervisor reviews the results. To save time, the tool uses an interquartile range (IQR) technique to flag "suspicious" boxes (outliers in size or position), allowing the human to focus only on likely errors.
  4. Background Replacement (Data Augmentation): Inspired by the film industry, the tool uses chroma keying in the HSV (Hue, Saturation, Value) color space to swap the green background for diverse, real-world scenes (wood, tile, cluttered desks). This "object-aware" augmentation significantly increases the model's ability to generalize to different environments.

Datasets

The authors evaluated tinyHulk using two complex assembly tasks:

  • Meccano Bike: A multi-step assembly of a toy motorcycle.
  • Toy Plane: A 10-step assembly of a toy aircraft.

They compared tinyHulk-generated data against:

  • R-M (Real-Manual): Real-world images manually labeled in CVAT.
  • S-M (Synthetic-Model): Data generated using CAD models and the Unity Perception package.
  • G-M & GD-M: Data generated by tinyHulk (Green-only vs. Green+Diverse backgrounds).

Key Insights and Ideas

  • The "Green Screen" Advantage: By constraining the initial recording environment (the green background), the authors simplify the computer vision task of object detection to a point where it can be automated with high precision (99.5% accuracy), bypassing the need for expensive manual segmentation.
  • Duplicate Removal is Non-Trivial: The paper highlights that simply having more data isn't better if that data is redundant. Removing duplicates via perceptual hashing actually improved model performance by preventing the DNN from over-weighting specific viewpoints.
  • Real Objects > Synthetic Textures: Models trained on tinyHulk’s "real-object-on-fake-background" data (GD-M) often outperformed those trained on purely synthetic data. This suggests that the subtle lighting and texture nuances of a real physical object are still difficult to perfectly replicate in 3D engines like Unity.

Key Results

  • Efficiency: tinyHulk achieved a 4x speedup in human effort. Manual labeling took roughly 0.4 seconds per frame, while tinyHulk required only 0.1 seconds of human supervision per frame.
  • Annotation Accuracy: The automated bounding-box generation achieved an average accuracy of 99.5% across all datasets.
  • Model Performance: In cluttered, real-world test environments, the tinyHulk-augmented model (GD-M) achieved an accuracy of 84.6%, significantly higher than the manually labeled model (66.1%) and the synthetic model (76.1%).

Limitations and Future Work

  • Sensitivity to Background Noise: The tool struggles if the "green screen" has wrinkles, shadows, or excessive detail, which can lead to inaccurate bounding boxes.
  • Camera Auto-Settings: Many smart cameras use automatic white balance and exposure, which can shift the "green" color into different HSV ranges mid-video, breaking the chroma keying logic.
  • Future Directions: The authors plan to develop a web-based version of tinyHulk and investigate more sophisticated image augmentation techniques to create even more diverse training sets from a single video source.

Summary updated

Abstract

<title>Abstract</title> Wearable Cognitive Assistance (WCA) employs DNNs models to provide real time step-by-step guidance to end-users for specific tasks. Collecting and annotating training data for such DNNs is painful and takes massive human effort and time. In this work, we propose tinyHulk, an automatic annotation tool that reduces human time and effort for these labor-intensive annotation tasks. Further, the tool is equipped with the background replacement, an object-aware data augmentation mechanism to increase the size and diversity of the training set. Our experimental results show that using tinyHulk can save human time up to fourfold compared to using manual methods for labeling tasks. The computer vision models trained with data annotated and generated by tinyHulk achieve high accuracy, outperforming the models trained with real or synthetic data using conventional methods for annotation.
BibTeX
@inproceedings{nguyen2024tinyhulk,
  author = {Chanh Nguyen and Roger Iyengar and Qifei Dong and Jim Blakley and Mahadev Satyanarayanan},
  title = {{tinyHulk: Lightweight Annotation for Wearable Cognitive Assistance}},
  booktitle = {Research Square},
  year = {2024},
  doi = {10.21203/rs.3.rs-3991530/v1},
}
Figure