SmartOverlays: A Visual Saliency Driven Label Placement for Intelligent Human-Computer Interfaces

Srinidhi Hegde, Jitender Maurya, Aniruddha Kalkar, Ramya HebbalaguppeWACV 2020

TL;DR

This paper introduces SmartOverlays, a novel framework for intelligently placing labels on objects in video feeds for augmented reality applications. By leveraging visual saliency and object tracking, SmartOverlays ensures labels are non-occluding, close to their objects, and temporally coherent, significantly improving AR visualization clarity.

What Problem Does This Paper Address?

Augmented reality (AR) applications often overlay contextual information, like text labels, onto real-world video feeds to enhance user understanding. However, a major challenge is how to place these labels effectively. Naive placement can lead to cluttered visuals, where labels obscure important scene elements or overlap with each other, diminishing the AR experience. Existing methods often rely on fixed rules or sensor data, which can be insufficient for dynamic scenes or generic applications. This paper tackles the problem of intelligent, dynamic label placement that prioritizes non-occlusion, proximity to objects, and temporal stability in video streams.

Methods

SmartOverlays is a multi-stage pipeline designed for real-time AR applications. Here's a breakdown of its key components:

  • Object Detection and Label Generation: The framework uses YOLOv2, a state-of-the-art object detector, to identify objects in video frames and generate corresponding class labels. This provides the raw material for labeling.
  • Saliency Estimation: A Saliency Attention Model (SAM) is employed to predict eye fixation points and generate saliency maps. These maps highlight regions of interest in the scene, guiding the label placement away from visually important areas.
  • Voronoi Partitioning for Proximity: To ensure labels are close to their objects, the image space is partitioned using Voronoi diagrams. The centroids of detected object bounding boxes serve as seed points for these partitions. This geometric approach guarantees that a label's placement point is closer to its corresponding object's centroid than any other object's centroid.
  • Label Placement Optimization: The core of SmartOverlays involves optimizing label placement based on several objectives:
  • Minimizing Occlusion over Saliency (LOS): A novel metric, LOS, is introduced to quantify how much a label would occlude salient regions. Labels are prioritized for placement in areas with low LOS.
  • Proximity: Achieved through Voronoi partitioning as described above.
  • Non-intersection of Connectors: Leader lines connecting objects to labels are designed to avoid intersecting each other, ensuring visual clarity.
  • Diagonal Heuristics and Central Bias: The placement considers aesthetic principles like placing labels along diagonal angle bisectors and accounting for the natural human tendency to focus on the center of an image.
  • Temporal Coherence: To prevent "jittery" labels in videos, two tracking-based methods are used:
  • Fixed Label: Assumes small object motion and updates anchor points without changing label locations for short intervals.
  • Tracking by Optical Flow: Extends the fixed label method by updating object centroid locations using exponential weighted averages and optical flow tracking to maintain smooth label movement synchronized with objects.
  • Adaptive Color Scheme: To improve legibility against dynamic backgrounds, the text color of labels adapts to provide strong color contrast.

Datasets

  • SALICON dataset: Used for training and evaluating the Saliency Attention Model (SAM).
  • COCO dataset: Used for pre-training the YOLOv2 object detector.
  • DIEM dataset: Used for analyzing the runtimes of the sub-modules.

Key Insights and Ideas

  • Saliency as a Guiding Principle: The paper effectively demonstrates that visual saliency is a powerful, inherent feature of images that can be leveraged for intelligent label placement, moving beyond simple geometric or rule-based approaches.
  • Voronoi Partitions for Proximity: Using Voronoi partitions based on object centroids is an elegant geometric solution to ensure labels remain close to their associated objects without complex search algorithms.
  • Novel LOS Metric: The introduction of the Label Occlusion over Saliency (LOS) metric is a significant contribution, providing a quantitative way to measure and minimize label occlusion of important scene content.
  • Temporal Coherence is Crucial for AR: The paper highlights that real-time AR requires not just accurate placement but also smooth, stable label movement across frames, addressing the common issue of jitter.
  • Generic Applicability: The framework is designed to be generic, working with any object detection technique and adaptable to various AR applications.

Key Results

  • Qualitative Improvements: Figure 1 and Figure 5 showcase visually compelling results where SmartOverlays produces cleaner, more organized label placements compared to naive methods, avoiding overlaps and occlusions.
  • Temporal Coherence: Figure 8 demonstrates that the proposed temporal coherence methods (Fixed Label and Optical Flow Tracking) significantly reduce temporal jitter compared to naive label placement, with the Optical Flow method showing superior performance.
  • Runtime: The entire pipeline takes approximately 4.34 seconds per frame, with the object detector and SAM being the main bottlenecks.
  • Effectiveness of Label Filtering: Figure 6 shows that larger label dimensions can lead to filtering due to space constraints, which can be beneficial in cluttered scenes.

Limitations and Future Work

  • Runtime Bottlenecks: The current runtime of 4.34s per frame is noted as a limitation for achieving truly seamless real-time performance. Future work aims to explore computationally efficient methods for object detection and saliency prediction.
  • Label Content and Size Constraints: The paper acknowledges that very long overlay texts can lead to wrapping, potentially obscuring saliency objects, and that labels larger than Voronoi partitions might be missed.
  • Future Directions: The authors suggest exploring more advanced saliency computation models that incorporate temporal data, creating compressed versions of SmartOverlays for portable devices, and investigating the use of temporal elements in saliency prediction. They also propose further research into optimizing the placement of labels with complex geometries.

Summary updated

BibTeX
@inproceedings{hegde2020smartoverlays,
  author = {Srinidhi Hegde and Jitender Maurya and Aniruddha Kalkar and Ramya Hebbalaguppe},
  title = {{SmartOverlays: A Visual Saliency Driven Label Placement for Intelligent Human-Computer Interfaces}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure