Seeing through Events: Real-Time Moving Object Sonification for Visually Impaired People Using Event-Based Camera

Zihao Ji, Weijian Hu, Ze Wang, Kailun Yang, Kaiwei Wang

TL;DR

This paper introduces a novel real-time sonification framework that uses event-based cameras to help visually impaired individuals (VIP) perceive moving objects in their environment. By translating event data into distinct auditory cues, the system allows VIP to better understand object size, speed, and trajectory, offering a more intuitive and comfortable sensory experience than existing methods.

What Problem Does This Paper Address?

Visually impaired individuals often face challenges navigating and understanding their surroundings. While Electronic Travel Aids (ETAs) exist, many focus on static obstacles and struggle to convey the dynamic nature of the environment, particularly the presence and movement of multiple objects. Biological vision, in contrast, is highly attuned to motion. This paper aims to bridge this gap by leveraging event-based cameras, which excel at capturing motion with high temporal resolution and low latency, to create a sonification system that specifically highlights moving objects. The challenge is to translate this rich, dynamic visual information into an auditory experience that is both informative and aesthetically pleasing, avoiding information overload.

Methods

The core of the proposed framework involves several key steps:

  • Event Data Acquisition: An event-based camera captures asynchronous pixel-level brightness changes, generating event streams that represent motion.
  • Unsupervised Object Clustering: An adaptive Gaussian Mixture Model (GMM) is employed to cluster these events into distinct moving objects without prior knowledge of their number or characteristics. This GMM is adaptive, meaning it can dynamically adjust the number of clusters based on incoming event data, unlike standard GMMs that require a predefined number of clusters.
  • Real-Time Sonification: A parameter mapping sonification technique is used to translate object attributes into musical parameters using MIDI. Specifically:
  • Object Size: Mapped to different instrument timbres (e.g., "pad" for larger objects, "lead" for smaller ones).
  • Motion Speed: Mapped to note density (faster speed = denser notes).
  • Object Abscissa (Horizontal Position): Mapped to the stereo pan (left/right channel).
  • Object Ordinate (Vertical Position): Mapped to pitch.
  • New Appearance Attribute (Width): Explored using pedal (for wide objects) and polyphony (for narrow objects) to enhance distinguishability.
  • Auditory Output: The generated MIDI data is converted into an audio waveform and played through bone conduction headphones.

Datasets

The paper introduces a custom dataset created using PowerPoint animations to simulate two-dimensional translational movement of objects. This dataset was designed to control object appearance, motion properties, and number, aligning with the sonification design goals. The dataset was used for both training participants on the mapping principles and for testing the sonification method. The event camera was used to record the laptop screen displaying these animations.

Key Insights and Ideas

  • Event Cameras for Dynamic Perception: Event-based cameras are uniquely suited for capturing motion with high fidelity, making them ideal sensors for sonification systems focused on dynamic environments.
  • Adaptive Clustering for Real-World Scenarios: The adaptive GMM approach is crucial for real-time sonification, as it allows the system to handle varying numbers of moving objects without manual intervention.
  • Multi-Dimensional Sonification for Rich Information: Mapping multiple object attributes (size, speed, position) to distinct musical parameters creates a richer auditory experience, enabling VIP to perceive more about their environment.
  • The Trade-off Between Information and Artistry: While adding more attributes can increase information density, there's a limit to human perceptual capacity. The study highlights that a balance is needed to maintain both informativeness and auditory comfort.
  • VIP's Enhanced Auditory Perception: The results suggest that VIP may have a stronger ability to extract critical information from sonified audio compared to sighted individuals, particularly in complex multi-object scenarios (cocktail party effect).

Key Results

  • Distinguishable Object Attributes: Experiments showed that both VIP and sighted people could clearly distinguish the number, size, motion speed, and motion trajectories of multiple objects using the sonified audio.
  • Improved Auditory Comfort: The proposed method was found to be more comfortable to hear than existing methods, indicating a better aesthetic quality.
  • Objective Performance: In control group experiments, VIP showed a higher correct rate than sighted people in recognizing appearance and motion attributes, and a better match between audio and objects.
  • Experimental Group 1 (Added Width Attribute): While adding a "width" attribute improved the perception of object size and number for VIP, it slightly reduced the perception of other attributes compared to the control group. However, overall scores remained positive, indicating the scientific and artistic nature of the method.
  • Experimental Group 2 (Hu et al. Method): This group, using a different sonification approach, showed lower objective performance, particularly in distinguishing multiple objects, suggesting the proposed method's superiority for complex scenarios.

Limitations and Future Work

  • Stationary Camera Assumption: The current framework assumes a stationary camera. The authors acknowledge that extending this to scenarios with camera ego-motion is a future direction, requiring more complex algorithms and potentially impacting real-time performance.
  • Lack of Depth Information: The current sonification does not utilize depth information, which is typically mapped to volume. Extending the method to 3D scenes by incorporating depth is suggested.
  • Handling Multiple Instruments: The authors plan to explore methods to mitigate noise and improve clarity when a large number of musical instruments are playing simultaneously in scenes with many objects.
  • Real-World Robustness: Further research is needed to test the system's robustness in more complex and dynamic real-world environments.

Summary updated

Abstract

Scene sonification is a powerful technique to help Visually Impaired People (VIP) understand their surroundings. Existing methods usually perform sonification on the entire images of the surrounding scene acquired by a standard camera or on the priori static obstacles acquired by image processing algorithms on the RGB image of the surrounding scene. However, if all the information in the scene are delivered to VIP simultaneously, it will cause information redundancy. In fact, biological vision is more sensitive to moving objects in the scene than static objects, which is also the original intention of the event-based camera. In this paper, we propose a real-time sonification framework to help VIP understand the moving objects in the scene. First, we capture the events in the scene using an event-based camera and cluster them into multiple moving objects without relying on any prior knowledge. Then, sonification based on MIDI is enabled on these objects synchronously. Finally, we conduct comprehensive experiments on the scene video with sonification audio attended by 20 VIP and 20 Sighted People (SP). The results show that our method allows both participants to clearly distinguish the number, size, motion speed, and motion trajectories of multiple objects. The results show that our method is more comfortable to hear than existing methods in terms of aesthetics.
BibTeX
@inproceedings{ji2021seeing,
  author = {Zihao Ji and Weijian Hu and Ze Wang and Kailun Yang and Kaiwei Wang},
  title = {{Seeing through Events: Real-Time Moving Object Sonification for Visually Impaired People Using Event-Based Camera}},
  booktitle = {Sensors},
  year = {2021},
  url = {https://openreview.net/forum?id=OFf1OupkItpy},
}
Figure