DMR: Decomposed Multi-Modality Representations for Frames and Events Fusion in Visual Reinforcement Learning

Haoran Xu, Peixi Peng, Guang Tan, Yuan Li, Xinhai Xu, Yonghong Tian

Abstract

Standard visual Reinforcement Learning (RL) often struggles in challenging environments due to the inherent limitations of RGB sensors, such as low dynamic range and motion blur. While bio-inspired event cameras (DVS) offer high dynamic range and low latency, effectively fusing them with RGB data is difficult because simple integration often introduces task-irrelevant noise. We present Decomposed Multi-Modality Representations (DMR), a framework that explicitly separates task-relevant "co-features" from modality-specific noise. By employing contrastive and reconstruction losses, DMR achieves superior performance in complex autonomous driving scenarios, such as midnight driving and heavy rain, significantly outperforming existing state-of-the-art fusion methods.

Introduction

Autonomous agents operating in real-world environments must contend with diverse lighting and weather conditions. Standard RGB cameras, while providing rich texture and semantic information, have a limited dynamic range (typically around 70 dB). This makes them unreliable in extreme conditions like tunnels, night driving, or facing direct glare. Conversely, event cameras (DVS) offer a massive dynamic range (120 dB) and microsecond-level latency by recording asynchronous brightness changes. However, DVS sensors only capture motion; they fail to perceive static objects, which are crucial for navigation.

Integrating these complementary modalities in a Reinforcement Learning context is challenging. Existing multi-modality RL methods often suffer from noise injection, where task-irrelevant information—such as lens flare in RGB frames or sensor artifacts in DVS streams—is embedded into the agent's latent representation. Furthermore, the sparse reward signals typical of RL provide insufficient supervision for the model to distinguish between useful signals and modality-specific noise.

DMR focuses on task-relevant features like pedestrians and road edges compared to single-modality baselines which are distracted by environmental noise.

The DMR Framework

The proposed Decomposed Multi-Modality Representation (DMR) framework addresses these challenges by explicitly disentangling the input data into three distinct components. Instead of a single monolithic encoder, DMR utilizes a three-branch architecture:

  1. Co-feature Branch: This branch processes both RGB and DVS inputs to extract features that are shared across modalities and critical for the RL task, such as the position of pedestrians or the curvature of the road.
  2. RGB Noise Branch: This branch isolates features unique to the RGB camera that do not contribute to the driving task (e.g., reflections or color shifts).
  3. DVS Noise Branch: This branch captures modality-specific noise inherent to event-based sensors that is irrelevant to the agent's objectives.

Optimization Objectives

To achieve effective decomposition without manual labels, DMR employs three specialized constraints:

  • Task Guidance: The co-features are trained to predict the next state and the expected reward using a DeepMDP approach. This ensures that the co-feature latent space contains all information necessary for optimal decision-making.
  • Contrastive Repelling: A contrastive loss function is used to "repel" noise features from co-features in the latent space. This forces the model to ensure that the noise branches do not capture any information that is already represented in the task-relevant co-feature branch.
  • Information Completeness: To prevent the model from discarding essential information, reconstruction decoders are used. These decoders must be able to reconstruct the original RGB and DVS observations by combining the task-relevant co-features with their respective modality-specific noise features.
The DMR architecture uses three encoders to decompose inputs into task-relevant co-features and modality-specific noise, guided by task-based and reconstruction losses.

Experimental Evaluation

The researchers evaluated DMR using the Carla simulator, a high-fidelity environment for autonomous driving research. They established a rigorous benchmark focusing on two particularly difficult scenarios:

  • JayWalk (JW): The vehicle must navigate safely while avoiding stationary and moving pedestrians in low-visibility conditions.
  • HighBeam (HB): The vehicle must maintain control while facing the intense glare of oncoming high beams.

Both scenarios were tested under Midnight (extreme low light) and Hardrain (high visual noise) settings to push the limits of the sensors.

Key Results and Insights

DMR consistently outperformed both single-modality baselines and multi-modal state-of-the-art (SOTA) fusion methods like TransFuser, EFNet, and RENet.

  • Performance Gains: In the "JayWalk-Midnight" scenario, DMR achieved a cumulative reward of 194, significantly higher than DVS-only (136) or RGB-only (102) policies.
  • Safety and Reliability: In the "HighBeam-Hardrain" test, the DMR agent managed to drive an average of 150 meters before a collision, nearly doubling the 91-meter average of standard RGB-based agents.
  • Visual Focus: Qualitative analysis using Class Activation Maps (CAMs) reveals that DMR's latent space focuses precisely on task-critical elements like pedestrians and road boundaries. In contrast, standard models often focus on irrelevant environmental features like buildings or lens flares, leading to sub-optimal actions.

Conclusion

DMR demonstrates that explicit modality decomposition is a superior strategy for multi-modal visual RL. By separating task-relevant signals from sensor-specific noise, agents can effectively leverage the complementary strengths of RGB and DVS sensors. This approach not only improves performance in extreme environments but also provides a more interpretable latent space for autonomous decision-making.

Paper Citation: Xu, H., Peng, P., Tan, G., Li, Y., Xu, X., & Tian, Y. (2024). DMR: Decomposed Multi-Modality Representations for Frames and Events Fusion in Visual Reinforcement Learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

Summary updated

Abstract

We explore visual reinforcement learning (RL) using two complementary visual modalities: frame-based RGB cam-era and event-based Dynamic Vision Sensor (DVS). Ex-isting multi-modality visual RL methods often encounter challenges in effectively extracting task-relevant information from multiple modalities while suppressing the in-creased noise, only using indirect reward signals instead of pixel-level supervision. To tackle this, we propose a Decomposed Multi-Modality Representation (DMR) framework for visual RL. It explicitly decomposes the inputs into three distinct components: combined task-relevant features (co-features), RGB-specific noise, and DVS-specific noise. The co-features represent the full information from both modalities that is relevant to the RL task; the two noise components, each constrained by a data reconstruction loss to avoid information leak, are contrasted with the co-features to maximize their difference. Extensive experiments demonstrate that, by explicitly separating the different types of information, our approach achieves substan-tially improved policy performance compared to state-of-the-art approaches.
BibTeX
@inproceedings{xu2024dmr,
  author = {Haoran Xu and Peixi Peng and Guang Tan and Yuan Li and Xinhai Xu and Yonghong Tian},
  title = {{DMR: Decomposed Multi-Modality Representations for Frames and Events Fusion in Visual Reinforcement Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.02503},
}
Figure