Low to High Dimensional Modality Hallucination Using Aggregated Fields of View

Kausic Gunasekar, Qiang Qiu, Yezhou Yang

Introduction

Autonomous systems, ranging from robotic manipulators to self-driving vehicles, rely on multi-modal sensor suites (RGB, Depth, LIDAR) for robust perception. However, hardware is susceptible to failure due to physical damage, power issues, or environmental interference. When a primary modality like RGB is lost, system performance typically degrades sharply. This paper proposes a software-based redundancy through "modality hallucination"—specifically, the reconstruction of high-dimensional RGB data from lower-dimensional depth maps.

Overview of the modality hallucination framework for autonomous systems in distress.

The Challenge of Low-to-High Hallucination

While estimating depth from RGB (monocular depth estimation) is a well-explored field, the inverse—recovering rich color and texture from sparse geometric depth—is an under-constrained problem. The authors aim to develop a task-agnostic, online-trainable method that can "fill in the blanks" when an RGB camera fails, utilizing only the remaining depth data to maintain operational decision-making capabilities.

Methodology: Aggregated Fields of View

The core innovation is an encoder-decoder neural network architecture featuring two novel building blocks: AggConv (Aggregated Convolution) and AggTrConv (Aggregated Transpose Convolution).

Unlike standard convolutions that use a single kernel size (e.g., 3x3), AggConv runs multiple kernels of varying sizes (from 3x3 to 11x11) and dilations in parallel. This allows the network to capture broader spatial context, which is essential for predicting color; for instance, identifying a surface as part of a desk helps the model infer its likely color. To maintain computational efficiency, the authors employ dilated convolutions, which expand the receptive field without a proportional increase in parameters.

The model is trained using a dual-component loss function:

  1. Root Mean Square Error (RMSE): Captures the general mapping between depth and color spaces.
  2. Edge-Aware Smoothness: Uses a Huber function to ensure local smoothness while preserving sharp object boundaries.

Experimental Evaluation

The method was evaluated on three major benchmarks: UWRGBD (University of Washington RGB-D), NYUD-v1 (NYU Depth), and the TUM Dataset (to test generalization to new environments). The proposed AggConv architecture was compared against standard baselines, including pix2pix (a Conditional GAN) and LinkNet.

Key Results

The utility of the hallucinated images was validated by feeding them into standard vision models for classification (AlexNet) and segmentation (LinkNet).

  • Classification (UWRGBD): Raw depth alone yielded only 2.12% accuracy. Hallucinated RGB from AggConv boosted this to 51.14%, significantly outperforming pix2pix (27.48%) and LinkNet (29.19%).
  • Segmentation (NYUD): AggConv achieved 35.78% pixel accuracy, nearly doubling the 18.25% achieved with raw depth.
  • Image Quality: AggConv demonstrated superior structural consistency compared to GAN-based approaches, which often produced artifacts or missed key objects entirely.
Qualitative comparison of depth maps (a), hallucinated RGB (b, c), and ground truth RGB (d) across different scenes.

Discussion and Limitations

A key insight of this work is that context is paramount; low-to-high mapping cannot be performed effectively on a pixel-by-pixel basis. Interestingly, the authors found that hallucinated data can even supplement real data, improving performance in some tasks when combined with actual RGB and depth.

However, limitations remain. The model's accuracy is bound by the "constancy of relationships" learned during training; it may struggle with novel colors that do not correlate with known object geometries. Additionally, while a "Regularizer Network" (shown as Hal* in Figure 4) can improve visual quality, its implementation for real-time online training presents further challenges. Future work will need to address temporal consistency for high-speed applications like autonomous flight or driving.

Conclusion

By aggregating multiple fields of view, this research demonstrates that meaningful high-dimensional visual information can be recovered from geometric data. This approach provides a viable software-based fail-safe for autonomous systems, ensuring they remain functional even under sensor degradation.

Summary updated

Abstract

Real-world robotics systems deal with data from a multitude of modalities, especially for tasks such as navigation and recognition. The performance of those systems can drastically degrade when one or more modalities become inaccessible, due to factors such as sensors' malfunctions or adverse environments. Here, we argue modality hallucination as one effective way to ensure consistent modality availability and thereby reduce unfavorable consequences. While hallucinating data from a modality with richer information, e.g., RGB to depth, has been researched extensively, we investigate the more challenging low-to-high modality hallucination with interesting use cases in robotics and autonomous systems. We present a novel hallucination architecture that aggregates information from multiple fields of view of the local neighborhood to recover the lost information from the extant modality. The process is implemented by capturing a non-linear mapping between the data modalities and the learned mapping is used to aid the extant modality to mitigate the risk posed to the system in the adverse scenarios which involve modality loss. We also conduct extensive classification and segmentation experiments on UWRGBD and NYUD datasets and demonstrate that hallucination allays the negative effects of the modality loss. Implementation and models: https://github.com/kausic94/Hallucination.
BibTeX
@misc{gunasekar2020low,
  author = {Kausic Gunasekar and Qiang Qiu and Yezhou Yang},
  title = {{Low to High Dimensional Modality Hallucination Using Aggregated Fields of View}},
  year = {2020},
  eprint = {2007.06166},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.06166},
}
Figure