FlexLoc: Conditional Neural Networks for Zero-Shot Sensor Perspective Invariance in Object Localization with Distributed Multimodal Sensors

Jason Wu, Ziqi Wang, Xiaomin Ouyang, Ho Lyun Jeong, Colin Samplawski, Lance M. Kaplan, Benjamin M. Marlin, Mani Srivastava

TL;DR

This paper introduces FlexLoc, a novel approach for object localization that makes systems robust to changes in sensor viewpoints without requiring recalibration. By injecting sensor pose information into conditional neural networks, FlexLoc achieves significant accuracy improvements in zero-shot scenarios, making it more practical for real-world deployments.

What Problem Does This Paper Address?

Accurate object localization is crucial for many applications, from warehouse automation to healthcare. Modern systems often fuse data from multiple sensors (cameras, radar, etc.) at different viewpoints to improve robustness and accuracy. However, these systems typically rely on deep learning models trained on specific sensor configurations. When the sensors' positions or orientations change in real-world use (e.g., due to maintenance or relocation), these models can fail dramatically, leading to extreme inaccuracies. Retraining models for every new sensor setup is impractical. This paper tackles the challenge of achieving "sensor perspective invariance" – enabling localization systems to adapt to unseen sensor viewpoints without needing new training data or calibration.

Methods

FlexLoc builds upon an early-fusion, end-to-end deep learning architecture. The core innovation lies in injecting sensor pose information (position and orientation) into the network using conditional neural networks. This allows the model to dynamically adapt its internal workings based on the current sensor setup. Two primary methods for injecting this pose information are proposed:

  • Conditional 1D Convolution (CondConv): This method uses the sensor pose to generate weights and biases for 1D convolutional kernels. These adaptive kernels then process the extracted sensor features, effectively transforming them with the context of the sensor's viewpoint.
  • Conditional Layer Normalization (CLN): A more lightweight approach, CLN replaces the learnable scale and shift parameters in normalization layers (like LayerNorm or BatchNorm) with values derived from the sensor pose. This subtly influences the feature processing throughout the network.

Both methods are integrated into a transformer encoder-based architecture that fuses features from multiple sensor modalities.

Datasets

The authors introduce and utilize the GDTM (Geospatial Tracking Dataset with Multimodal Sensors) dataset. This dataset was collected over nine hours, featuring three sensor nodes with RGB camera, depth, mmWave radar, and audio modalities. Crucially, it includes 22 unique sensor node perspectives, specifically designed to study the impact of viewpoint changes. The dataset is split into training, validation, and testing sets, with a rigorous procedure to ensure that test viewpoints are distinct from training and validation viewpoints.

Key Insights and Ideas

  • Sensor Pose is Critical: The paper strongly demonstrates that even small deviations in sensor pose can lead to catastrophic failures in standard deep learning localization models. This highlights a significant practical limitation of current approaches.
  • Conditional Networks as a Solution: Injecting sensor pose information dynamically via conditional neural networks is a powerful way to achieve perspective invariance. The network learns how the pose affects predictions during training and uses this knowledge at inference time.
  • CondConv vs. CLN Trade-off: CondConv offers superior performance, likely due to its more direct influence on feature transformation. CLN, while less performant, is significantly more lightweight and offers a good trade-off for resource-constrained environments.
  • Importance of Pretraining: Pre-training backbone networks is crucial, especially for the CLN approach, as it makes the network less sensitive to the initial weights and more robust to the conditional modifications.
  • Sensor Dropout for Robustness: Training with modality dropout (randomly disabling modalities) forces the model to learn from diverse sensor combinations, improving its resilience to missing sensors.

Key Results

  • FlexLoc (specifically the CondConv variant) improves localization accuracy by almost 50% in zero-shot scenarios (unseen perspectives) compared to baseline methods.
  • CondConv achieves an impressive 32% improvement over a single-perspective late-fusion model and comes within 10% of a single-perspective early-fusion model, demonstrating its robustness to completely unseen viewpoints.
  • The low 90% error for FlexLoc is 20 cm less than the next closest baseline method.
  • The CondConv variant adds only 0.19% more parameters and 0.57% more computational overhead (MACs) compared to the unconditional model.
  • The CLN variant is even more efficient, adding 0.04% fewer parameters and 0.32% fewer MACs.

Limitations and Future Work

  • Sensor Pose Estimation: The current system relies on external motion capture systems (OptiTrack) for ground truth pose at test time. Future work aims to implement self-localization techniques (e.g., using AprilTags or markerless methods) to make the system fully end-to-end and independent of external infrastructure.
  • Environment Complexity: The current evaluation is on a relatively simple indoor track. Future work will explore more complex indoor and outdoor environments with diverse settings, occlusions, and weather conditions.
  • Multi-Object Localization: The paper suggests extending the architecture to handle multi-object localization by replacing the transformer encoder with a decoder similar to DETR.
  • Late Fusion Comparison: While early fusion was used for superior performance, the authors acknowledge the advantages of late fusion (flexibility, communication overhead) and plan to investigate optimal strategies for integrating these benefits, particularly in distributed computing settings.

Summary updated

Abstract

Localization is a critical technology for various applications ranging from navigation and surveillance to assisted living. Localization systems typically fuse information from sensors viewing the scene from different perspectives to estimate the target location while also employing multiple modalities for enhanced robustness and accuracy. Recently, such systems have employed end-to-end deep neural models trained on large datasets due to their superior performance and ability to handle data from diverse sensor modalities. However, such neural models are often trained on data collected from a particular set of sensor poses (i.e., locations and orientations). During real-world deployments, slight deviations from these sensor poses can result in extreme inaccuracies. To address this challenge, we introduce FlexLoc, which employs conditional neural networks to inject node perspective information to adapt the localization pipeline. Specifically, a small subset of model weights are derived from node poses at run time, enabling accurate generalization to unseen perspectives with minimal additional overhead. Our evaluations on a multimodal, multi-view indoor tracking dataset showcase that FlexLoc improves the localization accuracy by almost 50% in the zero-shot case (no calibration data available) compared to the baselines. The source code of FlexLoc is available in https://github.com/nesl/FlexLoc.
BibTeX
@inproceedings{wu2024flexloc,
  author = {Jason Wu and Ziqi Wang and Xiaomin Ouyang and Ho Lyun Jeong and Colin Samplawski and Lance M. Kaplan and Benjamin M. Marlin and Mani Srivastava},
  title = {{FlexLoc: Conditional Neural Networks for Zero-Shot Sensor Perspective Invariance in Object Localization with Distributed Multimodal Sensors}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  doi = {10.1109/iros58592.2024.10802096},
}
Figure