SenseShift6D: Multimodal RGB-D Benchmarking for Robust 6D Pose Estimation across Environment and Sensor Variations

Yegyu Han, Taegyoon Yoon, Dayeon Woo, Sojeong Kim, Hyung-Sin Kim

Abstract

Existing 6D pose estimation benchmarks often overlook the impact of varying environmental lighting and camera sensor configurations. We present SenseShift6D, a comprehensive RGB-D dataset capturing thousands of physical sensor and lighting permutations. Our evaluation reveals that state-of-the-art models significantly degrade under hardware-level shifts. However, by employing test-time sensor control—dynamically adjusting exposure, gain, and depth modes—performance can be improved by up to 19.5% without additional training.

Introduction

Estimating the 6D pose of objects is a critical task for robotics, autonomous vehicles, and augmented reality. While modern AI models achieve high accuracy on standard benchmarks like LineMOD or YCB-V, these datasets are typically captured under strictly controlled, static conditions. In real-world scenarios, ambient lighting changes constantly, prompting cameras to use auto-exposure and varying gain, while depth sensors switch capture modes.

These hardware adjustments introduce complex, non-linear physical artifacts—such as photon-shot noise, color shifts, and structured depth errors—that synthetic data augmentation cannot accurately replicate. There is a significant gap in the research community's ability to measure how 6D pose models degrade under these hardware-level shifts and whether robustness can be improved by tuning camera settings on the fly.

The SenseShift6D Dataset

To address this gap, we developed a highly controlled physical testbed using a darkroom, programmable smart bulbs, a motorized turntable, and an Intel RealSense D455 camera. We systematically captured objects under a massive grid of physical sensor and lighting configurations, resulting in the SenseShift6D dataset.

Grid of RGB sensor control variations across different illumination levels (B5 to B100), comparing auto-exposure (AE) with manual sensor control.

The dataset includes five household objects (spray bottle, Pringles, tincase, sandwich replica, and mouse) captured across 13 RGB exposures, 9 RGB gains, auto-exposure, 4 depth-capture modes, and 5 illumination levels. This yields 166.4k RGB images and 16.7k depth images, providing 1,380 unique sensor-lighting permutations per object pose.

Visualization of a colormapped depth map from the SenseShift6D dataset, illustrating the depth modality used for multimodal pose estimation.

Methods and Evaluation

We evaluated several state-of-the-art 6D pose estimation models, including generalizable "foundation" models (GigaPose, SAM-6D, FoundationPose) and instance-level models (ZebraPose, GDRNPP, HiPose). To test the potential of test-time sensor control, we compared model performance under five regimes: factory defaults (Baseline), Oracle-fixed settings, and Oracle-dynamic settings (choosing the best setting for each specific scene).

Key Results and Insights

Our experiments yielded several critical insights into the relationship between hardware and AI performance:

  • Massive Gains Without Retraining: For pretrained generalizable models, using dynamic, per-scene sensor control (Oracle-Dynamic) improved performance by a staggering +19.5 percentage points (pp) over the baseline.
  • Multimodal Synergy: Tuning just the RGB sensor or just the depth sensor helps (+12.3 pp and +10.5 pp respectively), but tuning both together yields compounding, synergistic benefits.
  • Hardware as a Tunable Layer: Mitigating domain shift at inference time by physically optimizing the camera sensor can be more effective—and compute-efficient—than forcing a model to learn every possible variation through data augmentation.
  • Increased Stability: Dynamic sensor control reduced the performance gap between the best- and worst-performing objects from 40.0 pp down to 13.6 pp, significantly increasing model reliability.

Conclusion

SenseShift6D demonstrates that factory default camera settings, often tuned for human perception, are suboptimal for computer vision tasks. By treating the camera sensor as a tunable layer of the inference pipeline, we can achieve significant performance boosts. Future work will expand the dataset to include multi-object clutter and directional lighting, with the ultimate goal of developing real-time, adaptive sensor-control algorithms that can automatically optimize camera settings in the wild.

Summary updated

Abstract

Recent advances on 6D object-pose estimation have achieved high performance on representative benchmarks such as LM-O, YCB-V, and T-Less. However, these datasets were captured under fixed illumination and camera settings, leaving the impact of real-world variations in illumination, exposure, gain or depth-sensor mode—and the potential of test-time sensor control to mitigate such variations—largely unexplored. To bridge this gap, we introduce SenseShift6D, the first RGB-D dataset that physically sweeps 13 RGB exposures, 9 RGB gains, auto-exposure, 4 depth-capture modes, and 5 illumination levels. For six common household objects (spray, pringles, tincase, sandwich, mouse, and duck), we acquire 198.8k RGB and 20.0k depth images, which can provide 1,380 unique sensor-lighting permutations per object pose. Experiments with state-of-the-art models on our dataset demonstrate that applying multimodal sensor control at test time yields substantial performance gains, achieving a 18.0 pp improvement on pretrained generalizable models. It also enhances robustness precisely where those models tend to fail. Moreover, even instance-level pose estimators, where train and test set share identical object and background, performance still varies under environmental and sensor change, demonstrating that test-time sensor control remains effective compared to costly expansions in the quantity and diversity of real-world training data, without any additional training. SenseShift6D extends the object pose evaluation paradigm from data-centered to sensor-aware robustness, laying a foundation for adaptive, self-tuning perception systems capable of operating robustly in uncertain real-world environments.
BibTeX
@misc{han2025senseshift,
  author = {Yegyu Han and Taegyoon Yoon and Dayeon Woo and Sojeong Kim and Hyung-Sin Kim},
  title = {{SenseShift6D: Multimodal RGB-D Benchmarking for Robust 6D Pose Estimation across Environment and Sensor Variations}},
  year = {2025},
  eprint = {2507.05751},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.05751},
}
Figure