SenseShift6D: Multimodal RGB-D Benchmarking for Robust 6D Pose Estimation across Environment and Sensor Variations

Yegyu Han, Taegyoon Yoon, Dayeon Woo, Sojeong Kim, Hyung-Sin KimSubmission to ICLR 2026 · Desk rejected

TL;DR

Current 6D object pose estimators are typically trained and evaluated on data captured under fixed camera settings and static lighting, causing them to falter when facing real-world sensor noise and illumination shifts. To solve this, the authors introduce SenseShift6D, the first RGB-D dataset physically sweeping exposure, gain, depth-sensor capture modes, and illumination across 795k+ scenes. Crucially, they show that dynamically tuning camera and depth-sensor parameters at test time boosts pose accuracy by up to 18 percentage points—without any model retraining.

What Problem Does This Paper Address?

Estimating the 6D pose (3D translation and 3D orientation) of objects is vital for robotics, augmented reality, and embodied AI. While state-of-the-art models excel on classic benchmarks like LM-O or YCB-V, these benchmarks share a major blind spot: they are collected under strictly controlled lighting with static camera parameters.

In practical deployment, ambient light changes, auto-exposure flickers, and depth sensors alter their internal modes. These hardware-level dynamics introduce complex, non-linear color artifacts, photon-shot noise, and structured depth dropouts that standard synthetic augmentations fail to replicate. Moreover, while recent work in image classification has started exploring "test-time sensor adaptation" (tuning sensor parameters dynamically instead of retraining large networks), the community lacked both a benchmark and an empirical evaluation of multimodal (RGB + Depth) sensor adaptation for 6D pose estimation.

Methods

Rather than designing a new neural network architecture, the authors establish an empirical framework and dataset to study sensor-aware robustness and test-time sensor control:

  1. Systematic Physical Parameter Sweeps: Using an Intel RealSense D455 camera mounted in a calibrated darkroom with programmable smart lighting, the authors physically capture every permutation of exposure, gain, depth mode, and ambient brightness for identical object poses on a motorized turntable.
  2. Evaluation Regimes: To quantify the potential of sensor control, they evaluate models under five operational regimes:
  3. Baseline: Factory defaults (Auto-exposure RGB, Default depth preset).
  4. Depth-Only Oracle: Dynamically picks the best depth preset per scene.
  5. RGB-Only Oracle: Dynamically selects the best RGB exposure and gain per scene.
  6. Oracle-Fixed: Finds the single best static RGB-D configuration globally across the dataset.
  7. Oracle-Dynamic: Jointly selects the optimal RGB and depth parameters on a per-scene basis (serving as an empirical performance upper bound).

Datasets

The authors introduce SenseShift6D, comprising:

  • Physical Sweeps: 13 RGB exposure times, 9 RGB gains, auto-exposure, 4 depth capture presets (Default, High Accuracy, High Density, Medium Density), and 5 brightness levels (5% to 100%).
  • Scale: 6 household objects (spray bottle, Pringles tube, tincase, sandwich replica, vertical mouse, rubber duck), yielding 198.8k RGB frames, 20.0k depth frames, and 795.2k paired RGB-D scenes with 1,380 permutations per pose.
  • Ground Truth: Calibrated via ChArUco boards and refined with the BOP toolkit, maintaining a tight ~2 mm error.
  • Data Splits: Organized into default and variable splits (Train-Def, Train-Var, Test-Def, Test-Var) ensuring no pose overlap between train and test distributions.

Key Insights and Ideas

  • Factory Camera Defaults Are Suboptimal for Models: Standard camera firmware tunes auto-exposure and tone-mapping for human visual perception, not machine perception. Single-modality oracle selection alone outperforms factory settings by over 11 percentage points.
  • Multimodal Synergy: Tuning RGB exposure/gain helps feature matching, while selecting the right stereo depth mode prevents edge bleeds and fill-rate collapse. Jointly adapting both RGB and depth yields an additional boost (+6.6 pp over RGB-only adaptation).
  • Dynamic Adaptation Beats Global Tuning: Finding a single "better" camera setting (Oracle-Fixed) only yields a negligible +1.2 pp bump and often harms specific objects. Parameter selection must be adaptive and per-scene.
  • Sensor Control Beats Brute-Force Data Scaling: For instance-level models, dynamically optimizing sensor parameters on a model trained only on default data outperformed models trained on heavily augmented data or expanded physical sweeps (Train-Var). Correcting sensor inputs at acquisition is strictly better than asking downstream neural networks to hallucinate through severe sensor degradation.

Key Results

  • Pretrained Generalizable Models (FoundationPose, SAM-6D, GigaPose):
  • Dynamic Multimodal Control: Achieved an average +18.0 percentage point (pp) gain in AUC@[0:0.1] over the default baseline across all models (reaching up to +21.4 pp on GigaPose).
  • Closing the Failure Gap: On challenging objects like the Pringles can and tincase, accuracy jumped by over 40 pp in several configurations.
  • Variance Reduction: Reduced the performance gap between the easiest and hardest objects from 42.5 pp down to 14.2 pp, while cutting scene-level performance variance by 8.3 pp standard deviation.
  • Instance-Level Models (ZebraPose, GDRNPP, HiPose):
  • Adapting RGB settings on Test-Var allowed baseline-trained models to exceed their original Test-Def scores.
  • HiPose achieved an overall AUC increase of +10.07 pp using dynamic multimodal oracle control compared to default settings.

Limitations and Future Work

  • Scene Complexity: Currently restricted to single-object tabletop settings with uniform, diffuse lighting. It does not account for complex multi-object clutter, heavy occlusions, or sharp directional shadows.
  • Oracle vs. Real-Time Policy: The paper evaluates an Oracle selection bound to reveal what is possible. It does not implement a lightweight, closed-loop policy network to predict optimal camera/depth parameters on the fly in real-time.
  • Future Directions: Expanding the benchmark to cluttered multi-object scenes and engineering real-time, self-tuning hardware controllers for deployment on robots and AR headsets.

Summary updated

Abstract

Recent advances on 6D object-pose estimation have achieved high performance on representative benchmarks such as LM-O, YCB-V, and T-Less. However, these datasets were captured under fixed illumination and camera settings, leaving the impact of real-world variations in illumination, exposure, gain or depth-sensor mode—and the potential of test-time sensor control to mitigate such variations—largely unexplored. To bridge this gap, we introduce SenseShift6D, the first RGB-D dataset that physically sweeps 13 RGB exposures, 9 RGB gains, auto-exposure, 4 depth-capture modes, and 5 illumination levels. For six common household objects (spray, pringles, tincase, sandwich, mouse, and duck), we acquire 198.8k RGB and 20.0k depth images, which can provide 1,380 unique sensor-lighting permutations per object pose. Experiments with state-of-the-art models on our dataset demonstrate that applying multimodal sensor control at test time yields substantial performance gains, achieving a 18.0 pp improvement on pretrained generalizable models. It also enhances robustness precisely where those models tend to fail. Moreover, even instance-level pose estimators, where train and test set share identical object and background, performance still varies under environmental and sensor change, demonstrating that test-time sensor control remains effective compared to costly expansions in the quantity and diversity of real-world training data, without any additional training. SenseShift6D extends the object pose evaluation paradigm from data-centered to sensor-aware robustness, laying a foundation for adaptive, self-tuning perception systems capable of operating robustly in uncertain real-world environments.
BibTeX
@misc{han2025senseshift,
  author = {Yegyu Han and Taegyoon Yoon and Dayeon Woo and Sojeong Kim and Hyung-Sin Kim},
  title = {{SenseShift6D: Multimodal RGB-D Benchmarking for Robust 6D Pose Estimation across Environment and Sensor Variations}},
  year = {2025},
  eprint = {2507.05751},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.05751},
}
Figure