Abstract
Existing 6D pose estimation benchmarks often overlook the impact of varying environmental lighting and camera sensor configurations. We present SenseShift6D, a comprehensive RGB-D dataset capturing thousands of physical sensor and lighting permutations. Our evaluation reveals that state-of-the-art models significantly degrade under hardware-level shifts. However, by employing test-time sensor control—dynamically adjusting exposure, gain, and depth modes—performance can be improved by up to 19.5% without additional training.
Introduction
Estimating the 6D pose of objects is a critical task for robotics, autonomous vehicles, and augmented reality. While modern AI models achieve high accuracy on standard benchmarks like LineMOD or YCB-V, these datasets are typically captured under strictly controlled, static conditions. In real-world scenarios, ambient lighting changes constantly, prompting cameras to use auto-exposure and varying gain, while depth sensors switch capture modes.
These hardware adjustments introduce complex, non-linear physical artifacts—such as photon-shot noise, color shifts, and structured depth errors—that synthetic data augmentation cannot accurately replicate. There is a significant gap in the research community's ability to measure how 6D pose models degrade under these hardware-level shifts and whether robustness can be improved by tuning camera settings on the fly.
The SenseShift6D Dataset
To address this gap, we developed a highly controlled physical testbed using a darkroom, programmable smart bulbs, a motorized turntable, and an Intel RealSense D455 camera. We systematically captured objects under a massive grid of physical sensor and lighting configurations, resulting in the SenseShift6D dataset.
The dataset includes five household objects (spray bottle, Pringles, tincase, sandwich replica, and mouse) captured across 13 RGB exposures, 9 RGB gains, auto-exposure, 4 depth-capture modes, and 5 illumination levels. This yields 166.4k RGB images and 16.7k depth images, providing 1,380 unique sensor-lighting permutations per object pose.
Methods and Evaluation
We evaluated several state-of-the-art 6D pose estimation models, including generalizable "foundation" models (GigaPose, SAM-6D, FoundationPose) and instance-level models (ZebraPose, GDRNPP, HiPose). To test the potential of test-time sensor control, we compared model performance under five regimes: factory defaults (Baseline), Oracle-fixed settings, and Oracle-dynamic settings (choosing the best setting for each specific scene).
Key Results and Insights
Our experiments yielded several critical insights into the relationship between hardware and AI performance:
- Massive Gains Without Retraining: For pretrained generalizable models, using dynamic, per-scene sensor control (Oracle-Dynamic) improved performance by a staggering +19.5 percentage points (pp) over the baseline.
- Multimodal Synergy: Tuning just the RGB sensor or just the depth sensor helps (+12.3 pp and +10.5 pp respectively), but tuning both together yields compounding, synergistic benefits.
- Hardware as a Tunable Layer: Mitigating domain shift at inference time by physically optimizing the camera sensor can be more effective—and compute-efficient—than forcing a model to learn every possible variation through data augmentation.
- Increased Stability: Dynamic sensor control reduced the performance gap between the best- and worst-performing objects from 40.0 pp down to 13.6 pp, significantly increasing model reliability.
Conclusion
SenseShift6D demonstrates that factory default camera settings, often tuned for human perception, are suboptimal for computer vision tasks. By treating the camera sensor as a tunable layer of the inference pipeline, we can achieve significant performance boosts. Future work will expand the dataset to include multi-object clutter and directional lighting, with the ultimate goal of developing real-time, adaptive sensor-control algorithms that can automatically optimize camera settings in the wild.
Summary updated