Traditional tracking systems rely heavily on line-of-sight (LOS) observations, such as camera frames, depth maps, or laser scans. While these modalities offer high spatial precision, they share a fundamental vulnerability: they fail catastrophically when the target is occluded, shadowed, or rotated out of view.
To build a tracking system that survives real-world environments, researchers have turned to complementary sensor fusion. The core intuition is simple: fuse high-precision line-of-sight sensors with non-line-of-sight (NLOS) or proprioceptive sensors. Because different sensor modalities possess entirely independent failure modes, one sensor can act as a temporal or spatial bridge precisely when another goes blind.
The Vulnerability of Single-Modality Tracking
Early tracking algorithms attempted to mitigate occlusions by combining different visual features. For instance, Birchfield in Elliptical Head Tracking Using Intensity Gradients and Color Histograms demonstrated that fusing shape-independent color data with shape-dependent boundary information allowed a tracker to maintain a lock on a human head during 360-degree out-of-plane rotations. In robotics, Gu in Tactic-Based Motion Modeling and Multi-Sensor Tracking utilized short-range proximity sensors to complement high-bandwidth vision, providing a reliable trigger for state transitions when the primary camera lost visibility.
However, when tracking complex 3D trajectories, articulated bodies, or objects in dynamic environments, purely visual adjustments reach their physical limits. If light from the target cannot reach the sensor, the tracking state must be maintained through alternative physical properties.
Inertial Dead Reckoning as a Temporal Bridge
The most common partner to visual tracking is inertial sensing. Inertial Measurement Units (IMUs) capture high-frequency acceleration and angular velocity. Crucially, IMUs do not require a line of sight; they operate entirely on internal physics and are immune to visual occlusions or low-light conditions. However, IMUs suffer from integration drift, meaning their errors accumulate rapidly over time.
Fusing these two modalities creates a symbiotic relationship: the visual sensor constantly calibrates the IMU and resets its drift, while the IMU provides a high-frequency motion prior that bridges gaps during visual dropouts.
[Visual Data Available] --> Calibrates IMU & Provides Absolute Position
│
▼
[Visual Occlusion Hits] --> System Switches to IMU Dead Reckoning (No Drift Accumulation Yet)
│
▼
[Visual Data Restored] --> Resets Accumulated IMU Drift & Resumes Absolute Tracking
This mechanism is highly effective across scales:
- Object and Aerial Tracking: Lee et al. in Visual-Inertial Telepresence for Aerial Manipulation fused fiducial marker tracking with high-frequency visual-inertial odometry (VIO), allowing an aerial robot to estimate an object's relative position via dead reckoning when the camera's view was temporarily shadowed. Similarly, Loianno et al. in VIPose: Real-time Visual-Inertial 6D Object Pose Tracking leveraged camera ego-motion and inertial data to provide a robust motion prior, maintaining 6D object tracking through severe visual blockages.
- Urban Infrastructure: This architecture also applies to macro-scale cooperative tracking. Reitberger et al. in Cooperative Tracking of Cyclists Based on Smart Devices and Infrastructure combined static, infrastructure-based stereo cameras with inertial data from a cyclist’s smartphone. When the cyclist disappeared behind an obstacle at an urban intersection, the tracking filter seamlessly switched to dead reckoning using the phone's velocity and yaw rate.
Non-Line-of-Sight Modalities: Radio and Touch
When targets must be tracked through solid obstacles or close-range physical interactions, researchers introduce alternative non-line-of-sight (NLOS) modalities like radio-frequency ranging or tactile feedback.
Radio-Frequency Fusion
Radio signals can pass directly through many physical obstructions. By fusing Ultra-Wideband (UWB) radio ranging or Angle-of-Arrival (AOA) signals with inertial data, systems can maintain continuous tracking without any visual input.
Bi et al. in Accurate 3D Localization for MAV Swarms by UWB and IMU Fusion and Holz et al. in Accurately Tracking Relative Positions of Moving Trackers based on UWB Ranging and Inertial Sensing without Anchors demonstrated that anchorless UWB ranging paired with IMU dead reckoning can bypass signal occlusions. When non-line-of-sight conditions are detected, the system dynamically weights the inertial data more heavily, bridging gaps in radio connectivity. For mobile robot navigation, Jin et al. in A Robust Autonomous Following Method for Mobile Robots in Dynamic Environments paired a 2D laser range finder with an AOA radio tag. The radio tag acts as a re-identification key to maintain target lock during complete visual occlusion, while the laser provides fine-grained geometric corrections when the line of sight is clear.
Visuotactile and Wearable Sensing
For close-up human or robotic manipulation, objects are frequently occluded by the hands manipulating them. Xiao et al. in Human-Robot Interaction by Understanding Upper Body Gestures solved this by combining external camera tracking with a wearable data glove, ensuring fine-grained hand posture data remains continuous even when objects block the camera's view.
In robotic manipulation, Suresh et al. introduced NeuralFeels with neural fields: Visuotactile perception for in-hand manipulation, which fuses vision with tactile sensing. When a robot grips an object, the parts of the object hidden by the fingers are tracked and reconstructed via touch, significantly outperforming vision-only methods.
In extreme scenarios, vision can be bypassed entirely. Wang et al. in Capturing the Unseen: Vision-Free Facial Motion Capture Using Inertial Measurement Units developed a facial motion capture system using micro-IMUs attached to the face. Because it relies entirely on inertial data, it tracks complex expressions in total darkness or during physical occlusions (such as eating).
Spatial Redundancy and Learned Fallbacks
When physical sensors cannot be attached directly to a target, tracking robustness must be engineered either through spatial redundancy or predictive modeling.
As outlined by Blasch et al. in Special Issue: Dynamic Data-Driven Applications Systems (DDDAS) Concepts in Signal Processing, integrating visual cameras with underlying depth and motion models allows a tracking system to emulate missing data when a target is temporarily hidden. Spatial redundancy achieves this by deploying multiple overlapping viewpoints. Wang et al. in Whole-Body Teleoperation Control of Dual-Arm Robot Using Sensor Fusion combined data from multiple depth cameras and motion controllers to eliminate single-sensor blind spots. For wider environments, Mateus et al. in Efficient and robust Pedestrian Detection using Deep Learning for Human-Aware Navigation integrated onboard robot sensors with external static infrastructure cameras, passing pedestrian tracks between perspectives to maintain continuity.
┌─────────────────┐ ┌─────────────────┐
│ Camera Array A │ │ Object IMU Data │
│ (Visual Mask) │ │ (Rotation Only) │
└────────┬────────┘ └────────┬────────┘
│ │
└───────────┬─────────────┘
▼
┌───────────────────────────┐
│ Joint Optimization │
│ (Resolves Occluded Poses) │
└───────────────────────────┘
When tracking multiple interacting agents, Zhang et al. in HOI-M3: Capture Multiple Humans and Objects Interaction within Contextual Environment combined dense multi-camera arrays with object-mounted IMUs. By optimizing visual masks alongside inertial rotation data, the system resolves exact 3D poses even when objects are completely hidden from the cameras' line of sight.
Alternatively, learned models can act as an algorithmic safety net. Naik et al. in 3D-POP - An Automated Annotation Approach to Facilitate Markerless 2D-3D Tracking of Freely Moving Birds with Marker-Based Motion Capture demonstrated a hybrid strategy where a markerless tracking model is trained on high-fidelity marker-based motion capture data. If the physical markers on a subject become occluded from the motion-capture cameras, the system falls back to the learned markerless model to fill the data gaps, maintaining tracking continuity far more accurately than standard linear interpolation.
Key Foundational Work
Tactic-Based Motion Modeling and Multi-Sensor Tracking (2005) — contributed 1 concept elements to this direction.
Elliptical Head Tracking Using Intensity Gradients and Color Histograms (1998) — contributed 1 concept elements to this direction.