Abstract: This paper proposes a method for learning continuous control policies for active landmark localization and exploration using an information-theoretic cost. We consider a mobile robot detecting landmarks within a limited sensing range, and tackle the problem of learning a control policy that maximizes the mutual information between the landmark states and the sensor observations. We employ a Kalman filter to convert the partially observable problem in the landmark state to Markov decision process (MDP), a differentiable field of view to shape the reward, and an attention-based neural network to represent the control policy. The approach is further unified with active volumetric mapping to promote exploration in addition to landmark localization. The performance is demonstrated in several simulated landmark localization tasks in comparison with benchmark methods.
Introduction
In traditional robotics, mapping is often a passive process where a robot follows a pre-defined path and sensors record data. Active perception shifts this paradigm: the robot autonomously chooses its movements to maximize the information gained about its environment. This is critical for tasks like landmark localization where sensor range is limited.
However, active perception faces two significant challenges. First, generalization: many existing methods rely on open-loop optimization for specific environments, requiring re-calculation if landmarks move. Second, the discontinuous reward problem: sensors with a limited Field of View (FoV) create a "step function" in reward signals (a landmark is either in or out of view). This lack of gradient information makes it difficult for standard deep reinforcement learning agents to learn how to approach targets they cannot yet see.
Methodology
The proposed framework addresses these issues by mapping the robot's belief state directly to continuous velocity commands using a closed-loop control policy.
Key Components:
- Kalman Filter State: To manage the partially observable nature of the environment, a Kalman Filter maintains a posterior distribution (mean and covariance) of landmark positions, effectively converting the problem into a Markov Decision Process (MDP).
- Differentiable FoV: To solve the "jumpy" reward problem, the authors use a probit function based on the signed distance to the FoV boundary. This smooths the reward signal, providing the agent with feedback even when a landmark is just outside its sight.
- Attention-Based Architecture: Instead of a standard multi-layer perceptron, the policy uses an attention mechanism. This allows the robot to handle a variable number of landmarks and prioritize those with the highest uncertainty (lowest information).
- Information-Theoretic Reward: The reward is defined by the change in Mutual Information (the log-determinant of the information matrix), mathematically representing the reduction in environmental uncertainty.
Experimental Results
The approach was evaluated in custom 2D simulations and a high-fidelity 3D Unity environment.
Performance Highlights:
- Superior Accuracy: In an 8-landmark scenario, the proposed PPO-att method achieved significantly higher cumulative rewards (~38.25) compared to open-loop optimizers (~29.18) and standard MLP policies (~16.36).
- Generalization and Robustness: Unlike optimization-based methods, the learned policy generalizes to new landmark configurations without re-training. It also remains robust to Gaussian noise in robot dynamics.
- Active Exploration: The model successfully balances landmark localization with volumetric mapping, allowing the robot to discover new areas while refining its knowledge of known objects.
Conclusion and Future Work
By combining differentiable sensor models with attention-based reinforcement learning, this framework enables robots to perform complex active perception tasks in continuous environments. While the current model assumes perfect data association and clear lines of sight, future work will focus on handling occlusions and deploying the system on physical hardware to test against real-world sensor noise and environmental constraints.
Summary updated