Introduction
Deep learning models typically rely on massive, pre-annotated datasets for training. However, in robotics, a drone may encounter a "zero-day" task—such as tracking a specific vehicle not present in its original training set. In such scenarios, stopping a mission to collect data, label it, and retrain on a remote server is often impossible.
This paper introduces a "curiosity agent" designed to enable drones to train their own object detectors in real-time. By utilizing reinforcement learning (RL) in a simulated environment, the agent learns to navigate a drone to optimal vantage points and strategically request human input. This approach balances the need for high-accuracy training with physical constraints and the limited patience of a human operator.
System Architecture
The proposed system consists of two primary components: a Trainee (the object detector being trained) and a Curiosity Agent (the RL-based navigator).
- The Curiosity Agent: Built using a Deep Q-Network (DQN) with an Long Short-Term Memory (LSTM) layer for temporal memory, the agent processes the drone's current view, the detector's output, and the drone's position.
- Action Space: The agent operates in a 2D planar space, allowing it to move left/right (orbiting the target) and forward/backward (adjusting distance). It can also choose to remain stationary to focus on training.
- The Reward Function: The core innovation is a weighted reward function that considers:
- Incremental Training Benefit (ITB): The improvement in the model's precision after a training step.
- Annoyance Penalty: A negative reward for requesting human ground truth, acting as a "knob" for human-robot collaboration.
- Time Penalty: A penalty based on elapsed time, accounting for airspeed and processing latency.
Procedural Simulation
Training RL agents in the real world is both slow and risky. Consequently, the authors utilized a procedurally generated 3D simulation built in Unity with the ML-Agents API.
The environment generates randomized scenes with seven subject types (e.g., cars, planes, ships). To increase difficulty, the simulation includes "confusors"—objects that resemble the target but differ in color—and various obstructions to mimic real-world clutter. The authors emphasize that variety in simulation is more critical than visual realism for successful sim-to-real transfer.
Key Insights and Results
Hardware-Aware AI
The research revealed that an agent's strategy must adapt to its physical "body." For instance, a drone with a slow flight speed but a fast GPU should prioritize multiple training rounds from a single location. Conversely, a fast drone with a slow GPU should move more frequently to find the single most informative view before processing. Agents trained with specific physical parameters consistently outperformed mismatched agents, proving that robotics RL must incorporate platform-specific time costs.
The Annoyance Tradeoff
By adjusting the annoyance parameter ($w_n$), the agent can be tuned to be either a "high-maintenance" student that learns rapidly or a "self-sufficient" student that minimizes interruptions. Increasing this parameter resulted in an 8x improvement in training benefit per human interaction, though it led to a roughly 9% decrease in overall time efficiency as the agent spent more time searching for the "perfect" shot.
Limitations and Future Work
Currently, the drone is limited to 2D movement at a fixed altitude. Future iterations could expand this to 3D movement, including altitude and pitch adjustments. Additionally, while simulation variety helps, the system requires further validation in unpredictable real-world weather and lighting conditions. The authors also suggest that the "annoyance parameter" could eventually be adjusted dynamically by the human operator based on their real-time workload.
Summary updated