TagTeam: Towards wearable-assisted, implicit guidance for human-drone teams

Kasthuri Jayarajah, Aryya Gangopadhyay, Nicholas R. Waytowich

Introduction

In high-stakes environments such as indoor search-and-rescue or tactical reconnaissance, small-scale drones (nano-quadcopters) offer significant advantages due to their ability to navigate tight spaces and operate safely in close proximity to humans. However, these platforms face a dual challenge: severe Size, Weight, and Power (SWAP) constraints that limit onboard autonomy, and a "coordination gap" in human-robot interaction. Traditional "explicit" control—where a human must provide manual inputs or voice commands—imposes a high cognitive workload, often distracting the operator from their primary mission.

To bridge this gap, the TagTeam framework introduces a paradigm of implicit guidance. By leveraging data from a human’s wearable sensors, the system allows a drone to infer human intent and state, enabling the team to function as a synchronized unit without requiring direct commands.

The Coordination Gap and Implicit Guidance

The core philosophy of TagTeam is to move away from the drone as a tool to be operated and toward the drone as an autonomous partner. Current nano-drones lack the computational overhead for complex environmental mapping or independent decision-making. Simultaneously, human operators in the field cannot afford to split their attention between their surroundings and a drone controller.

Implicit guidance solves this by treating the human's wearable device—such as smart glasses or "earables"—as a remote sensor suite for the drone. By monitoring the human’s Area of Attention (AoA) and identifying their Area of Uncertainty (AoU), the drone can autonomously decide where to provide assistance.

System Architecture

The researchers implemented a three-tier architecture to manage the computational demands of the system:

  1. The Wearable (Microsoft HoloLens 2): Serves as the primary sensor hub. It tracks the human’s head position, gaze, and spatial orientation to determine the user's focus in real-time.
  2. The Drone (Bitcraze Crazyflie 2.1): A 27-gram nano-drone equipped with an "AI deck" for image streaming and a "flow deck" for stable flight. Its role is to act as the mobile sensor platform.
  3. The Middleware (Nvidia Jetson Nano): Acts as the system's brain. Because the drone cannot run heavy AI models, the Jetson Nano receives positional data via MQTT, performs coordinate transformations between the human and drone frames, and runs an SSD MobileNet V2 detector on the drone’s video feed to identify potential threats or objects of interest.

Spatial alignment is maintained through dead-reckoning. When the drone identifies an object in a human's blind spot, it provides feedback via Spatial Sound through the HoloLens, creating an intuitive alert that sounds as if it is originating from the direction of the detected object.

Operational Modes: Complementary vs. Supplementary

TagTeam defines two primary modes of implicit cooperation:

  • Complementary Scanning: The drone monitors what the human cannot see. By patrolling the human’s blind spots (Areas of Uncertainty), the drone extends the user's situational awareness without requiring them to turn their head.
  • Supplementary Scanning: The drone provides a "second look" at what the human is looking at. This mode is used to provide higher-resolution scans or alternative angles of an object the human is currently investigating, helping to resolve ambiguity in complex scenes.

Experimental Results

The preliminary evaluation of TagTeam focused on motion awareness—the system's ability to synchronize the drone's movements with the human's trajectory.

Using a custom experimental setup in a controlled indoor environment, the authors recorded and annotated movement sequences to establish ground truth. By applying Dynamic Time Warping (DTW) to compare the trajectories of the human and the drone, the system achieved a synchronization score of ~0.92 (where 1.0 represents perfect alignment). This high degree of synchronization demonstrates that wearable data is a viable proxy for guiding drone movement in real-time. Furthermore, the functional prototype successfully demonstrated a closed-loop alert system using directional audio feedback based on real-time object detection.

Limitations and Future Directions

While TagTeam represents a significant step toward implicit human-drone teaming, several challenges remain:

  • Scaling to Swarms: The current prototype is a 1:1 pairing. Expanding this to multi-human, multi-drone teams (e.g., a tactical unit) will require sophisticated data prioritization to manage conflicting intents and avoid aerial collisions.
  • Human Factors: The psychological impact of an autonomous drone hovering near a human's head is not yet fully understood. Establishing trust and ensuring the drone's movements are predictable to the human partner is critical for adoption.
  • Advanced Sensor Fusion: Future iterations aim to integrate more nuanced physiological signals, such as EEG (brain activity) or EOG (eye movement), to detect "affirmative or negative thinking." This would allow the drone to learn from the human's internal reactions to its assistance, further refining the implicit feedback loop.

Conclusion

TagTeam demonstrates that by repurposing wearable technology as a guidance system, we can overcome the SWAP constraints of nano-drones and the cognitive limitations of explicit control. This framework paves the way for a future where drones act as seamless extensions of human perception, providing assistance that is felt rather than commanded.

Summary updated

BibTeX
@inproceedings{jayarajah2022tagteam,
  author = {Kasthuri Jayarajah and Aryya Gangopadhyay and Nicholas R. Waytowich},
  title = {{TagTeam: Towards wearable-assisted, implicit guidance for human-drone teams}},
  booktitle = {SmartWear@MobiCom},
  year = {2022},
  doi = {10.1145/3556560.3560715},
}
Figure