CoCap: Coordinated motion Capture for multi-actor scenes in outdoor environments

Aditya Rauniyar, Micah Corah, Sebastian Scherer

TL;DR

CoCap is a multi-drone coordination framework designed to capture high-quality motion data of multiple actors in cluttered outdoor environments. By adapting Conflict-Based Search (CBS) to viewpoint planning, it allows a team of UAVs to "negotiate" their positions, resolving collisions and occlusions to ensure diverse, non-redundant coverage that outperforms traditional greedy planning.

What Problem Does This Paper Address?

Motion capture (mocap) is traditionally a "studio" technology, relying on controlled indoor environments and fixed camera arrays. Moving mocap into the "wild"—outdoor settings with trees, buildings, and moving obstacles—is a major goal for robotics and computer graphics.

While single-drone systems can follow a single actor, multi-actor scenes are significantly harder. When multiple drones try to film multiple people, they often suffer from egocentric behavior: each drone tries to get the "best" shot for itself, leading to redundant views, occlusions where one drone blocks another, or even mid-air collisions. Existing "sequential" planners (where Drone 1 plans, then Drone 2 plans around it) often lead to deadlocks or poor coverage for the drones lower in the priority list.

Methods

The authors introduce CoCap (Coordinated motion Capture), which treats multi-view planning as a constrained optimization problem. The core components include:

  • Conflict-Based Search (CBS) for Perception: Originally designed for multi-agent pathfinding, CBS is adapted here to manage perception. It uses a two-level search: a high-level "constraint tree" that identifies conflicts (like two drones wanting the same spot or filming the same angle) and a low-level solver that re-plans individual paths to satisfy those constraints.
  • Submodular Reward Function: The system optimizes for "pixel density"—essentially how much detail of an actor's face/body is captured. Because the reward is submodular, the system naturally understands diminishing returns: if Drone A is already capturing an actor's face, Drone B gets more "reward" by moving to a different angle rather than duplicating the shot.
  • Single-Agent View Search: To make the system viable for real-time use, the authors developed a heuristic search based on the A* algorithm. This replaces computationally expensive Value Iteration, allowing drones to find near-optimal viewpoints in seconds rather than hours.

Datasets

The study does not use a standard static dataset. Instead, it utilizes a custom-built simulation environment designed to stress-test coordination:

  • Corridor Scenario: Two actors moving through a narrow passage with two drones. This tests the ability of drones to "get out of each other's way" while maintaining sightlines.
  • Bottleneck Scenario: Four actors with intersecting paths in a confined region, monitored by four drones. This tests high-density coordination and complex occlusion management.

Key Insights and Ideas

  • Negotiating Pixels: The most significant insight is that multi-drone mocap is a negotiation. By using CBS, the drones don't just avoid hitting each other; they effectively negotiate who covers which "pixel" of the actors to maximize the total information gathered by the team.
  • Breaking the Priority Chain: Unlike sequential planning—where the last drone often gets "stuck" with the worst views—CoCap's tree-based approach allows the system to reconsider the paths of earlier drones to find a globally better solution for the whole team.
  • Efficiency vs. Optimality: The authors demonstrate that while Markov Decision Process (MDP) solvers provide the "best" theoretical paths, a heuristic-based search can achieve nearly the same reward with a 1,400x speedup, making it feasible for live outdoor deployments.

Key Results

  • Higher Total Reward: In the "Corridor" test, CoCap (MDP) achieved a total scaled reward of 4662, significantly higher than the 4127 achieved by standard sequential planning.
  • Closing the Gap: CoCap's performance closely approached the "ideal" scenario (planning without any collision constraints), proving that coordination can mitigate the costs of safety constraints.
  • Real-Time Capability: The proposed "CoCap (Search)" method reduced computation time from 3981 seconds (over an hour) to just 2.7 seconds for the same scenario, with only a minor dip in total reward.

Limitations and Future Work

  • 2.5D Constraints: The current system uses a 2.5D height map, meaning it cannot easily reason about "overhangs" like bridges, tunnels, or flying under tree canopies.
  • Simplified Actor Models: Actors are represented as cuboids. Real-world human geometry is much more complex, and self-occlusions (e.g., an arm blocking a torso) aren't yet fully modeled.
  • Centralized Planning: The current version relies on a centralized planner. In real-world outdoor settings, maintaining a perfect high-bandwidth communication link between all drones and a central server can be challenging.
  • Heuristic Limitations: The search heuristic currently lacks a "long-term" exploration strategy, which might cause it to miss high-reward viewpoints that only become visible much later in a trajectory.

Summary updated

Abstract

Motion capture has become increasingly important, not only in computer animation but also in emerging fields like the virtual reality, bioinformatics, and humanoid training. Capturing outdoor environments offers extended horizon scenes but introduces challenges with occlusions and obstacles. Recent approaches using multi-drone systems to capture multiple actor scenes often fail to account for multi-view consistency and reasoning across cameras in cluttered environments. Coordinated motion Capture (CoCap), inspired by Conflict-Based Search (CBS), addresses this issue by coordinating view planning to ensure multi-view reasoning during conflicts. In scenarios with high occlusions and obstacles, where the likelihood of inter-robot collisions increases, CoCap demonstrates performance that approaches the ideal outcomes of unconstrained planning, outperforming existing sequential planning methods. Additionally, CoCap offers a single-robot view search approach for real-time applications in dense environments.
BibTeX
@misc{rauniyar2024cocap,
  author = {Aditya Rauniyar and Micah Corah and Sebastian Scherer},
  title = {{CoCap: Coordinated motion Capture for multi-actor scenes in outdoor environments}},
  year = {2024},
  eprint = {2412.20695},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.20695},
}
Figure