TL;DR
This paper introduces "Follow Anything" (FAn), a real-time robotic system that can detect, track, and follow any object specified by text, image, or click queries. By leveraging powerful foundation models, FAn offers unprecedented flexibility for object following in robotics, operating efficiently even on hardware with limited resources.
What Problem Does This Paper Address?
Traditional object detection and tracking systems for robots are often "closed-set," meaning they can only recognize and follow objects they were explicitly trained on. This severely limits their adaptability to new scenarios or objects. Furthermore, specifying the object of interest is often restricted to class labels, which can be unintuitive for users. The authors aim to bridge this gap by creating a system that can follow any object, specified in a flexible, multimodal way, and operate in real-time on resource-constrained robotic platforms.
Methods
FAn is built upon several key components:
- Foundation Models: The system heavily relies on pre-trained foundation models like CLIP [1] for multimodal understanding (text, image) and DINO [2] for powerful visual feature extraction. The Segment Anything Model (SAM) [3] is used for robust, class-agnostic instance segmentation.
- Open-Set, Multimodal Querying: Users can specify the object of interest using text descriptions ("a drone"), example images, or even by clicking on the object in a video frame. FAn then uses these queries to extract feature descriptors.
- Detection and Segmentation Pipeline:
- Segmentation: SAM generates initial segmentation masks for all potential objects in a frame.
- Feature Extraction: DINO (or CLIP) extracts pixel-wise feature descriptors for the entire frame.
- Region Aggregation: Per-pixel descriptors within each SAM mask are aggregated (e.g., via average pooling) to create a region-level descriptor.
- Similarity Matching: The query descriptor is compared to each region descriptor using cosine similarity. Regions exceeding a threshold are identified as the target object.
- Real-time Tracking: For tracking, the system uses efficient trackers like SiamMask [51] or DEAOT [49, 50].
- Re-detection Mechanism: To handle occlusions or temporary loss of the object, FAn stores historical DINO features of the tracked object. If tracking is lost, it can use these stored features to re-detect the object.
- Visual Servoing: A lightweight visual servoing controller is used to command the robot (a micro aerial vehicle in their experiments) to follow the detected and tracked object.
Datasets
The paper does not explicitly mention training datasets for the core FAn system, as it leverages pre-trained foundation models. For evaluation, they used:
- Custom Datasets: Experiments were conducted with a quadrotor following various objects like drones, RC cars, bricks, and whales in real-world scenarios.
- Cholec80 Dataset [64]: Used for evaluating mask quality, specifically for detecting and tracking a "grasper" tool during surgery.
Key Insights and Ideas
- Foundation Models as a Universal Toolkit: The paper demonstrates how powerful, general-purpose foundation models can be effectively repurposed for real-time robotic tasks, overcoming the limitations of traditional, task-specific models.
- Multimodal Queries Unlock Flexibility: Allowing users to specify objects via text, images, or clicks significantly enhances the usability and adaptability of robotic following systems.
- Efficient Real-time Operation: By optimizing foundation models (e.g., using FastSAM, quantization, tracing) and employing efficient tracking, FAn achieves real-time performance (6-20 FPS) on a commodity laptop with a lightweight GPU.
- Robust Re-detection is Crucial: The proposed feature-descriptor storing mechanism for re-detection is a practical solution for maintaining tracking continuity in dynamic environments.
- SAM + DINO Synergy: Combining SAM's precise segmentation with DINO's rich semantic features proves to be a powerful approach for accurate object detection.
Key Results
- Real-time Performance: FAn achieves a throughput of 6-20 frames per second on a laptop with a lightweight GPU.
- High Detection Accuracy: In experiments, FAn achieved high true positive detection rates for various objects like cars and whales (e.g., 1.0 for cars with SAM+DINO in single-query tests).
- Successful Object Following: The system was demonstrated to autonomously detect, track, and follow objects like drones, RC cars, and bricks in real-world scenarios using a micro aerial vehicle.
- Robustness to Occlusion: The re-detection mechanism successfully recovered tracking for objects that passed under "tunnels."
- Trajectory Comparison: In a 4-minute tracking experiment with a ground robot, the mean Euclidean distance between the quadrotor and the target object was as low as 11 cm using the SAM+DINO detector and a PID controller.
Limitations and Future Work
- Mask Quality vs. Speed: While DINO-SOLO is faster, its masks are less refined than those produced by SAM, which can impact tracking performance. Conversely, SAM can sometimes miss important object regions, leading to missed detections.
- Hardware Constraints: While FAn is designed for efficiency, achieving higher frame rates or processing more complex scenes might still require more powerful hardware.
- Future Work: The authors suggest further optimization of segmentation/detection performance, potentially through automatic compression methods. They also highlight the potential for integrating more sophisticated controllers and exploring more complex multimodal interactions.
Summary updated