TL;DR
SPARC is an automated framework that generates high-quality spatial annotations (such as bounding boxes and trajectories) from robot demonstrations and assigns a well-calibrated reliability score to each. By leveraging physical interaction cues like gripper proximity and phase-aligned motion rather than just raw detector confidence, SPARC filters out noisy data while retaining three times more usable training samples than standard baselines.
What Problem Does This Paper Address?
Training modern Vision-Language-Action (VLA) models and embodied foundation models requires massive amounts of data annotated with spatial information (e.g., what object is being moved, where it starts, and where it goes). Manually labeling thousands of hours of video is prohibitively expensive, while simulation data struggles to cross the reality gap to the real world.
Existing automated pipelines try to solve this by chaining object detectors with trackers, but they have a major flaw: standard object detectors are poorly calibrated for correctness in cluttered, real-world robotic environments. A detector might confidently lock onto the wrong object or the robot's own gripper. This forces a harsh trade-off: either accept highly noisy training data or set a high confidence threshold that discards the vast majority of useful data.
Methods
SPARC (Spatial Annotations from Robot Demonstrations with Reliability Calibration) introduces an interaction-aware approach that uses the spatio-temporal structure of robot tasks to automatically label data and score its reliability. The pipeline operates in three stages:
- Subtask Decomposition & Phase Detection: Uses the robot's proprioceptive gripper signals to segment a long demonstration into grasp, interact, and release phases. A large language model (Qwen) then maps these phases to the language instructions to understand what action is happening.
- Candidate Proposal and Tracking: Generates object proposals at a keyframe using an open-vocabulary detector (LLMDet) with a low threshold to maximize recall. These proposals are segmented (SAM 2), tracked across time (AllTracker), and lifted into 3D space (MoGe-2).
- Reliability Scoring: Instead of relying on detector confidence, SPARC computes a Composite Annotation Reliability Score based on physical interaction evidence:
- Phase-aware motion: Checks if the object moves primarily when the gripper is closed.
- 3D gripper proximity: Measures how close the object stays to the gripper during manipulation.
- Robot-overlap filter: Applies a soft penalty if the tracked object overlaps heavily with the robot's own geometry, preventing the pipeline from mistaking the gripper for the manipulated object.
Datasets
- IA-Bench (Interaction-Aware Bench): A new benchmark introduced in this paper consisting of 1,748 human-annotated start and target locations of manipulated objects. It spans 12 diverse robot embodiments and settings (including bimanual setups) sourced from AgiBotWorld, DROID, BridgeData, and Open X-Embodiment.
- SPARC Dataset: The authors used SPARC to automatically annotate 280k robot trajectories, filtering them down to 167k high-quality trajectories to build a massive VQA dataset containing 511k samples for downstream VLM training.
Key Insights and Ideas
- Physics over Pixels: Detector confidence measures recognition confidence, not interaction truth. By switching the focus to physical cues (did it move when grabbed? is it next to the hand?), SPARC bypasses the visual ambiguity that trips up standard vision models.
- Soft Penalties Beat Hard Gating: Instead of completely discarding objects that overlap with the robot gripper (which happens constantly during manipulation), SPARC uses a continuous quadratic penalty attenuated by 3D depth separation. This preserves contact-rich interactions while filtering out pure gripper tracking errors.
- Calibrated Data Scaling: The reliability score provides a highly predictable knob for data curation. Unlike detector confidence, which admits more noise as you scale up the dataset, SPARC's score allows researchers to scale up dataset sizes safely without introducing catastrophic label noise.
Key Results
- Annotation Accuracy: On IA-Bench, SPARC achieves 80.2% object localization accuracy, significantly outperforming the detection-only baseline (58.1%).
- Data Retention: At a strict 95% precision target, SPARC retains 58% of samples, whereas the detector-confidence baseline drops to 0% usable data and the strongest trajectory baseline only retains 20%.
- Downstream Performance: VLMs fine-tuned on SPARC-generated data outperform models trained on human-annotated datasets (like EO-1.5M) across multiple spatial reasoning and pointing benchmarks (e.g., Where2Place, VABench).
- Real-World Policy Success: In a cluttered real-world tabletop environment, a VLA policy trained with SPARC annotations achieved a 31% success rate, more than doubling the detection-annotated policy (12%) and tripling the baseline without spatial reasoning (9%).
Limitations and Future Work
- Foundation Model Dependencies: SPARC is bounded by its underlying components; detector errors can limit part-level annotations, and tracking can fail due to identity switches among visually identical objects.
- Visual/Physical Constraints: The 3D lifting mechanism struggles with transparent surfaces, and the tracking quality degrades under strong camera motions (it assumes a mostly static camera).
- Grasp Dependency: The pipeline relies heavily on clear, distinct grasp phases, making it less reliable for tasks without explicit picking/placing actions (e.g., pushing or purely rotational movements).
- Future Directions: The authors suggest expanding the pipeline to generate more diverse VQA pairs and task instructions from the same underlying spatial annotations to boost scene-level reasoning.
Summary updated