TL;DR
Cleanly-DINO is a framework that allows robots to learn visual task spaces—like identifying what is "dirty" versus "clean"—from as few as five demonstrations. By projecting high-level DINOv2 features into a specialized task-progression subspace, the system enables robots to localize work-in-progress areas and execute cleaning tasks with 83% of human-level proficiency.
What Problem Does This Paper Address?
In robotics, most manipulation research focuses on rigid-body tasks: moving a block from point A to point B. These tasks are easily defined by "poses" (position and orientation in 3D space). However, many essential tasks, like cleaning a whiteboard or pressure-washing a wall, aren't about moving objects—they are about changing the visual state of a surface.
The challenge is that "cleanliness" is subjective and visually diverse. A dirty window looks different from a dirty plate. Traditional methods often rely on massive datasets or temporal labels (knowing exactly when a frame occurs in a video). The authors sought a way to teach a robot to recognize task progress across different domains using only a handful of examples, without needing dense temporal supervision or complex 3D modeling.
Methods
The authors introduce Cleanly-DINO, which builds upon the pre-trained DINOv2 Vision Transformer (ViT). DINOv2 provides rich, patch-level semantic features, but these features are general-purpose. Cleanly-DINO "refines" them for specific tasks through three key steps:
- Self-Supervised Projection: The model learns a lightweight projection layer ($g$) that maps high-dimensional ViT features into a lower-dimensional "task space." It uses a contrastive loss to pull "initial state" (dirty) patches together and push them away from "final state" (clean) patches.
- Trajectory-Aligned Abstention: This is a clever filtering mechanism. In a video of someone wiping a table, many patches change because of shadows or the person's arm moving, not because the table is getting cleaner. The model identifies a "task axis" (the direction of change from dirty to clean) and ignores (abstains from) patches that don't move along that specific trajectory.
- Progression Metric: Once the task space is defined, the model calculates a scalar value (0 to 1) for every patch in an image, representing how far it has progressed toward the "clean" state. This creates a "heat map" of where the robot still needs to work.
Datasets
The researchers evaluated the model on a custom dataset of six visually distinct cleaning tasks:
- Wiping a blackboard
- Loading/cleaning a dishwasher
- Clearing leaves
- Pressure washing a wall
- Wiping a tablecloth
- Cleaning a whiteboard
The dataset consists of 53 demonstrations (a mix of self-recorded and internet videos) with 490 hand-labeled frames for evaluation. For the real-world robot experiments, they used common condiments (ketchup, mustard, BBQ sauce) on various surfaces.
Key Insights and Ideas
- The "Abstention" Filter is Critical: Simply looking at feature changes isn't enough because ViT patches influence each other (attention). By filtering out patches that don't align with the primary "cleaning" trajectory, the model significantly reduces noise from moving arms or changing lighting.
- Sample Efficiency: The model reaches peak performance with $\le 5$ demonstrations. This makes it viable for "in-the-field" teaching where a user shows a robot a task a few times.
- Generalization vs. Specialization: While the model transfers better across domains than raw DINOv2, the authors found that a single in-domain demonstration is often more valuable than many out-of-domain ones. However, training on "foreign" tasks (e.g., learning about whiteboards to help clean a dishwasher) still provides a measurable boost.
- Robustness in RL: When used as an observation space for Reinforcement Learning, Cleanly-DINO-based policies were 50% more robust to visual changes (like changing the table's color or pattern) compared to using raw DINOv2 features.
Key Results
- Localization Accuracy: Cleanly-DINO achieved an AUROC of 0.763–0.771 in identifying dirty vs. clean patches, outperforming temporal baselines like Time-Contrastive Networks (TCN) and raw DINOv2 (0.721).
- Real-World Performance: In robotic wiping tests, the system achieved a success rating of 3.64/5, which the authors calculate as roughly 83% of the "human ceiling" (4.39/5).
- Visual Transfer: The model successfully generalized to different table patterns (stripes, checkers) and wooden surfaces, though it struggled with highly irregular, high-contrast patterns like dark floral tablecloths.
Limitations and Future Work
- Static Thresholds: The current system uses a constant threshold to decide when a task is "done." In reality, the "clean" signal varies by scene. Future work aims to dynamically estimate the "clean" state for the current context.
- Complex Patterns: The model can be "fooled" by complex visual textures that look like stains (e.g., the floral tablecloth mentioned above).
- Beyond Cleaning: The authors suggest extending this to other "visual state" tasks, such as painting, where the robot might need to track progress across multiple sub-axes (e.g., different colors).
- The "Impossible" Problem: The model currently doesn't distinguish between a task that is "not yet done" and a task that is "impossible to complete" (e.g., a permanent stain).
For more details, see the full paper: "Cleanly-DINO: Efficiently Learning Visual Task Spaces for Robotics" presented at the COMPOGEN Workshop at CoRL 2026.
Summary updated