Abstract
Manipulating deformable objects like clothing remains a significant challenge in robotics due to high degrees of freedom and self-occlusion. This paper presents a novel framework that leverages Virtual Reality (VR) to facilitate human-robot collaboration in cloth folding. By representing garments as graph-based skeletons rather than raw pixel data, the system allows human operators to define folding plans that generalize across different items within the same clothing category. Experimental results demonstrate that this topology-based approach enables high proposal accuracy and significantly reduces the manual effort required for robotic training.
Introduction
Robotic manipulation of deformable objects—such as t-shirts, trousers, or linens—is a notorious "hard problem." Unlike rigid objects, cloth can take an infinite number of shapes, and its parts frequently obscure one another during handling. Traditional AI models for these tasks require massive, manually labeled datasets, which are expensive to produce. Furthermore, models trained on specific visual features (like color or texture) often fail to generalize; a robot trained on a blue shirt may struggle to recognize the structure of a red one.
To address these challenges, we propose a framework that prioritizes the underlying topology of a garment over its surface appearance. By using a VR interface, we create an intuitive "propose-and-approve" loop where humans can teach robots complex folding sequences using simplified digital skeletons.
Methodology
The proposed framework consists of a two-stage process designed to bridge the gap between human intuition and robotic execution.
1. Skeleton Representation Extraction
The system begins by processing an RGB image of a garment. Through binarization and a thinning algorithm, the image is reduced to a 2D skeleton. This skeleton is then converted into a graph where nodes represent critical topological points (e.g., necklines, cuffs, or corners) and edges represent the connections between them. This abstraction strips away irrelevant data like color and texture, focusing purely on the garment's structure.
2. VR-Based Folding Plan Definition
Using an Oculus Quest 2 headset, a human operator enters a virtual environment containing the extracted skeleton. The operator defines a folding plan by selecting a sequence of "pick-and-place" nodes. To ensure precision, the framework allows users to adjust the robot's trajectory height by manipulating a middle-node between the pick and place coordinates.
Once a plan is established for a specific garment class, the system can apply it to new items. When a new garment is presented, the robot extracts its skeleton and "proposes" a folding action based on the stored template. The human operator remains in the loop to approve or refine the action before the physical Baxter research robot executes the movement.
Experimental Evaluation
Physical Automation Experiments
The framework was tested using a Baxter robot and a dataset of seven garments across three classes: short-sleeve tops, long-sleeve tops, and trousers. The goal was to determine how well a plan defined for one item could be applied to others in the same category.
In trials involving short-sleeve tops, the robot achieved a perfect proposal accuracy of 12 out of 12 trials for certain items. The system significantly reduced human workload; once a primary plan was defined, subsequent items required only minimal adjustments to node positions rather than a complete re-definition of the task.
Large-Scale Validation
To validate the robustness of skeleton-based representations, the researchers utilized a subset of the Deepfashion2 dataset, comprising 25,102 images across nine categories (e.g., skirts, vests, shorts). Using the Multi-View Graph Representation Learning (MVGRL) framework, the skeletons were used for unsupervised classification. The skeleton-based approach achieved a 90.7% Top-5 accuracy, proving that these lightweight graphs capture nearly as much class-discriminative information as full RGB images (52.4% vs 54.1% in direct comparison), while being far more efficient for robotic path planning.
Discussion and Key Insights
Topology Over Texture
The core insight of this research is that for tasks like folding, the topology of the garment—the spatial relationship between sleeves and the torso—is more critical than visual aesthetics. Skeletons provide a universal language that allows the robot to recognize that two structurally similar items should be handled in the same way, regardless of their visual differences.
The Propose-and-Approve Loop
By moving away from full autonomy (which is prone to failure in complex environments) and full manual teleoperation (which is labor-intensive), this framework finds a middle ground. The robot handles the mapping and spatial calculations, while the human provides high-level supervision and error correction.
Limitations and Future Work
While the framework shows high generalization potential, it is sensitive to "noise" in the skeleton extraction process. If a garment is laid out in a way that produces a significantly different skeleton, the proposed plan may fail. Additionally, the current system is limited to single-arm pick-and-place actions. Future iterations will focus on bimanual (two-handed) coordination and formal user studies to evaluate how quickly non-expert users can master the VR interface.
Summary updated