Introduction
Folding fabric is a significant challenge in robotics due to the high degrees of freedom and non-rigid dynamics of deformable objects. Traditional methods struggle with self-occlusions and the difficulty of mapping raw visual data to complex bimanual actions. FabricFlowNet (FFN) addresses these challenges by using optical flow as a central representation to bridge the gap between current states and desired goals.
The FabricFlowNet Architecture
FabricFlowNet is a goal-conditioned policy designed to perform bimanual pick-and-place actions. The system is trained entirely in simulation using supervised learning, where ground-truth particle positions provide the necessary flow data. The pipeline involves several key components:
- Flow Estimation: Given depth images of the current cloth state and the goal, a FlowNet-style architecture predicts a dense flow field. This field maps pixels from the current observation to their target coordinates in the goal image.
- PickNet: A two-stage convolutional neural network predicts optimal grasp points. PickNet1 identifies the first grasp point, while PickNet2 predicts the second, conditioned on the first to ensure coordinated bimanual movement.
- Flow-based Placing: The placement point for each grasp is determined by querying the flow field at the predicted pick location. This "secret sauce" assumes that to reach a goal, a point on the cloth should move to its corresponding location in the goal image.
- Bimanual Switching: The system dynamically chooses between single-arm and dual-arm actions based on the predicted trajectories to avoid collisions and optimize the fold.
By using optical flow, the authors decouple geometric understanding from action selection. This separation of concerns makes the learning task significantly more manageable than end-to-end "pixels-to-motor-commands" approaches.
Experimental Results
FFN demonstrates superior performance in both simulation and real-world environments. In SoftGym benchmarks, it achieved a mean particle distance error of 7.14 mm, significantly lower than previous model-based (Fabric-VSF at 8.27 mm) and reinforcement learning approaches (Lee et al. at 17.49 mm). Furthermore, FFN is highly efficient, with an inference time of approximately 0.007 seconds, compared to the hundreds of seconds required by planning-based methods.
Real-world experiments on a dual-arm Franka Emika Panda system showed that FFN generalizes well to different cloth shapes (rectangles, T-shirts) and colors, achieving a mean Intersection-over-Union (IoU) of 0.80. This zero-shot generalization suggests that the model learns local geometric relationships universal to fabric, regardless of the overall garment shape.
Discussion and Future Work
While FFN excels at pick-and-place folding, it is currently limited to static primitives and cannot perform dynamic maneuvers like "flinging" or "smoothing." Additionally, for complex multi-step folds, the system requires a sequence of sub-goal images rather than planning the entire sequence from a single final goal. Future research may involve integrating flow into more dynamic actions or learning to generate sub-goal sequences automatically.
Summary updated