InterACT: Inter-dependency Aware Action Chunking with Hierarchical Attention Transformers for Bimanual Manipulation

Andrew Lee, Ian Chuang, Ling-Yuan Chen, Iman Soltani

Abstract

Bimanual manipulation presents unique challenges compared to unimanual tasks due to the complexity of coordinating two robotic arms. In this paper, we introduce InterACT: Inter-dependency aware Action Chunking with Hierarchical Attention Transformers, a novel imitation learning framework designed specifically for bimanual manipulation. InterACT leverages hierarchical attention mechanisms to effectively capture inter-dependencies between dual-arm joint states and visual inputs. The framework comprises a Hierarchical Attention Encoder, which processes multi-modal inputs through segment-wise and cross-segment attention mechanisms, and a Multi-arm Decoder that generates each arm's action predictions in parallel, while sharing information between the arms through synchronization blocks by providing the other arm's intermediate output as context. Our experiments, conducted on various simulated and real-world bimanual manipulation tasks, demonstrate that InterACT outperforms existing methods. Detailed ablation studies further validate the significance of key components, including the impact of CLS tokens, cross-segment encoders, and synchronization blocks on task performance.

Introduction

Bimanual manipulation—tasks requiring two hands, such as unscrewing a bottle cap or plugging in a cable—represents a significant challenge in robotics. Unlike single-arm tasks, bimanual actions require millisecond-level coordination and a deep understanding of how the movement of one arm affects the other and the surrounding environment.

Current imitation learning methods, such as Action Chunking with Transformers (ACT), have made progress by predicting sequences of actions to mitigate compounding errors. However, these methods often treat the entire state (joint states of both arms and camera feeds) as a single flat sequence. This "bag of tokens" approach struggles to capture the nuanced, hierarchical relationship between the two arms, often resulting in suboptimal performance in high-precision tasks.

Methods

InterACT addresses these limitations through a Hierarchical Attention Transformer architecture, moving away from flat sequence processing. The architecture is defined by two primary innovations:

Hierarchical Attention Encoder

Instead of processing all input data simultaneously, InterACT treats inputs as distinct "segments" (e.g., Arm 1 joints, Arm 2 joints, and Visual features).

  • Segment-Wise Encoder: Processes each segment individually to capture internal relationships, such as the kinematic chain of a single arm.
  • Cross-Segment Encoder: Utilizes [CLS] (classification) tokens to summarize each segment. These summaries are passed through a cross-segment encoder, allowing the model to share high-level information between the arms and the vision system without the computational overhead of full global attention.

Multi-arm Decoder with Synchronization

The model employs two parallel decoder paths—one for each arm. A critical component is the Synchronization Block. Before final action prediction, intermediate outputs from both decoders are concatenated and processed through a self-attention layer. This mechanism ensures that each arm's plan is informed by the other's intermediate state, facilitating tight coordination.

The InterACT architecture featuring the Hierarchical Attention Encoder and Multi-arm Decoder with Synchronization Block.

Experiments and Results

The framework was evaluated on a variety of benchmarks using the ALOHA 2 hardware setup, including simulated tasks like Transfer Cube and Peg Insertion, and real-world tasks such as Slide Ziploc and Thread Velcro. New challenging tasks were also introduced: Slot Insertion (simulated), Insert Plug, Click Pen, Sweep, and Unscrew Cap (real-world).

InterACT consistently outperformed the baseline ACT model across all nine tasks. Notably, in the simulated Peg Insertion task, InterACT achieved a 44% success rate, more than doubling ACT’s 20%. In real-world scenarios, InterACT reached a 42% success rate in the Insert Plug task compared to ACT’s 30%.

Success rates of InterACT versus ACT across various simulated and real-world bimanual tasks.

Ablation studies confirmed the importance of the architectural choices. Removing [CLS] tokens or the Synchronization block led to significant performance drops; for instance, success in the Slot Insertion task plummeted from 100% to 24% without the cross-segment encoder.

Discussion and Conclusion

InterACT demonstrates that explicitly modeling inter-dependencies through hierarchical attention significantly improves bimanual coordination. Analysis of attention weights revealed that [CLS] token attention spikes during highly coordinated task phases, validating their role in capturing critical inter-dependency signals.

Future work will address current limitations, such as hyperparameter sensitivity regarding the number of [CLS] tokens. Additionally, integrating tactile feedback or force sensors could further enhance performance in tasks requiring fine-grained physical interaction, such as unscrewing caps. While InterACT excels in the 50-demonstration regime, its scalability to larger, multi-task datasets remains an area for future exploration.

Summary updated

Abstract

Bimanual manipulation presents unique challenges compared to unimanual tasks due to the complexity of coordinating two robotic arms. In this paper, we introduce InterACT: Inter-dependency aware Action Chunking with Hierarchical Attention Transformers, a novel imitation learning framework designed specifically for bimanual manipulation. InterACT leverages hierarchical attention mechanisms to effectively capture inter-dependencies between dual-arm joint states and visual inputs. The framework comprises a Hierarchical Attention Encoder, which processes multi-modal inputs through segment-wise and cross-segment attention mechanisms, and a Multi-arm Decoder that generates each arm's action predictions in parallel, while sharing information between the arms through synchronization blocks by providing the other arm's intermediate output as context. Our experiments, conducted on various simulated and real-world bimanual manipulation tasks, demonstrate that InterACT outperforms existing methods. Detailed ablation studies further validate the significance of key components, including the impact of CLS tokens, cross-segment encoders, and synchronization blocks on task performance. We provide supplementary materials and videos on our project page.
BibTeX
@inproceedings{lee2024interact,
  author = {Andrew Lee and Ian Chuang and Ling-Yuan Chen and Iman Soltani},
  title = {{InterACT: Inter-dependency Aware Action Chunking with Hierarchical Attention Transformers for Bimanual Manipulation}},
  booktitle = {Conference on Robot Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=lKGRPJFPCM},
}
Figure