Fast GraspNeXt: A Fast Self-Attention Neural Network Architecture for Multi-task Learning in Computer Vision Tasks for Robotic Grasping on the Edge

Alexander Wong, Yifan Wu, Saad Abbasi, Saeejith Nair, Yuhao Chen, Mohammad Javad ShafieeWorkshops @ CVPR 2023 · OpenReview

Abstract

Multi-task learning has shown considerable promise for improving the performance of deep learning-driven vision systems for the purpose of robotic grasping. However, high architectural and computational complexity can result in poor suitability for deployment on embedded devices that are typically leveraged in robotic arms for real-world manufacturing and warehouse environments. As such, the design of highly efficient multi-task deep neural network architectures tailored for computer vision tasks for robotic grasping on the edge is highly desired for widespread adoption in manufacturing environments. Motivated by this, we propose Fast GraspNeXt, a fast self-attention neural network architecture tailored for embedded multi-task learning in computer vision tasks for robotic grasping. To build Fast GraspNeXt, we leverage a generative network architecture search strategy with a set of architectural constraints customized to achieve a strong balance between multi-task learning performance and embedded inference efficiency. Experimental results on the MetaGraspNet benchmark dataset show that the Fast GraspNeXt network design achieves the highest performance (average precision (AP), accuracy, and mean squared error (MSE)) across multiple computer vision tasks when compared to other efficient multi-task network architecture designs, while having only 17.8M parameters (about >5x smaller), 259 GFLOPs (as much as >5x lower) and as much as >3.15x faster on a NVIDIA Jetson TX2 embedded processor.

Introduction

In modern manufacturing and warehouse environments, robotic arms rely on deep learning to perceive and interact with objects. Effective grasping requires solving several complex computer vision problems simultaneously: identifying visible object parts, estimating full shapes under occlusion (amodal masking), predicting the center of mass for stable lifting, and determining optimal suction grasp points.

The four primary computer vision tasks for robotic grasping: (a) Visible mask, (b) Amodal mask, (c) Center of mass, and (d) Suction grasp heatmap.

While multi-task learning improves overall performance, the resulting models are often too computationally expensive for deployment on "edge" devices—the embedded computers (e.g., NVIDIA Jetson TX2) integrated into robotic systems. Fast GraspNeXt addresses this by providing a model that is both multi-functional and optimized for low-power hardware, bridging the gap between high-level perception and real-time execution.

Methodology

The Fast GraspNeXt architecture was developed using Generative Network Architecture Search (NAS). Rather than manual design, a generative synthesis algorithm optimized the network structure based on predefined operational constraints for edge efficiency. This approach resulted in a heterogeneous, columnar architecture where different structures are used at different stages to maximize efficiency.

Key components of the architecture include:

  • Attention Condensers: These are highly efficient self-attention mechanisms. Unlike standard Transformers that are compute-heavy, these condensers learn condensed embeddings to capture joint local and cross-channel relationships with minimal overhead, providing the global scene understanding necessary for tasks like amodal perception.
  • Anti-Aliased Downsampling (AADS): This technique improves the network's stability and "shift-invariance," ensuring the robot recognizes an object consistently even if its position in the frame shifts slightly.
  • Feature Pyramid Network (FPN): The architecture uses a backbone to extract features, which are then fed into an FPN. This allows the model to detect and process objects at multiple scales.
  • Specialized Task Heads: The network branches into six unique heads, each tailored for a specific output such as bounding boxes, segmentation masks, or heatmaps.

Experimental Evaluation

The model was evaluated using the MetaGraspNet benchmark, a large-scale dataset specifically designed for robotic bin-picking. It contains 217,000 images across 5,884 scenes and 82 object types. The dataset includes synthetic data with physics-based rendering to simulate realistic occlusions and stacking, providing a rigorous test for the model's amodal perception capabilities.

Results and Discussion

Fast GraspNeXt was compared against several efficient architectures, including ResNet-50, EfficientNet-B0, and MobileNetV3, on an NVIDIA Jetson TX2 embedded processor. The results demonstrate a significant leap in efficiency:

  • Inference Speed: Fast GraspNeXt achieved an inference time of 1106 ms, making it 3.15x faster than the ResNet-GraspNeXt variant.
  • Model Compactness: It has only 17.8M parameters, which is >5x smaller than the ResNet version.
  • Computational Efficiency: It requires only 259 GFLOPs, compared to 1314 GFLOPs for the ResNet baseline.

Despite its smaller size, Fast GraspNeXt outperformed all baseline models across every accuracy metric. Notably, it showed a 2.9% improvement in Amodal Bbox Average Precision (AP) and a significant reduction in Heatmap Mean Squared Error (MSE). The results confirm that joint training on related tasks (e.g., center of mass and suction heatmaps) actually enhances feature representation compared to single-task training.

Conclusion

Fast GraspNeXt demonstrates that high-performance multi-task robotic vision can be achieved on edge hardware through efficient architectural design and generative search. By utilizing attention condensers and a heterogeneous structure, the model provides the necessary perception for complex grasping tasks without the prohibitive computational cost of traditional deep learning models. Future work will focus on real-world deployment in physical manufacturing environments to validate performance under variable lighting and environmental conditions.

Summary updated

Abstract

Multi-task learning has shown considerable promise for improving the performance of deep learning-driven vision systems for the purpose of robotic grasping. However, high architectural and computational complexity can result in poor suitability for deployment on embedded devices that are typically leveraged in robotic arms for real-world manufacturing and warehouse environments. As such, the design of highly efficient multi-task deep neural network architectures tailored for computer vision tasks for robotic grasping on the edge is highly desired for widespread adoption in manufacturing environments. Motivated by this, we propose Fast GraspNeXt, a fast self-attention neural network architecture tailored for embedded multi-task learning in computer vision tasks for robotic grasping. To build Fast GraspNeXt, we leverage a generative network architecture search strategy with a set of architectural constraints customized to achieve a strong balance between multitask learning performance and embedded inference efficiency. Experimental results on the MetaGraspNet benchmark dataset show that the Fast GraspNeXt network design achieves the highest performance (average precision (AP), accuracy, and mean squared error (MSE)) across multiple computer vision tasks when compared to other efficient multi-task network architecture designs, while having only 17.8M parameters (about >5× smaller), 259 GFLOPs (as much as >5× lower) and as much as >3.15× faster on a NVIDIA Jetson TX2 embedded processor.
BibTeX
@inproceedings{wong2023fast,
  author = {Alexander Wong and Yifan Wu and Saad Abbasi and Saeejith Nair and Yuhao Chen and Mohammad Javad Shafiee},
  title = {{Fast GraspNeXt: A Fast Self-Attention Neural Network Architecture for Multi-task Learning in Computer Vision Tasks for Robotic Grasping on the Edge}},
  booktitle = {Workshops @ CVPR},
  year = {2023},
  url = {https://openreview.net/forum?id=9_riA0T0Fpd},
}
Figure