RGB-X Classification for Electronics Sorting

Abhimanyu, Tejas Zodage, Umesh Thillaivasan, Xinyue Lai, Rahul Chakwate, Javier Santillan, Emma Oti, Ming Zhao, Ralph Boirum, Howie Choset, Matthew J. Travers

Abstract

Effectively disassembling and recovering materials from waste electrical and electronic equipment (WEEE) is a critical step in moving global supply chains from carbon-intensive, mined materials to recycled and renewable ones. Conventional recycling processes rely on shredding and sorting waste streams, but for WEEE—which is comprised of numerous dissimilar materials—targeted disassembly offers a superior path for improved material recovery. However, many WEEE objects share key external features and look nearly identical, despite varying internal component layouts and material compositions. This work introduces RGB-X, a multi-modal image classification approach that utilizes features from external RGB images and internal X-ray images. By employing a novel Iterative Class Activation Mapping (iCAM) architecture and a synthetic data generation pipeline using domain randomization, we achieve a classification accuracy of 98.6% on 10 generations of modern smartphones, significantly outperforming individual RGB (89.1%) and X-ray (97.9%) models.

Introduction

As the global community moves toward a circular economy, the recovery of precious materials—such as rare earth elements, tungsten, and steel—from electronic waste is paramount. Current industrial recycling often utilizes "shred and sort" methods, which are inherently inefficient and lead to material cross-contamination. Targeted disassembly, performed by specialized robotic systems, allows for cleaner material separation. The bottleneck for such systems is the initial classification step: a robot must know exactly what device it is handling to apply the correct disassembly sequence.

Identifying electronics is notoriously difficult. For example, an iPhone 7 and an iPhone 8 are visually indistinguishable to most standard computer vision systems, especially when the devices are damaged, dirty, or missing external components. While X-ray imaging can reveal internal layouts that distinguish these models, the lack of large, annotated X-ray datasets has historically hindered the development of robust deep learning models in this domain.

Methodology

Iterative Class Activation Mapping (iCAM)

To address the need for fine-grained classification, we developed the iCAM architecture. Rather than processing an entire image in a single pass, iCAM utilizes an attention-based iterative strategy. The model identifies the most discriminative region for classification (the "heat map"), crops that specific area, and passes it back through the network. By performing this iteratively (typically $n=3$), the model "zooms in" on the subtle details—such as a specific battery connector or a unique screw pattern—that differentiate similar models.

Synthetic X-ray Data and Domain Randomization

To overcome the scarcity of labeled X-ray data, we developed a novel synthetic data pipeline. We utilized 3D CT scans of electronic devices and projected them into 2D space to simulate X-ray imagery. To ensure these synthetic images generalized to real-world scans, we applied Domain Randomization (DR). This involved randomly varying parameters such as device orientation, sensor noise, and "transfer functions" (which simulate different X-ray machine energy settings). This approach allowed us to train models on approximately 25,000 synthetic images that translated seamlessly to real-world hardware.

RGB-X Fusion

The RGB-X system processes color images and X-ray images through separate iCAM streams. The features extracted from both modalities are then integrated using a "Fast Fusion" method in the deeper layers of the network. This allows the model to rely on external visual cues (like logos or port placements) when internal data is noisy, and internal layouts when the external casing is deceptive or damaged.

Experimental Results

Performance on Smartphone Datasets

We evaluated our approach on a dataset of 10 generations of modern smartphones (SP1-SP10). The results demonstrate the clear advantage of multi-modal sensing:

  • RGB Only: 89.1% accuracy
  • X-ray Only: 97.9% accuracy
  • RGB-X Fusion: 98.6% accuracy

The high performance of the X-ray-only model, trained entirely on synthetic data, highlights the effectiveness of our domain randomization pipeline. Ablation studies showed that varying the 3D projector position was the most critical factor; without it, accuracy dropped to 31.5%.

Generalization and Speed

To verify the iCAM architecture's versatility, we tested it on the CUB-200 bird species dataset, where it proved effective for general fine-grained classification tasks. In terms of industrial utility, the RGB-X model operates at approximately 30 FPS, making it suitable for real-time sorting on high-speed industrial conveyor belts.

Conclusion and Future Work

RGB-X provides a robust framework for the automated identification of complex electronic waste. By combining the external context of RGB with the internal structural data of X-rays, we can achieve the precision required for automated disassembly. Future research will focus on expanding the dataset to include a wider variety of manufacturers and evolving the iCAM module to attend to multiple regions of interest simultaneously. Furthermore, we aim to improve the interpretability of these models to provide clear justifications for classification decisions in industrial settings.

Summary updated

Abstract

Effectively disassembling and recovering materials from waste electrical and electronic equipment (WEEE) is a critical step in moving global supply chains from carbon-intensive, mined materials to recycled and renewable ones. Conventional recycling processes rely on shredding and sorting waste streams, but for WEEE, which is comprised of numerous dissimilar materials, we explore targeted disassembly of numerous objects for improved material recovery. Many WEEE objects share many key features and therefore can look quite similar, but their material composition and internal component layout can vary, and thus it is critical to have an accurate classifier for subsequent disassembly steps for accurate material separation and recovery. This work introduces RGB-X, a multi-modal image classification approach, that utilizes key features from external RGB images with those generated from X-ray images to accurately classify electronic objects. More specifically, this work develops Iterative Class Activation Mapping (iCAM), a novel network architecture that explicitly focuses on the finer-details in the multi-modal feature maps that are needed for accurate electronic object classification. In order to train a classifier, electronic objects lack large and well annotated X-ray datasets due to expense and need of expert guidance. To overcome this issue, we present a novel way of creating a synthetic dataset using domain randomization applied to the X-ray domain. The combined RGB-X approach gives us an accuracy of 98.6% on 10 generations of modern smartphones, which is greater than their individual accuracies of 89.1% (RGB) and 97.9% (X-ray) independently. We provide experimental results3 to corroborate our results.
BibTeX
@inproceedings{abhimanyu2022rgb,
  author = {Abhimanyu and Tejas Zodage and Umesh Thillaivasan and Xinyue Lai and Rahul Chakwate and Javier Santillan and Emma Oti and Ming Zhao and Ralph Boirum and Howie Choset and Matthew J. Travers},
  title = {{RGB-X Classification for Electronics Sorting}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=kiZuurAxTTk2},
}
Figure