RGB-X Classification for Electronics Sorting

Abhimanyu, Tejas Zodage, Umesh Thillaivasan, Xinyue Lai, Rahul Chakwate, Javier Santillan, Emma Oti, Ming Zhao, Ralph Boirum, Howie Choset, Matthew J. Travers

TL;DR

Researchers have developed RGB-X, a multi-modal classification system that combines standard photos with X-ray imagery to identify electronic devices for automated disassembly. By using a novel Iterative Class Activation Mapping (iCAM) architecture and synthetic data generation, the system achieves 98.6% accuracy in distinguishing between nearly identical smartphone models, paving the way for high-efficiency material recovery.

What Problem Does This Paper Address?

As the world moves toward a circular economy, recovering precious materials (like rare earth elements, tungsten, and steel) from Waste Electrical and Electronic Equipment (WEEE) is critical. Current recycling often relies on "shred and sort" methods, which are messy and inefficient. Targeted disassembly—where robots like Apple’s Daisy or Dave take devices apart—is much better, but it requires a perfect "first step": accurate classification.

The challenge is that many electronics look identical on the outside (think of an iPhone 7 vs. an iPhone 8). Furthermore, devices in the waste stream are often damaged, dirty, or modified. Standard RGB (color) cameras struggle with these subtle external differences, and while X-rays can "see" internal layouts, there aren't enough labeled X-ray datasets to train robust AI models.

Methods

The authors propose a two-pronged solution involving a new neural network architecture and a clever data generation pipeline:

  1. Iterative Class Activation Mapping (iCAM): Instead of looking at the whole image at once, iCAM uses an attention-based strategy. It identifies the most "important" region for classification (the heat map), crops it, and passes it back through the network. By doing this iteratively (the paper uses $n=3$ iterations), the model "zooms in" on fine-grained details—like a specific internal component or a tiny logo—that distinguish one model from another.
  2. Domain Randomization (DR) for X-rays: To solve the data scarcity problem, the team created a synthetic X-ray pipeline. They took 3D CT scans of devices and projected them into 2D space, randomly varying parameters like orientation, noise, and "transfer functions" (which simulate different X-ray machine settings). This allowed them to train models on thousands of virtual X-rays that generalize perfectly to real-world scans.
  3. RGB-X Fusion: The system processes RGB and X-ray images through separate iCAM streams and then "fuses" their features in the deeper layers of the network (a method called "Fast Fusion") to make a final prediction.

Datasets

  • Smartphone Dataset: 10 classes of modern smartphones (referred to as SP1-SP10).
  • RGB: 799 real images for testing; training used a mix of 1,000 synthetic (Blender-generated) and 250 real images per class.
  • X-ray: 749 real images for testing; training was done on ~25,000 synthetic images generated from CT scans.
  • CUB-200: A standard benchmark dataset of 200 bird species used to prove that the iCAM architecture works for general fine-grained classification tasks beyond just electronics.

Key Insights and Ideas

  • Internal vs. External: The paper proves that while X-ray data is more informative than RGB for electronics (97.9% vs 89.1% accuracy), the combination is the most robust. RGB helps when internal components are missing, while X-ray helps when the external casing is damaged or identical to another model.
  • The Power of "Zooming In": The iCAM module significantly outperformed standard architectures (like ResNet or Inception). By forcing the model to focus on localized features, it effectively mimics how a human expert might look for a specific screw pattern or battery shape to identify a device.
  • Synthetic Data is Enough: One of the most impressive findings is that the X-ray model was trained entirely on synthetic data and still achieved near-perfect accuracy on real-world X-ray scans. This suggests that the "reality gap" in X-ray imaging can be bridged effectively with domain randomization.

Key Results

  • Top Accuracy: The combined RGB-X approach reached 98.6% accuracy on the smartphone dataset.
  • Speed: The model runs at approximately 30 FPS, making it fast enough for real-time industrial conveyor belt sorting.
  • Ablation Success: Removing the "projector position" randomization in the synthetic data pipeline caused accuracy to plummet to 31.5%, proving that simulating different 3D viewpoints is the "secret sauce" for training these models.

Limitations and Future Work

While the results are stellar, the study focused on 10 specific smartphone models (primarily iPhones, based on the figures). The authors acknowledge that future work needs to expand this to a wider variety of manufacturers like Samsung and Motorola.

Additionally, they plan to evolve the iCAM module to attend to multiple important regions simultaneously rather than just one cropped area. Finally, they aim to move beyond classification to "interpretability," helping the system explain why it identified a device a certain way, which is vital for debugging in a high-stakes recycling environment.

Summary updated

Abstract

Effectively disassembling and recovering materials from waste electrical and electronic equipment (WEEE) is a critical step in moving global supply chains from carbon-intensive, mined materials to recycled and renewable ones. Conventional recycling processes rely on shredding and sorting waste streams, but for WEEE, which is comprised of numerous dissimilar materials, we explore targeted disassembly of numerous objects for improved material recovery. Many WEEE objects share many key features and therefore can look quite similar, but their material composition and internal component layout can vary, and thus it is critical to have an accurate classifier for subsequent disassembly steps for accurate material separation and recovery. This work introduces RGB-X, a multi-modal image classification approach, that utilizes key features from external RGB images with those generated from X-ray images to accurately classify electronic objects. More specifically, this work develops Iterative Class Activation Mapping (iCAM), a novel network architecture that explicitly focuses on the finer-details in the multi-modal feature maps that are needed for accurate electronic object classification. In order to train a classifier, electronic objects lack large and well annotated X-ray datasets due to expense and need of expert guidance. To overcome this issue, we present a novel way of creating a synthetic dataset using domain randomization applied to the X-ray domain. The combined RGB-X approach gives us an accuracy of 98.6% on 10 generations of modern smartphones, which is greater than their individual accuracies of 89.1% (RGB) and 97.9% (X-ray) independently. We provide experimental results <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> Experimental work done at Biorobotics Lab, Robotics Institute, Carnegie Mellon University to corroborate our results.
BibTeX
@inproceedings{abhimanyu2022rgb,
  author = {Abhimanyu and Tejas Zodage and Umesh Thillaivasan and Xinyue Lai and Rahul Chakwate and Javier Santillan and Emma Oti and Ming Zhao and Ralph Boirum and Howie Choset and Matthew J. Travers},
  title = {{RGB-X Classification for Electronics Sorting}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=kiZuurAxTTk2},
}

Loading related content...

Figure