Dynamic recognition and mirage using neuro-metamaterials

Chao Qian, Zhedong Wang, Haoliang Qian, Tong Cai, Bin Zheng, Xiao Lin, Yichen Shen, Ido Kaminer, Er‐Ping Li, Hongsheng ChenNature Communications 2022 · DOI

TL;DR

This paper introduces "neuro-metamaterials," a novel optical computing approach that enables entirely photonic, real-time 3D object recognition and dynamic optical mirages. By leveraging metamaterials optimized with deep learning, the system can identify object postures and even transform one object's movement into another's holographic display, bypassing traditional electronic bottlenecks.

What Problem Does This Paper Address?

The paper tackles the limitations of current object recognition systems, which typically rely on a two-step process: capturing images with a camera and then processing them with digital computers. While effective, this approach is increasingly insufficient for the demands of Big Data and the Internet of Things, where speed-of-light parallel information processing is crucial. Existing optical computing methods, while promising, still require digital processing for in-situ applications, tethering them to Moore's Law. The authors aim to overcome this by developing an entirely optical, real-time 3D object recognition system that integrates both sensing and processing within metamaterials.

Methods

The core innovation is the concept of neuro-metamaterials. These are fabricated from inhomogeneous metamaterials or transmission metasurfaces and are optimized using techniques like topology optimization and deep learning. Essentially, these metamaterials act as "scattering neural networks," where their physical structure is designed to process scattered electromagnetic (EM) waves directly.

The process involves:

  • Data Collection: Simulating various scenarios (e.g., different rabbit sizes, postures, angles, and distances) to create a comprehensive dataset.
  • Scattering Matrix Optimization: Employing a diffractive neural network algorithm to train the metamaterial's response. This is analogous to training an artificial neural network, where the goal is to minimize a loss function related to accurate output.
  • Neuro-Metamaterial Design: Constructing the physical metamaterial based on the optimized scattering matrix. The paper focuses on a two-layer design, where subwavelength meta-atoms act as individual neurons.

For the optical mirage demonstration, the neuro-metamaterials are optimized to transform a sequence of input images (rabbit movements) into a different output sequence (giraffe movements). This is framed as an optimization task rather than a simple inference task.

Datasets

The paper utilizes a newly generated dataset of simulated rabbit movements. This dataset was created using CST Microwave Studio and includes variations in:

  • Rabbit size (large, medium, small)
  • Rabbit posture (standing, sitting, walking)
  • Angle of observation relative to the neuro-metamaterial (-30° to 30°)
  • Distance from the rabbit to the neuro-metamaterial (10 mm to 100 mm)

This dataset was split into 80% for training and 20% for testing the neuro-metamaterial's performance.

Key Insights and Ideas

  • "Scattering Neural Networks": The fundamental idea that the physical structure of a metamaterial can be engineered to perform complex computations (like object recognition) directly on incoming EM waves, bypassing electronic processing.
  • Entirely Optical 3D Object Recognition: Moving beyond 2D image processing to recognize objects and their postures in three dimensions using only light.
  • Dynamic Optical Mirages: Demonstrating that these neuro-metamaterials can not only recognize but also transform dynamic visual information, creating illusions like a rabbit's movement appearing as a giraffe's. This opens doors for novel display and encryption technologies.
  • Bridging the Gap to Real-World Applications: The work provides a tangible pathway for implementing advanced optical computing concepts in practical scenarios, moving beyond theoretical proposals.

Key Results

  • Object Recognition Accuracy: The neuro-metamaterials achieved an accuracy rate of 99.5% for training and 98.2% for testing in recognizing representative rabbit postures (walking, standing, sitting) in simulations.
  • Experimental Validation: Experiments with living rabbits confirmed the ability to perceive postures, with the strongest signals appearing at expected locations.
  • Optical Mirage Performance: The dynamic optical mirage of a giraffe achieved a high Structural Similarity (SSIM) index of 93.51% in simulations. Experimental results also demonstrated the transformation of rabbit movements into a giraffe's, with the giraffe's head moving down gradually.
  • Speed Advantage: The processing is described as "light speed," implying a significant advantage over electronic processing, especially for real-time applications.

Limitations and Future Work

  • Frequency Dependence: The current design is optimized for a specific microwave frequency (8.6 GHz). For different frequencies, new metamaterials need to be designed due to material dispersion.
  • Sensitivity to Fabrication: The paper notes that the sharp changes in phase and amplitude in certain regions of the dispersion relation can make fabrication sensitive to slight variations. While using more layers can mitigate this, it can also degrade robustness and induce mutual coupling. The authors utilize the working regime outside these sensitive regions.
  • Scalability and Complexity: While the concept is demonstrated, scaling to more complex objects or scenarios will require further research.
  • Future Directions: The authors suggest extending this technology to applications like intelligent metasurfaces for wireless communication, illusive cloaking, information processing and display, and potentially ushering in an "Optical Internet of Things" era. They also mention the possibility of embedding nonlinear components for extended capabilities and exploring optical neuromorphic computing.

Summary updated

Abstract

Breakthroughs in the field of object recognition facilitate ubiquitous applications in the modern world, ranging from security and surveillance equipment to accessibility devices for the visually impaired. Recently-emerged optical computing provides a fundamentally new computing modality to accelerate its solution with photons; however, it still necessitates digital processing for in situ application, inextricably tied to Moore's law. Here, from an entirely optical perspective, we introduce the concept of neuro-metamaterials that can be applied to realize a dynamic object- recognition system. The neuro-metamaterials are fabricated from inhomogeneous metamaterials or transmission metasurfaces, and optimized using, such as topology optimization and deep learning. We demonstrate the concept in experiments where living rabbits play freely in front of the neuro-metamaterials, which enable to perceive in light speed the rabbits' representative postures. Furthermore, we show how this capability enables a new physical mechanism for creating dynamic optical mirages, through which a sequence of rabbit movements is converted into a holographic video of a different animal. Our work provides deep insight into how metamaterials could facilitate a myriad of in situ applications, such as illusive cloaking and speed-of-light information display, processing, and encryption, possibly ushering in an "Optical Internet of Things" era.
BibTeX
@inproceedings{qian2022dynamic,
  author = {Chao Qian and Zhedong Wang and Haoliang Qian and Tong Cai and Bin Zheng and Xiao Lin and Yichen Shen and Ido Kaminer and Er‐Ping Li and Hongsheng Chen},
  title = {{Dynamic recognition and mirage using neuro-metamaterials}},
  booktitle = {Nature Communications},
  year = {2022},
  doi = {10.1038/s41467-022-30377-6},
}
Figure