TL;DR
This paper introduces a novel hybrid optical-digital neural network (ONN-DNN) for real-time Terahertz (THz) imaging. By using an optical neural network to compress THz signals before they are captured by a sensor, and then a digital neural network to reconstruct the image, the system achieves high-quality, diffraction-free imaging at video rates without needing bulky hardware or GPUs.
What Problem Does This Paper Address?
Terahertz (THz) imaging offers unique capabilities for non-invasive, non-destructive sensing across various fields, from biomedical applications to security screening. However, current THz imaging systems face significant hurdles: hardware limitations lead to information loss and slow data acquisition, hindering real-time applications. Traditional methods like raster scanning are too slow for dynamic scenes, while focal-plane array (FPA) sensors, though faster, are susceptible to diffraction artifacts that degrade image quality, especially for fine interior details. Existing computational imaging approaches, like compressed sensing, also struggle with trade-offs between speed, resolution, and field of view. This paper aims to overcome these limitations by developing a system that is both fast and produces high-quality, artifact-free images.
Methods
The core innovation is a hybrid Optical-Digital Neural Network (ONN-DNN) framework. This system employs a two-stage process:
- Optical Encoding (ONN): A physical Optical Neural Network (ONN), composed of multiple diffractive layers, acts as an optical encoder. It takes the high-dimensional THz signal (carrying amplitude and phase information) and compresses it into a lower-dimensional representation (intensity form). This compression is achieved through light-speed optical computations, leveraging wave propagation and light-matter interactions. The ONN effectively reduces the data dimensionality to match the constraints of THz sensor arrays.
- Digital Decoding (DNN): The compressed optical signals are then captured by a THz sensor array and digitized. A jointly trained Digital Neural Network (DNN) acts as the decoder. It takes the compressed digital data and decompresses it, reconstructing the original, high-dimensional THz image. Crucially, the DNN is designed to compensate for signal degradations (like absorption and diffraction) that occur during propagation and capture, effectively mitigating diffraction artifacts.
The ONN and DNN are trained jointly using a Variational Autoencoder (VAE) framework. This approach optimizes both the encoder (ONN) and decoder (DNN) simultaneously to maximize the likelihood of the observed data, ensuring accurate reconstruction and a well-behaved latent space. The physical ONN layers are fabricated using a 3D-printable resin composite and a high-precision MSLA 3D printer.
Datasets
The paper utilized a dataset of 6,656 uppercase alphabet letter images (A-Z) sourced from Kaggle for training. For evaluating the model's generalization ability, a separate dataset of lowercase letters (26 categories, 8 samples per category) was used.
Key Insights and Ideas
- Offloading Computation to Optics: The ONN performs a significant portion of the signal processing (compression and initial feature extraction) at the speed of light, drastically reducing the computational burden on digital hardware.
- Diffraction Mitigation through Joint Training: By jointly training the ONN and DNN, the system learns to not only compress signals but also to reconstruct them in a way that actively counteracts diffraction artifacts, a major challenge in THz imaging.
- Bridging Optical and Digital Domains: The framework seamlessly integrates physical optical components (ONN) with digital processing (DNN), creating a synergistic system that leverages the strengths of both.
- Hardware Constraint Adaptation: The ONN's compression capability allows the system to work with lower-resolution THz sensor arrays, making it more practical and cost-effective.
Key Results
- Real-time Imaging: The system achieves real-time THz video capture at a rate of 2 frames per second without requiring a GPU.
- Enhanced Image Quality: Reconstructed images exhibit enhanced imaging quality and are diffraction-free.
- Expanded Field of View: The system demonstrates an expanded field of view (2.4 x 2.4 cm²) compared to the sensor array size, achieved without any lensing system.
- Reconstruction Accuracy: For letter images, the system achieved an average PSNR of 10.5 dB and an MSE of 65.5.
- Experimental Validation: The fabricated ONN showed a strong correlation with simulated results, achieving an average PSNR of 22.63 dB in experimental compressed imaging.
- Dynamic Scene Capture: The system successfully captured a moving screwdriver in real-time, demonstrating its ability to handle dynamic scenarios.
Limitations and Future Work
- Hardware Constraints: While the system achieves 2 fps, this rate is primarily limited by the current hardware setup. Integrating advanced GPUs could significantly increase frame rates.
- Material Properties: The performance of the 3D-printed ONN is dependent on the properties of the composite resin, including its refractive index and absorption loss.
- Generalizability: While the system showed good generalization to unseen letter data, further testing on more diverse and complex object types would be beneficial.
- Scalability: The paper suggests that the ONN-DNN framework is not limited to the THz band and could be applied to other electromagnetic and acoustic spectra (RF, mmWave, IR, visible light, ultrasonic). Future work could explore these extensions.
- Alternative Architectures: The framework is flexible and can integrate alternative neural network architectures like residual networks or transformers for further optimization.
Summary updated