Integration of Communication and Computational Imaging

Zhenming Yu, Liming Cheng, Hongyu Huang, Wei Zhang, Liang Lin, Kun Xu

Abstract

Communication enables the expansion of human visual perception beyond the limitations of time and distance, while computational imaging overcomes the constraints of depth and breadth. Although impressive achievements have been witnessed with the two types of technologies, the occlusive information flow between the two domains is a bottleneck hindering their ulterior progression. Herein, we propose a novel framework that integrates communication and computational imaging (ICCI) to break through the inherent isolation between communication and computational imaging for remote perception. By jointly considering the sensing and transmitting of remote visual information, the ICCI framework performs a full-link information transfer optimization, aiming to minimize information loss from the generation of the information source to the execution of the final vision tasks. We conduct numerical analysis and experiments to demonstrate the ICCI framework by integrating communication systems and snapshot compressive imaging systems. Compared with straightforward combination schemes, which sequentially execute sensing and transmitting, the ICCI scheme shows greater robustness against channel noise and impairments while achieving higher data compression. Moreover, an 80 km 27-band hyperspectral video perception with a rate of 30 fps is experimentally achieved. This new ICCI remote perception paradigm offers a high-efficiency solution for various real-time computer vision tasks.

The Technical Gap in Remote Perception

In traditional remote sensing applications, such as autonomous drones or deep-sea exploration, imaging and communication have historically been treated as separate, sequential silos. This "straightforward combination" (SC) approach involves capturing an image through Computational Imaging (CI), followed by compression and transmission via a standard communication stack.

Comparison between traditional sequential schemes (a) and the proposed ICCI framework (b).

However, this separation creates a significant "technical gap." Communication systems are typically designed to transmit generic bits without understanding the unique structure of optical measurements. Conversely, imaging algorithms are often fragile; a small amount of transmission noise can cause the entire reconstruction to fail. This leads to high latency, massive bandwidth requirements, and a "cliff-edge" performance drop where image quality collapses as signal strength weakens.

The ICCI Framework: End-to-End Optimization

The Integration of Communication and Computational Imaging (ICCI) framework addresses these limitations by treating the entire path—from photon capture to remote display—as a single unified optimization problem.

Methods and Implementation

The framework leverages two primary technological pillars:

  1. Optical Encoding: The system utilizes Snapshot Compressive Imaging (SCI) techniques, such as CASSI for spectral data. These methods use physical masks to compress high-dimensional data (like 3D video cubes) into a single 2D measurement at the sensor level.
  2. Deep Learning Integration: Instead of standard codecs, ICCI employs a custom Encoder-Interpreter neural network. The Encoder maps raw optical measurements into discrete-time analog symbols optimized for the specific transmission channel. The Interpreter at the receiving end performs "inverse projection," recovering visual information while simultaneously filtering out channel noise.
The R-CASSI hardware setup used for experimental validation of hyperspectral video perception.

By training the network end-to-end, the encoder learns to prioritize visual features that are most resilient to noise, while the interpreter learns the specific interference patterns of the communication channel.

Key Results and Insights

The ICCI framework demonstrates several advantages over traditional sequential methods:

  • Extreme Robustness: Unlike digital systems that fail completely past a noise threshold, ICCI exhibits "graceful degradation." It maintains recognizable images even at 0dB SNR, where noise is as loud as the signal.
  • Efficiency and Speed: By eliminating the redundant processing steps of traditional digital stacks, the system reduces total time complexity by nearly 80%, dropping from 137ms to 28ms per frame.
  • High-Performance Remote Sensing: The researchers experimentally achieved 27-band hyperspectral video at 30 fps over an 80 km fiber link. This corresponds to an equivalent transmission rate of 120 Tbit per second for hyperspectral video perception.
  • Task-Oriented Coding: ICCI is designed to minimize information loss relative to the final visual goal rather than perfect bit-reconstruction, allowing for a Joint Data Compression Ratio (DCR) as low as 0.002 symbols per voxel.

Conclusion and Future Directions

The ICCI framework offers a high-efficiency solution for real-time computer vision tasks in remote environments. While currently optimized for reconstruction, future iterations could be tailored for specific downstream tasks like real-time object detection or classification. As we move toward the Internet of Everything (IoE), further research will focus on adapting these neural networks to handle the rapidly changing channel conditions of highly dynamic mobile environments.

Summary updated

Abstract

Communication enables the expansion of human visual perception beyond the limitations of time and distance, while computational imaging overcomes the constraints of depth and breadth. Although impressive achievements have been witnessed with the two types of technologies, the occlusive information flow between the two domains is a bottleneck hindering their ulterior progression. Herein, we propose a novel framework that integrates communication and computational imaging (ICCI) to break through the inherent isolation between communication and computational imaging for remote perception. By jointly considering the sensing and transmitting of remote visual information, the ICCI framework performs a full-link information transfer optimization, aiming to minimize information loss from the generation of the information source to the execution of the final vision tasks. We conduct numerical analysis and experiments to demonstrate the ICCI framework by integrating communication systems and snapshot compressive imaging systems. Compared with straightforward combination schemes, which sequentially execute sensing and transmitting, the ICCI scheme shows greater robustness against channel noise and impairments while achieving higher data compression. Moreover, an 80 km 27-band hyperspectral video perception with a rate of 30 fps is experimentally achieved. This new ICCI remote perception paradigm offers a highefficiency solution for various real-time computer vision tasks.
BibTeX
@article{yu2025integration,
  author = {Zhenming Yu and Liming Cheng and Hongyu Huang and Wei Zhang and Liang Lin and Kun Xu},
  title = {{Integration of Communication and Computational Imaging}},
  journal = {IEEE Journal on Selected Areas in Communications},
  year = {2025},
  doi = {10.1109/jsac.2025.3623183},
}
Figure