Distributed Compressive Sensing: A Deep Learning Approach

Hamid Palangi, Rabab K. Ward, Li Deng

This paper presents a novel intersection between signal processing and deep learning, specifically targeting the Multiple Measurement Vector (MMV) problem in Compressive Sensing.

TL;DR

The authors introduce LSTM-CS, a deep learning-based reconstruction algorithm that uses Long Short-Term Memory (LSTM) networks to learn and exploit unknown dependencies between related sparse signals. By replacing the hard-coded "joint sparsity" assumption of traditional solvers with a data-driven model, they achieve significantly higher reconstruction accuracy and better noise robustness while maintaining the high speed of greedy algorithms.

What Problem Does This Paper Address?

Compressive Sensing (CS) allows us to reconstruct a signal from far fewer samples than the Nyquist rate, provided the signal is "sparse" (mostly zeros in some domain). In Distributed Compressive Sensing (DCS), we try to recover a set of related signals simultaneously.

The traditional "gold standard" assumption is joint sparsity: the idea that all signals in a set share the exact same non-zero locations (support). However, in real-world applications like video frames or multi-channel sensor data, signals are often related but not perfectly jointly sparse. Traditional algorithms (like SOMP or Bayesian methods) struggle when this strict assumption is violated. The authors tackle the challenge of capturing these complex, unknown dependencies without increasing the complexity of the encoder (the sensor).

Methods

The authors propose a hybrid approach called LSTM-CS, which embeds a deep neural network inside a greedy reconstruction framework.

  1. The Architecture: They use a Recurrent Neural Network (RNN) with LSTM cells followed by a softmax layer. LSTMs are specifically chosen for their ability to model long-term dependencies in sequences.
  2. The Greedy Loop: Instead of picking the most correlated "atom" from a dictionary (as in Orthogonal Matching Pursuit), the algorithm uses the LSTM to predict the conditional probability of an entry being non-zero, given the "residuals" (the unexplained part of the signal) from all previous channels.
  3. Two-Step Reconstruction:
    • Step 1 (Support Identification): The LSTM identifies the most likely location of a non-zero entry.
  4. Step 2 (Value Estimation): A standard Least Squares solver calculates the actual values for the identified support.
  5. Training: The model is trained using a cross-entropy cost function on a dataset of residuals and their corresponding ground-truth supports. Crucially, the training happens offline; the decoder just runs a forward pass during actual signal recovery.

Datasets

The researchers evaluated their approach on two primary types of data:

  • MNIST: The classic dataset of handwritten digits. They simulated a 4-channel system where each channel was a different digit (0, 1, 2, and 3).
  • MSR Cambridge Natural Image Dataset: They used images of buildings, cows, and flowers. These were made sparse using standard transforms like the Discrete Cosine Transform (DCT) and Wavelets.

Key Insights and Ideas

  • Learning vs. Assuming: The core "aha" moment is that we don't need to mathematically define the relationship between signals if we can learn it from data. The LSTM acts as a universal dependency learner.
  • Residuals as Input: By feeding the network the residuals of the signal, the model learns to focus on the information that hasn't been captured yet, effectively "cleaning up" the estimation at each step.
  • Preserving the CS Advantage: A major benefit of Compressive Sensing is the "dumb sensor, smart decoder" paradigm. LSTM-CS keeps the encoder as a simple random matrix multiplication (low power), moving all the heavy computational lifting to the decoder.
  • Greedy Efficiency: By keeping the overall structure of a greedy solver, the algorithm remains computationally efficient enough for practical use, unlike some heavy Bayesian optimization methods.

Key Results

  • Superior Accuracy: LSTM-CS significantly outperformed standard baselines like SOMP and state-of-the-art Bayesian methods (MT-BCS, T-SBL) across all datasets.
  • Noise Robustness: In high-noise scenarios (SNR of 6 dB), the performance of traditional Bayesian methods degraded sharply, while LSTM-CS remained remarkably stable.
  • Phase Transition: The model achieved "perfect recovery" at much lower measurement rates than competing algorithms. For example, on MNIST, it could recover signals with high accuracy using only 25-50% of the measurements required by other solvers.
  • Speed: The inference time was found to be nearly as fast as traditional greedy methods, making it orders of magnitude faster than complex Bayesian iterations.

Limitations and Future Work

  • Data Requirement: As a supervised learning method, LSTM-CS requires a large, representative training dataset for the specific class of signals it will encounter. It cannot generalize to completely unseen signal types as well as "model-based" methods.
  • Fixed Channel Order: The current implementation treats the channels as a sequence; the authors suggest that a bidirectional LSTM might better capture dependencies by looking at the "future" and "past" channels simultaneously.
  • Future Directions: The authors propose extending this work to non-linear compressive sensing, video compression (where temporal dependency is high), and multi-channel EEG signal processing.

Summary updated

Abstract

Various studies that address the compressed sensing problem with Multiple Measurement Vectors (MMVs) have been recently carried. These studies assume the vectors of the different channels to be jointly sparse. In this paper, we relax this condition. Instead we assume that these sparse vectors depend on each other but that this dependency is unknown. We capture this dependency by computing the conditional probability of each entry in each vector being non-zero, given the "residuals" of all previous vectors. To estimate these probabilities, we propose the use of the Long Short-Term Memory (LSTM)[1], a data driven model for sequence modelling that is deep in time. To calculate the model parameters, we minimize a cross entropy cost function. To reconstruct the sparse vectors at the decoder, we propose a greedy solver that uses the above model to estimate the conditional probabilities. By performing extensive experiments on two real world datasets, we show that the proposed method significantly outperforms the general MMV solver (the Simultaneous Orthogonal Matching Pursuit (SOMP)) and a number of the model-based Bayesian methods. The proposed method does not add any complexity to the general compressive sensing encoder. The trained model is used just at the decoder. As the proposed method is a data driven method, it is only applicable when training data is available. In many applications however, training data is indeed available, e.g. in recorded images and videos.
BibTeX
@article{palangi2016distributed,
  author = {Hamid Palangi and Rabab K. Ward and Li Deng},
  title = {{Distributed Compressive Sensing: A Deep Learning Approach}},
  journal = {IEEE Trans. Signal Process.},
  year = {2016},
  doi = {10.1109/tsp.2016.2557301},
}
Figure