Sampling stationary signals subject to bitrate constraints

Alon Kipnis, Yonina C. Eldar, Andrea J. Goldsmith

Introduction

Classical sampling theory, established by Shannon, Whittaker, and Kotelnikov, provides a definitive condition for perfect signal reconstruction: a bandlimited signal must be sampled at at least twice its highest frequency component, known as the Nyquist rate. However, this theoretical benchmark assumes infinite-precision samples. In practical digital systems, samples must be quantized into a finite number of bits, introducing distortion. Under a fixed bitrate constraint $R$, a fundamental trade-off emerges: sampling faster provides more data points but reduces the bits available per sample, whereas sampling slower allows for higher-precision quantization of fewer points. This paper characterizes the fundamental limits of this trade-off within a framework termed Analog-to-Digital Compression (ADX).

The Analog-to-Digital Compression (ADX) Framework

The authors model the problem as a joint sampling and source-coding task for stationary Gaussian processes. The ADX system consists of three primary stages: a sampler, an encoder, and a decoder.

The Analog-to-Digital Compression (ADX) system model, illustrating the transition from continuous-time analog signals to discrete-time samples, and finally to a digital bitstream.

The framework considers a broad class of bounded linear samplers, including uniform sampling, filter-banks, and non-uniform sampling. The goal is to minimize the Mean Squared Error (MSE) between the original signal $X(\cdot)$ and its reconstruction $\hat{X}(\cdot)$, given a sampling frequency $f_s$ and a bitrate $R$. By applying information-theoretic rate-distortion theory—specifically Kolmogorov’s "water-filling" approach—the authors determine the optimal allocation of bits across the signal's power spectral density (PSD).

Key Insights: The Critical Frequency $f_R$

The most significant finding is the existence of a "critical sampling frequency" $f_R$. For a given bitrate $R$, $f_R$ represents the minimum sampling rate required to achieve the lowest possible distortion $D_X(R)$. Crucially, $f_R$ is typically strictly smaller than the Nyquist rate $f_{\text{Nyq}}$, unless the signal's spectrum is perfectly flat.

Distortion $D^*(f_s, R)$ and MMSE as a function of sampling frequency $f_s$. The plot shows that the minimum distortion is achieved at $f_R$, which can be significantly lower than the Nyquist rate $f_{\text{Nyq}}$.

As shown in the figure above, sampling at any rate $f_s \geq f_R$ allows the system to achieve the same minimal distortion. This occurs because optimal sampling "aligns" with the signal's energy; the system ignores low-energy frequency bands that would require more bits than they are worth in terms of distortion reduction. For non-bandlimited signals, such as the Gauss-Markov process, the paper proves that a finite sampling rate is always optimal for any finite bitrate, despite the Nyquist rate being technically infinite.

Practical Implications and the Penalty of Oversampling

Beyond the theoretical optimum, the paper analyzes practical Pulse-Code Modulation (PCM) systems. In these sub-optimal architectures, oversampling is not merely redundant but can be actively detrimental. In PCM, as the sampling rate increases beyond an optimal point, the bits must be spread across more samples. The resulting increase in quantization noise often outweighs the gains from denser sampling, leading to a "U-shaped" distortion curve. This confirms that for real-world ADCs operating under bitrate constraints, "faster is not always better."

Conclusion

This work establishes that the Nyquist rate is not a fundamental requirement for lossy compression of analog signals. By identifying the critical frequency $f_R$, the authors provide a theoretical basis for designing more efficient sampling systems that prioritize quantization precision over sampling density. Future research may extend these results to non-Gaussian or sparse signal models and explore non-linear sampling architectures.

Summary updated

BibTeX
@article{kipnis2018sampling,
  author = {Alon Kipnis and Yonina C. Eldar and Andrea J. Goldsmith},
  title = {{Sampling stationary signals subject to bitrate constraints}},
  journal = {IEEE Trans. Information Theory},
  year = {2018},
  doi = {10.1109/tit.2018.2857822},
}
Figure