Traditional signal filtering relies heavily on frequency separation. If a signal is corrupted by noise, we typically design a bandpass filter to isolate the frequencies of interest. However, in real-world scenarios—ranging from mechanical vibrations and biomedical readings to software system logs—noise and meaningful signals are often tightly intertwined within the same frequency bands, or the data itself is highly non-stationary.
To solve this, a powerful paradigm has emerged: information-theoretic noise pruning. Instead of asking how fast a signal oscillates (its frequency), this approach asks how predictable, ordered, or relevant the data is. By shifting the perspective from the frequency domain to the information domain, researchers leverage mathematical tools like Entropy and Mutual Information (MI) to objectively separate structural signals from stochastic chaos.
The Core Pipeline: Decompose, Quantify, Filter
Because raw data is usually a complex mixture of overlapping phenomena, information-theoretic denoising rarely operates on the raw signal directly. Instead, it follows a structured three-step pipeline:
[Raw Noisy Data]
│
▼
[Decomposition] ──► Breaks data into simpler sub-components (e.g., IMFs, Wavelets)
│
▼
[Information Metric] ──► Evaluates each component via Entropy or Mutual Information
│
▼
[Adaptive Threshold] ──► Prunes high-entropy or low-mutual-information noise
│
▼
[Reconstructed Signal] ──► Sums remaining clean components
- Decomposition: The raw signal is broken down into a finite set of simpler, localized components. This is frequently achieved using Empirical Mode Decomposition (EMD) or its noise-robust variants, such as Complete Ensemble EMD with Adaptive Noise (CEEMDAN). These methods extract Intrinsic Mode Functions (IMFs) that represent distinct oscillatory modes.
- Quantification: Each individual component is evaluated using an information-theoretic metric to determine its underlying order or its statistical dependence on the source.
- Filtering and Reconstruction: Components identified as noise-dominated are systematically pruned. The clean signal is then reconstructed by summing only the highly informative components.
This pipeline allows for fully unsupervised and adaptive filtering, requiring no prior knowledge of the signal's underlying structure or frequency distribution.
Sifting Through Chaos: Entropy as an Internal Filter
Entropy measures the uncertainty, randomness, or complexity within a data segment. When a complex signal is decomposed, components that consist primarily of random noise will exhibit high structural disorder. By quantifying this disorder, various flavors of entropy can serve as a precise sieve.
In mechanical fault diagnosis, for example, subtle signatures of failing components are easily masked by background noise. To isolate these signatures, Wenxiao Guo et al. utilized Multi-Scale Sample Entropy (MSE) to evaluate decomposed signal components. By establishing a dynamic threshold based on simulated random noise, the authors successfully identified and discarded components whose complexity exceeded acceptable limits. Similarly, Chengwei Li et al. applied generalized refined composite multiscale sample entropy to isolate early rolling-bearing faults under severe noise conditions, demonstrating that robust entropy measures can reliably extract weak physical signatures.
Different types of entropy capture different structural properties of a signal:
- Permutation Entropy (PE): Focuses on the order relations between continuous values. Jianhua Liu et al. applied PE to axle box vibration data, treating high permutation entropy as an explicit signature of irrelevant noise and discarding those components to cleanly isolate railway rail corrugation.
- Fuzzy Entropy: Introduces fuzzy boundaries to measure similarity. Chengwei Li demonstrated that tracking the significant changes in fuzzy entropy between adjacent, sorted IMFs reveals a clear boundary between impact signals and background noise.
- Shannon Entropy: Measures the concentration of a probability distribution. In acoustic analysis, Norman Casagrande showed that finding a musical meter requires distinguishing ordered rhythmic structures from disordered noise. By calculating the Shannon entropy of a signal's phase distribution, the author demonstrated that a true periodic beat results in a highly concentrated phase distribution (low entropy), while random noise spreads out evenly (high entropy). Weighting the signal by the inverse of its entropy sharpens true periodicities and suppresses spurious noise.
This principle extends to spatial data as well. In neuroimaging, Mustafa S. Çetin et al. integrated an entropy-filtering mechanism into fMRI functional template generation. By prioritizing high-information voxels and filtering out low-entropy regions, the authors denoised the functional reference template, significantly boosting the accuracy of downstream disease classification.
Measuring Relevance: Mutual Information as an External Filter
While entropy evaluates a component's internal randomness, Mutual Information (MI) measures the external statistical dependence between two distinct variables. In a denoising context, MI answers a critical question: How much unique information does this decomposed component actually share with the original signal or the target goal?
When a signal is broken down into IMFs, some components capture the core structural dynamics of the system, while others capture purely localized, stochastic fluctuations. Long Han et al. and Yaan Li et al. demonstrated that calculating the MI between each individual IMF and the raw, un-decomposed signal provides an objective metric for component selection. Components that yield low MI share negligible statistical dependency with the source and can be safely discarded as noise.
[Decomposed Waveforms] ──► IMF 1 ──► High MI with Source ──► KEEP
──► IMF 2 ──► High MI with Source ──► KEEP
──► IMF 3 ──► Sharp Drop in MI ──► PRUNE (Noise Transition)
──► IMF 4 ──► Low MI with Source ──► PRUNE
Crucially, MI can identify the exact mathematical boundary where a signal transitions from informative to noisy. Chengwei Li demonstrated that by analyzing the mutual information between consecutive decomposed waveforms, a sharp drop in MI serves as an indicator of the transition from signal-dominant to noise-dominant modes. This allows for adaptive signal reconstruction without requiring any prior assumptions about the noise level. A similar strategy of correlating and evaluating individual components against the raw signal was utilized by Tan Lee 李丹 et al. to isolate highly informative brain-wave channels in multi-channel EEG data.
Beyond Waveforms: Generalizing to Discrete Data and Search Spaces
The intuition behind information-theoretic noise pruning is not restricted to continuous physical waveforms; it applies equally well to discrete event logs and large-scale data mining.
In software engineering, log-based anomaly detection models often suffer from "data obesity," where massive volumes of routine system logs obscure actual security or operational threats. Lingzhe Zhang et al. addressed this by using Mutual Information to quantify the correlation between specific log events and system state labels (normal vs. anomalous). Events that appear with identical frequencies in both normal and anomalous states possess zero discriminative power—they are statistical noise. By pruning these non-informative "anti-events," the authors reduced log volumes by over 70% while simultaneously eliminating false positives and boosting model accuracy.
Log Event A: Occurs only during crashes ──► High MI ──► Retain
Log Event B: Occurs equally in normal & crash ──► Zero MI ──► Prune (Noise)
Information-theoretic pruning also provides a theoretical foundation for accelerating complex computations. When searching for non-linear, multi-scale time-delay correlations across massive datasets, the search space can quickly become computationally intractable. Nguyen Ho et al. formulated a noise pruning theory defining "noise" as any data segment that decreases the total mutual information of a given data window. This principle provides a mathematically rigorous stopping criterion: if expanding a data partition dilutes the overall statistical signal, the algorithm immediately discards that region of the search space. This prevents the algorithm from wasting resources on unpromising data segments, yielding massive computational speedups without sacrificing accuracy.
Why It Works
Information-theoretic noise pruning succeeds because it treats noise as a fundamental property of data distribution rather than a specific frequency or energy threshold. Whether analyzing geophysical data via multiscale dispersion entropy (Jiangtao Han et al.) or filtering out non-discriminative system events, the core mechanism remains unified:
- Entropy exposes internal chaos, allowing systems to drop components that lack structural order.
- Mutual Information exposes external irrelevance, allowing systems to drop components that do not share a meaningful statistical relationship with the target phenomenon.
By formalizing "noise" as a lack of information content, this paradigm provides a robust, mathematically principled framework for cleaning data across diverse scientific and engineering domains.
Key Foundational Work
Finding Meter in Music Using An Autocorrelation Phase Matrix and Shannon Entropy (2005) — contributed 4 concept elements to this direction.
Friction Signal Denoising Using Complete Ensemble EMD with Adaptive Noise and Mutual Information (2015) — contributed 2 concept elements to this direction.
Feature Extraction Method of Rolling Bearing Fault Signal Based on EEMD and Cloud Model Characteristic Entropy (2015) — contributed 1 concept elements to this direction.