Abstract
Classical state estimation, exemplified by the Kalman filter, focuses on minimizing Mean Squared Error (MSE), often resulting in "blurry" reconstructions that lack natural statistical characteristics. While the perception-distortion tradeoff is well-documented for static signals, its application to causal, online systems introduces a unique challenge: temporal consistency. This paper introduces the Perceptual Kalman Filter (PKF), a theoretical framework that guarantees perfect perceptual quality in real-time. By introducing an "unutilized information process," the PKF ensures that reconstructed sequences remain consistent with natural dynamics, revealing a fundamental "temporal consistency dilemma" where realism necessitates a deliberate, temporary increase in estimation error.
1. Introduction
In signal processing and control theory, the gold standard for linear state estimation is the Kalman filter, which provides the optimal solution for minimizing Mean Squared Error (MSE). However, in applications where the output is consumed by human observers—such as video restoration or audio enhancement—minimizing MSE often leads to undesirable artifacts. Because the MSE-optimal estimate is a conditional mean, it tends to produce "average" results that appear blurry or muted compared to natural signals.
Recent research has formalized this as the perception-distortion tradeoff, proving that as one approaches perfect perceptual quality (where the reconstruction is statistically indistinguishable from the ground truth), the minimum achievable distortion (MSE) must increase. While this tradeoff is well-understood for static tasks like image denoising, it has remained largely unexplored in causal/online settings. In these scenarios, a filter must process data frame-by-frame without knowledge of the future, while ensuring the entire sequence—not just individual frames—looks natural.
2. The Challenge of Temporal Consistency
The primary obstacle in online perceptual filtering is maintaining temporal consistency. If a filter attempts to satisfy a perceptual constraint on each frame independently, the resulting sequence often suffers from "flicker" or jitter. This occurs because the errors made at each time step are not coordinated. To appear realistic to a human observer, a reconstructed video or audio stream must follow the underlying laws of physics or dynamics inherent in the original signal. This requires the entire distribution of the estimated sequence to match the distribution of the ground-truth process over time.
3. The Perceptual Kalman Filter (PKF)
The authors propose the Perceptual Kalman Filter (PKF) to address this challenge. The PKF extends the classical Kalman framework by imposing a "perfect perceptual-quality constraint": the probability distribution of the estimated sequence must be identical to the distribution of the ground-truth signal.
3.1 The Unutilized Information Process
The core theoretical innovation of the PKF is the unutilized information process. In a standard Kalman filter, new measurements are incorporated via an "innovation process"—the difference between the observed data and the filter's prediction. The PKF, however, cannot always use this innovation immediately without violating the perceptual constraint.
Instead, the PKF tracks information that has been revealed by observations but not yet used by the filter. This unutilized information is stored in a buffer and introduced into the state estimate gradually. This mechanism ensures that the output transitions smoothly and adheres to natural dynamics, even when new data suggests the current estimate is incorrect.
3.2 Recursive Implementation
Unlike many perceptual optimization problems that require computationally expensive batch processing, the authors derive a recursive, closed-form solution for the PKF. This allows for efficient, real-time estimation where filter gains are calculated dynamically to balance new evidence with the requirement for statistical realism.
4. Experimental Evaluation
The PKF was evaluated using two distinct models to test its analytical and practical performance:
- Harmonic Oscillator: A synthetic 2D system representing position and velocity. This setup allowed the authors to precisely measure the MSE gap between the PKF and the standard Kalman filter, verifying the theoretical perception-distortion tradeoff in a controlled environment.
- Dynamic Texture (Lake Waves): To test the filter on complex, high-dimensional data, the authors modeled a video of lake waves using a 128-dimensional latent space. The task involved restoring the video from noisy, downsampled observations or predicting future frames.
5. Key Insights: The Temporal Consistency Dilemma
The study reveals a critical phenomenon termed the Temporal Consistency Dilemma. When a perceptual filter realizes it has made an error (e.g., it "hallucinated" a wave moving in the wrong direction based on previous noisy data), it faces a choice. A standard Kalman filter would immediately correct the state to minimize MSE, causing a visual "jump" or glitch.
The PKF, constrained to be perceptually perfect, cannot jump. To remain consistent with its own past decisions and the natural dynamics of the system, it must "stick to its guns" and transition back to the correct state slowly. This results in a significantly higher MSE than the standard filter, particularly in the moments following a measurement update after a gap. This "cost of consistency" is the necessary price for maintaining a realistic, flicker-free output in a causal system.
6. Results and Conclusion
Qualitatively, the PKF outperforms both the standard Kalman filter and "inconsistent" perceptual filters. In video tasks, the Kalman filter eventually produces a blurry, static average as uncertainty grows. Inconsistent perceptual filters produce sharp but wildly flickering frames. The PKF, however, maintains both spatial detail and natural motion, producing a sequence that continues to look like moving waves.
Quantitatively, the PKF's Root MSE on the harmonic oscillator was approximately 30% higher than a non-recursive perceptual optimizer, highlighting the additional difficulty of maintaining realism without seeing the future.
While currently limited to linear-Gaussian systems, the PKF framework provides a foundational theory for the next generation of generative models. The authors suggest that the "unutilized information" concept could be integrated into recurrent neural networks (RNNs) or diffusion models to solve the persistent problem of temporal instability in AI-generated video.
Summary updated