TL;DR
Predictive Coding Dynamics (PCD) introduces a powerful feedback mechanism for modeling irregular and sparse time series. By treating prediction errors as informative signals rather than just training losses, PCD iteratively refines a model's internal "latent state," leading to significantly more accurate trajectory reconstructions and forecasts.
What Problem Does This Paper Address?
Real-world data—from ICU heart rate monitors to weather stations—is rarely neat. Observations are often sparse, irregular, and asynchronous. While continuous-time models like Neural ODEs have become the standard for handling this messiness, they typically follow a "one-way" inference path: encode observations into a latent state, evolve that state forward, and decode the result.
The authors identify a major missed opportunity: these models don't "check their work" during inference. If a model’s predicted trajectory deviates from an actual observation, that error contains vital information about how the model should adjust its internal state. Standard models only use this error to update weights during training; PCD uses it to refine the state estimation itself.
Methods
PCD introduces an iterative Refinement Loop that can be plugged into existing backbones like ODE-RNN or Latent ODE.
The process works as follows:
- Reconstruction & Caching: The model generates an initial trajectory and stores it in a cache (using an exponential moving average).
- Error Calculation: A "PC Predictor" looks at the cached trajectory and calculates the signed residuals (the difference in both magnitude and direction) between the prediction and the actual observations at specific timestamps.
- Augmented Input: These residuals are masked (to ignore missing data) and weighted. They are then concatenated with the original observations to create an "augmented input."
- State Refinement: This augmented input is fed back into the backbone's encoder. The model now sees not just what happened, but how wrong it was in its previous guess, allowing it to correct its latent trajectory.
- Energy Penalty: During training, a new "PC Energy Loss" term constrains these weighted prediction errors, forcing the model to stay consistent with observed data.
Datasets
The authors evaluated PCD across a diverse range of domains:
- MuJoCo: A physical simulation (Hopper dynamics) used for forecasting with missing history and sparse trajectory recovery.
- PhysioNet 2012: Real-world clinical ICU measurements, characterized by high irregularity and noise.
- USHCN: Meteorological records for climate monitoring.
- Human Activity: Sensor data for motion forecasting.
- Darts: A benchmark for regular-grid time series forecasting to test generalizability.
Key Insights and Ideas
- Errors are Data: The core "aha moment" is that prediction residuals are not just failures; they are high-quality inference signals. A signed residual tells the model exactly which way to "nudge" its latent state to better match reality.
- Backbone Agnostic: PCD isn't a new model architecture; it's a mechanism. It successfully improved both event-driven models (ODE-RNN) and latent-variable models (Latent ODE).
- Signed vs. Unsigned: The paper finds that signed residuals are crucial. Knowing the direction of the error allows for precise state adjustment, whereas squared or absolute errors (which discard direction) perform significantly worse.
- Dynamics Matter: PCD excels in systems with deterministic, smooth dynamics (like physics). In highly stochastic environments (like clinical data), the benefits are more modest because errors might stem from random noise rather than a misestimated state.
Key Results
PCD demonstrated substantial improvements, particularly in sparse data scenarios.
- MuJoCo Forecasting: On the MuJoCo dataset, adding PCD to an ODE-RNN reduced the Mean Squared Error (MSE) from 0.3280 to 0.0047 in the regular setting. Even when 70% of historical data was removed, PCD maintained a low MSE of 0.0080.
- Trajectory Recovery: For Latent ODEs, PCD reduced reconstruction error by nearly 95% (from 0.285 to 0.0125 in $10^{-2}$ MSE units) at a 50% observation rate.
Limitations and Future Work
While powerful, PCD has some hurdles to clear:
- Noise Sensitivity: Because the feedback signal uses raw residuals, it can struggle to distinguish between a genuine "state drift" and simple sensor noise or outliers.
- Complexity of Dependencies: The current weighting system is relatively simple and doesn't fully account for complex, coupled dependencies between different variables in a multivariate system.
- Future Directions: The authors suggest moving toward probabilistic uncertainty modeling for errors to better handle noise. They also see potential in integrating PCD with diffusion models and large pre-trained time-series models.
Summary updated