Unlocking Cross-Modal Biosignal Synthesis: A Temporally-Aware VAE-Diffusion Model

Chenyang Xu, Dezhen Wang, Hao WangICML 2026 · OpenReview

Synthesizing heart sounds (phonocardiograms, or PCG) from ubiquitous electrical heart signals (electrocardiograms, or ECG) represents a major leap forward for accessible cardiac monitoring. By combining the structural stability of Variational Autoencoders (VAEs) with the generative finesse of Diffusion models, this research bridges the gap between morphological realism and temporal precision, achieving state-of-the-art performance in cross-modal biosignal synthesis.

The Challenge: Electromechanical Coupling

In cardiac monitoring, ECG (electrical) and PCG (mechanical/acoustic) provide complementary diagnostic information. While ECG is easily captured via smartwatches and portable monitors, PCG requires specialized acoustic sensors and quiet environments. Synthesizing PCG directly from ECG enables dual-modality cardiac assessment without additional hardware.

However, capturing the heart's complex electromechanical coupling is notoriously difficult. Previous attempts generally fell into two traps:

  1. Regression-based models (e.g., Transformers): These achieve good temporal alignment but produce over-smoothed, "blurry" waveforms that lack the rich acoustic texture of real heart sounds.
  2. Pure Generative models (e.g., Diffusion): These generate realistic-sounding audio but struggle to stay perfectly aligned with the ECG trigger, leading to "hallucinated" or poorly timed heartbeats.

Methodology: Temporally-Aware VAE-Diffusion

To resolve the trade-off between temporal precision and physiological fidelity, the authors introduce a hybrid three-stage architecture:

  • Stage 1: Latent Manifold Learning (VAE): Instead of diffusing raw, noisy waveforms, a VAE learns a structured, low-dimensional latent space for PCG. This provides a "scaffold" that preserves local morphology.
  • Stage 2: Conditional Latent Diffusion: A 1D U-Net denoiser operates within this latent space, guided heavily by the input ECG.
  • Stage 3: Shared Decoder Reconstruction: A weight-shared decoder converts the denoised latent map back into a high-fidelity PCG waveform.

Key Architectural Innovations: * Enhanced Condition Fusion: Rather than simple concatenation, the model uses multi-scale convolutions and cross-attention to inject ECG features into the diffusion process, ensuring tight physiological alignment. * Temporal Attention Blocks: These explicitly model long-range cardiac dynamics, capturing the specific physiological delay between the electrical R-peak and the mechanical S1/S2 heart sounds. * Phased Training: The model employs a three-phase curriculum: VAE pre-training, latent diffusion training, and a crucial joint fine-tuning phase where all components are optimized together.

Datasets and Evaluation

The framework was rigorously evaluated across multiple datasets to ensure robustness:

  • EPHNOGRAM: The primary dataset for training and in-distribution testing, featuring simultaneous ECG and PCG recordings.
  • PhysioNet/CinC 2016 (MITHSDB subset): Used for zero-shot transfer evaluation. The model was tested on this synchronized dataset without any target-domain training to prove its robustness under domain shift, including on pathological recordings.
  • CirCor DigiScope: Utilized to pre-train the feature extractors for perceptual evaluation metrics (like Fréchet Distance).

Key Insights

  • Latent Space Efficiency: Ablation studies demonstrate that diffusing in the VAE latent space is significantly more effective than diffusing raw waveforms. The VAE handles the structural integrity while the diffusion model focuses on acoustic texture.
  • The Power of Joint Loss: Joint fine-tuning (Phase 3) is essential. Without it, the VAE latent space and the diffusion prior drift apart, leading to a sharp drop in performance across all metrics.
  • Physiological Inductive Biases: Explicitly modeling the temporal delays inherent in heart cycles via attention allows the model to generalize better to pathological cases (like arrhythmias) where heart rates are irregular.
  • Zero-Shot Robustness: The model maintains high waveform fidelity and clinically relevant timing structure on unseen datasets, suggesting it has learned the underlying physics of electromechanical coupling rather than just memorizing a specific dataset's artifacts.

Results and Performance

The Temporally-Aware VAE-Diffusion model sets a new state-of-the-art on the EPHNOGRAM benchmark, significantly outperforming leading diffusion and Transformer baselines:

  • Pearson Correlation: 0.910 ± 0.008.
  • S1 Detection Accuracy: 95.95%.
  • Timing Error: A precise 12.0 ms (drastically lower than the ~20 ms error of previous diffusion models).
  • Inference Efficiency: Using DDIM sampling, the model achieves an 8x speedup (down to ~54 ms per 12-second segment) with negligible loss in quality, making it highly viable for near-real-time applications.

Limitations and Future Directions

While the synthetic waveforms are morphologically and temporally accurate, the authors note that downstream diagnostic validation—proving these synthetic signals can diagnose specific pathologies (like murmurs) as accurately as real PCGs—remains an important next step. Additionally, broader validation across more diverse patient demographic populations and acquisition devices is needed. Future work will explore multi-modal synthesis (incorporating blood pressure or respiration) and model quantization for deployment on edge devices like smartwatches.

Summary updated

Abstract

Synthesizing authentic phonocardiograms (PCG) from ubiquitous electrocardiograms (ECG) is a critical task for accessible cardiac monitoring. Existing generative models, however, struggle to capture the heart's complex electromechanical coupling, failing to meet the dual requirements of temporal precision and physiological fidelity needed for clinically relevant waveform analysis. We introduce the Temporally-Aware VAE-Diffusion model, a synergistic hybrid architecture that resolves this trade-off. Our architecture enforces tight physiological coupling through an Enhanced Condition Fusion mechanism and explicitly models long-range cardiac dynamics via Temporal Attention Blocks. On the EPHNOGRAM benchmark, our model sets a new state-of-the-art, achieving a Pearson correlation of 0.910$\pm$0.008, 95.95\% S1 detection accuracy, and a precise 12.0 ms timing error, significantly outperforming leading diffusion and Transformer baselines. Crucially, our work provides a reproducible zero-shot transfer evaluation for ECG-to-PCG synthesis. Evaluated on the synchronized PhysioNet/CinC 2016 training-a/MITHSDB subset without target-domain training, our model preserves high waveform fidelity and clinically relevant timing structure under domain shift, including on pathological recordings. These results support cross-dataset robustness of the proposed synthesis framework, while downstream diagnostic validation remains an important direction for future work.
BibTeX
@inproceedings{xu2026unlocking,
  author = {Chenyang Xu and Dezhen Wang and Hao Wang},
  title = {{Unlocking Cross-Modal Biosignal Synthesis: A Temporally-Aware VAE-Diffusion Model}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=aComqAqP6j},
}
Figure