FusionCLIP: Multimodal Contrastive Pretraining for Fusion Plasma Diagnostics

Qiang Chen, Xiao Wang, Qingquan Yang, Hao Si, Zikang Yan, Guosheng Xu, Jin TangarXiv 2026 · arXiv

TL;DR

FusionCLIP is a multimodal contrastive pretraining framework designed to master fusion plasma diagnostics by aligning visible-light videos with multivariate time-series signals. By introducing specialized encoders to handle redundant video backgrounds and heterogeneous signal sampling rates, it achieves an 89.07% aggregate score across critical tasks like disruption prediction and confinement regression, significantly outperforming general-purpose AI backbones.

What Problem Does This Paper Address?

Monitoring a tokamak—a device used to harness nuclear fusion—requires synthesizing data from diverse diagnostic systems. Researchers primarily rely on two streams: visible-light videos, which show the spatial evolution of plasma structures, and multivariate diagnostic signals, which record physical quantities like density and current.

However, two major hurdles prevent standard machine learning models from effectively using this data:

  1. Visual Redundancy: In plasma videos, the actual "action" (the plasma) often occupies only a small fraction of the frame, surrounded by static machine backgrounds and repeated observations.
  2. Sampling Heterogeneity: Diagnostic signals are sampled at wildly different frequencies, ranging from 25 Hz to 250 kHz. Standard methods often resample these onto a fixed grid, which either blurs high-frequency details or creates artificial data points for low-frequency signals.

Existing fusion AI models usually focus on reconstructing data within a single modality. FusionCLIP instead asks: can we learn better representations by forcing the model to understand the relationship between what it sees in the video and what it measures in the signals?

Methods

The authors propose FusionCLIP, a framework that uses a bidirectional contrastive objective to align video and signal representations from the same time window.

Fig. 1: Overview of EAST-MMD and FusionCLIP. (a) Paired plasma videos with redundant backgrounds and diagnostic signals with heterogeneous sampling rates. (b) Spatiotemporal Stability Selection and Temporal Query Encoder encode the two modalities for contrastive pretraining, followed by independent fine-tuning for disruption prediction, ELM classification, and H98 regression.

Key innovations include:

  • Spatiotemporal Stability Selection (SSS): Instead of processing every pixel, SSS selects the most informative visual tokens based on three criteria: feature stability (robustness to high-frequency noise), spatial consistency (neighboring regions should behave similarly), and temporal persistence (structures should exist across multiple frames). This effectively "masks" the background and focuses the model on the plasma.
  • Temporal Query Encoder (TQE): To solve the sampling rate issue, TQE uses learnable "time queries." Instead of resampling the raw data, the model uses these queries to attend to diagnostic events based on their original timestamps, incorporating a time-distance bias to prioritize nearby signal changes.
Fig. 2: Overview of FusionCLIP. Spatiotemporal Stability Selection selects video tokens using feature stability, spatial consistency, and temporal persistence, while Temporal Query Encoder maps irregularly sampled diagnostic signals onto a shared temporal grid.

Datasets

The researchers constructed the EAST Multimodal Diagnostics (EAST-MMD) dataset, curated from 1,307 discharges of the Experimental Advanced Superconducting Tokamak (EAST).

  • Pretraining: 667 discharges, resulting in over 2.7 million paired video-signal windows.
  • Downstream Evaluation: 640 discharges used for three specific tasks: disruption prediction (safety), edge-localized mode (ELM) classification (plasma state), and $H_{98}$ regression (energy confinement efficiency).

Key Insights and Ideas

  • Background is Noise: Standard Vision Transformers often "waste" computation on background tokens. By explicitly selecting plasma-related regions via SSS, the model retains finer details of the plasma's evolution.
  • Time is a Feature, Not a Grid: Treating time as a continuous query rather than a fixed grid allows the model to respect the original physical resolution of high-frequency diagnostics, which is crucial for detecting fast-moving instabilities.
  • Cross-Modal Synergy: Pretraining by aligning video and signals provides a stronger supervisory signal than just reconstructing one or the other. The model learns that a specific visual "wobble" in the plasma corresponds to a specific spike in magnetic probe data.

Key Results

FusionCLIP achieved an aggregate score of 89.07% across the three downstream tasks.

  • Disruption Prediction: It reached a 93.88% F1-score and a 99.54% AUC, significantly reducing false alarms compared to models trained from scratch or using general-purpose backbones like DINO or MAE.
  • Efficiency: The SSS mechanism with a 50% mask ratio provided the best balance, outperforming models that used "full tokens" (no masking) by over 2 percentage points.
  • Qualitative Proof: Attention visualizations show that FusionCLIP focuses sharply on visible plasma structures, whereas standard CLIP-based models often get distracted by the tokamak's internal wall.
Comparison of attention maps showing FusionCLIP's focus on plasma vs. CLIP-Base's focus on background.

Limitations and Future Work

The primary limitation is that the model was trained and evaluated exclusively on data from the EAST tokamak. Because different fusion devices have different geometries and diagnostic setups, the "holy grail" is cross-device transferability. The authors suggest that future work will focus on testing how well these pretrained representations generalize to other tokamaks, which is essential for building a universal "foundation model" for nuclear fusion.

Summary updated

Abstract

Visible-light videos and multivariate diagnostic signals provide complementary observations of tokamak discharges. However, existing fusion pretraining methods, primarily learn representations by reconstructing or predicting observations, leaving the correspondence between these modalities underexplored. Representation learning is further challenged by the fact that plasma-related regions occupy only part of each video frame and diagnostic variables have substantially different sampling rates. We propose FusionCLIP, a multimodal contrastive pretraining framework. Spatiotemporal Stability Selection retains informative visual tokens, while a Temporal Query Encoder integrates asynchronous diagnostic signals using their observation timestamps. Contrastive learning then aligns the two modality representations. We construct EAST Multimodal Diagnostics (EAST-MMD), comprising 1307 EAST discharges, and evaluate the model on three independently fine-tuned tasks: disruption prediction, edge-localized mode (ELM) classification, and $H_{98}$ regression. FusionCLIP achieves an aggregate score of 89.07\%, exceeding the best general-purpose pretraining configuration and an in-domain contrastive pretraining configuration without Spatiotemporal Stability Selection by 6.27 and 2.12 percentage points, respectively. Ablations and attention visualizations further support the contributions of its key components.
BibTeX
@misc{chen2026fusionclip,
  author = {Qiang Chen and Xiao Wang and Qingquan Yang and Hao Si and Zikang Yan and Guosheng Xu and Jin Tang},
  title = {{FusionCLIP: Multimodal Contrastive Pretraining for Fusion Plasma Diagnostics}},
  year = {2026},
  eprint = {2611.09493},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.09493},
}
Figure