TL;DR
FusionCLIP is a multimodal contrastive pretraining framework designed to master fusion plasma diagnostics by aligning visible-light videos with multivariate time-series signals. By introducing specialized encoders to handle redundant video backgrounds and heterogeneous signal sampling rates, it achieves an 89.07% aggregate score across critical tasks like disruption prediction and confinement regression, significantly outperforming general-purpose AI backbones.
What Problem Does This Paper Address?
Monitoring a tokamak—a device used to harness nuclear fusion—requires synthesizing data from diverse diagnostic systems. Researchers primarily rely on two streams: visible-light videos, which show the spatial evolution of plasma structures, and multivariate diagnostic signals, which record physical quantities like density and current.
However, two major hurdles prevent standard machine learning models from effectively using this data:
- Visual Redundancy: In plasma videos, the actual "action" (the plasma) often occupies only a small fraction of the frame, surrounded by static machine backgrounds and repeated observations.
- Sampling Heterogeneity: Diagnostic signals are sampled at wildly different frequencies, ranging from 25 Hz to 250 kHz. Standard methods often resample these onto a fixed grid, which either blurs high-frequency details or creates artificial data points for low-frequency signals.
Existing fusion AI models usually focus on reconstructing data within a single modality. FusionCLIP instead asks: can we learn better representations by forcing the model to understand the relationship between what it sees in the video and what it measures in the signals?
Methods
The authors propose FusionCLIP, a framework that uses a bidirectional contrastive objective to align video and signal representations from the same time window.
Key innovations include:
- Spatiotemporal Stability Selection (SSS): Instead of processing every pixel, SSS selects the most informative visual tokens based on three criteria: feature stability (robustness to high-frequency noise), spatial consistency (neighboring regions should behave similarly), and temporal persistence (structures should exist across multiple frames). This effectively "masks" the background and focuses the model on the plasma.
- Temporal Query Encoder (TQE): To solve the sampling rate issue, TQE uses learnable "time queries." Instead of resampling the raw data, the model uses these queries to attend to diagnostic events based on their original timestamps, incorporating a time-distance bias to prioritize nearby signal changes.
Datasets
The researchers constructed the EAST Multimodal Diagnostics (EAST-MMD) dataset, curated from 1,307 discharges of the Experimental Advanced Superconducting Tokamak (EAST).
- Pretraining: 667 discharges, resulting in over 2.7 million paired video-signal windows.
- Downstream Evaluation: 640 discharges used for three specific tasks: disruption prediction (safety), edge-localized mode (ELM) classification (plasma state), and $H_{98}$ regression (energy confinement efficiency).
Key Insights and Ideas
- Background is Noise: Standard Vision Transformers often "waste" computation on background tokens. By explicitly selecting plasma-related regions via SSS, the model retains finer details of the plasma's evolution.
- Time is a Feature, Not a Grid: Treating time as a continuous query rather than a fixed grid allows the model to respect the original physical resolution of high-frequency diagnostics, which is crucial for detecting fast-moving instabilities.
- Cross-Modal Synergy: Pretraining by aligning video and signals provides a stronger supervisory signal than just reconstructing one or the other. The model learns that a specific visual "wobble" in the plasma corresponds to a specific spike in magnetic probe data.
Key Results
FusionCLIP achieved an aggregate score of 89.07% across the three downstream tasks.
- Disruption Prediction: It reached a 93.88% F1-score and a 99.54% AUC, significantly reducing false alarms compared to models trained from scratch or using general-purpose backbones like DINO or MAE.
- Efficiency: The SSS mechanism with a 50% mask ratio provided the best balance, outperforming models that used "full tokens" (no masking) by over 2 percentage points.
- Qualitative Proof: Attention visualizations show that FusionCLIP focuses sharply on visible plasma structures, whereas standard CLIP-based models often get distracted by the tokamak's internal wall.
Limitations and Future Work
The primary limitation is that the model was trained and evaluated exclusively on data from the EAST tokamak. Because different fusion devices have different geometries and diagnostic setups, the "holy grail" is cross-device transferability. The authors suggest that future work will focus on testing how well these pretrained representations generalize to other tokamaks, which is essential for building a universal "foundation model" for nuclear fusion.
Summary updated