Fine-Tuning Vision-Language Models for Neutrino Event Analysis in High-Energy Physics Experiments

Dikshant Sagar, Kaiwen Yu, Alejandro Yankelevich, Jianming Bian, Pierre BaldiarXiv 2025 · DOI · arXiv

Introduction

In high-energy physics (HEP), identifying specific neutrino interactions is a fundamental challenge. Experiments like the Deep Underground Neutrino Experiment (DUNE) utilize Liquid Argon Time Projection Chambers (LArTPCs) to produce high-resolution, sparse 2D pixelated "photographs" of subatomic collisions. Distinguishing between electron neutrino charged current ($\nu_e$ CC), muon neutrino charged current ($\nu_mu$ CC), and neutral current (NC) interactions is critical for measuring neutrino oscillation parameters.

Traditionally, physicists have relied on hand-engineered features or Convolutional Neural Networks (CNNs). While CNNs are effective, they often function as "black boxes," providing classifications without physical justification. This research explores the application of Vision-Language Models (VLMs) to bridge this gap, combining visual pattern recognition with the reasoning capabilities of Large Language Models (LLMs) to provide "explainable physics."

Methods and Architecture

The researchers utilized LLaMA 3.2 Vision 11B, a multimodal model capable of simultaneous image and text processing. To adapt this general-purpose model to specialized physics data efficiently, the team employed QLoRA (Quantized Low-Rank Adaptation). This technique allows for fine-tuning by freezing the base model and training a small set of 4-bit quantized adapter weights, making the process feasible on modest hardware.

Figure 1: Workflow of the supervised fine-tuning and inference process for neutrino event classification using LLaMA 3.2 Vision and QLoRA.

As shown in Figure 1, the model is provided with a system prompt describing the physical characteristics of different interactions (e.g., "fuzzy electron showers" for $\nu_e$ CC or "long tracks" for $\nu_mu$ CC). The input consists of dual-view pixel maps (XZ and YZ planes). To ensure machine-readable outputs, a phrasal constraint was implemented, forcing the model to begin its response with a specific prefix followed by the classification label.

Dataset and Simulation

The study utilized a simulated dataset of 190,000 labeled events generated via GENIE and GEANT4 to mimic LArTPC detector readouts. Each event is represented by two $512 \times 512$ grayscale images. The dataset includes three primary interaction classes: $\nu_e$ CC, $\nu_mu$ CC, and NC. The VLM was compared against a Siamese CNN baseline (3.4M parameters), a standard architecture for processing multi-view detector data.

Results and Performance

The fine-tuned VLM significantly outperformed the CNN baseline across all evaluated metrics. The VLM achieved an overall accuracy of 0.87, compared to 0.68 for the CNN. Notably, the VLM showed superior performance in discriminating Neutral Current (NC) events, which are traditionally difficult to distinguish from signal events.

Figure 4: ROC curves comparing LLaMA 3.2 Vision and the CNN baseline across (a) muon neutrino, (b) electron neutrino, and (c) neutral current interactions.

As illustrated in Figure 4, the VLM achieved an Area Under the Curve (AUC) of 0.97 for both $\nu_mu$ CC and $\nu_e$ CC, and 0.95 for NC events, consistently exceeding the CNN's performance (AUC 0.90–0.94). This suggests that the VLM's ability to integrate semantic physical descriptions from the system prompt enhances its classification accuracy on sparse data.

Metric LLaMA 3.2 Vision CNN Baseline
Accuracy 0.87 0.68
Precision 0.87 0.78
Recall 0.87 0.78
AUC-ROC 0.96 0.93

Discussion and Limitations

While the VLM offers superior accuracy and the potential for natural-language rationales (e.g., describing a "fuzzy shower"), it introduces significant computational overhead. The VLM requires approximately 25.4 GB of VRAM and 3.3 seconds per sample for inference. In contrast, the CNN baseline requires only 1.0 GB of VRAM and processes samples in 20 milliseconds. This latency makes the current VLM approach unsuitable for real-time "online" triggers but highly valuable for "offline" analysis where interpretability and precision are paramount.

Conclusion

This work demonstrates that general-purpose Vision-Language Models can be successfully adapted to highly specialized scientific domains. By outperforming domain-specific CNNs, VLMs prove to be powerful backbones for physics analysis. Future research will focus on model distillation and the development of domain-specific foundation models to reduce computational costs while maintaining the interpretability benefits of the VLM approach.

Summary updated

Abstract

Recent progress in large language models (LLMs) has shown strong potential for multimodal reasoning beyond natural language. In this work, we explore the use of a fine-tuned Vision-Language Model (VLM), based on LLaMA 3.2, for classifying neutrino interactions from pixelated detector images in high-energy physics (HEP) experiments. We benchmark its performance against an established CNN baseline used in experiments like NOvA and DUNE, evaluating metrics such as classification accuracy, precision, recall, and AUC-ROC. Our results show that the VLM not only matches or exceeds CNN performance but also enables richer reasoning and better integration of auxiliary textual or semantic context. These findings suggest that VLMs offer a promising general-purpose backbone for event classification in HEP, paving the way for multimodal approaches in experimental neutrino physics.
BibTeX
@misc{sagar2025fine,
  author = {Dikshant Sagar and Kaiwen Yu and Alejandro Yankelevich and Jianming Bian and Pierre Baldi},
  title = {{Fine-Tuning Vision-Language Models for Neutrino Event Analysis in High-Energy Physics Experiments}},
  year = {2025},
  eprint = {2508.19376},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.19376},
}
Figure