Adapting Vision-Language Models for Neutrino Event Classification in High-Energy Physics

Dikshant Sagar, Kaiwen Yu, Alejandro Yankelevich, Jianming Bian, Pierre BaldiarXiv 2025 · DOI · arXiv

Introduction

In high-energy physics (HEP), experiments such as the Deep Underground Neutrino Experiment (DUNE) generate vast amounts of high-dimensional data. A fundamental task in analyzing this data is event classification: identifying neutrino flavors (electron, muon, or tau) based on the visual patterns of particle interactions within a detector. While Convolutional Neural Networks (CNNs) have become a standard tool for this task, they often function as "black boxes," lacking interpretability and showing sensitivity to changes in data resolution or detector conditions. This paper explores the adaptation of Vision-Language Models (VLMs) to provide a more robust and explainable alternative.

Methodology

The researchers adapted LLaMA 3.2 Vision 11B, a multimodal large language model, for neutrino event classification using a specialized fine-tuning and prompting pipeline.

The supervised fine-tuning pipeline for LLaMA 3.2 Vision using QLoRA and specific physics-based prompts.

Fine-tuning and Prompt Engineering

To manage the computational demands of an 11-billion-parameter model, the team employed Quantized Low-Rank Adaptation (QLoRA). This allowed for efficient training on consumer-grade hardware by updating only a small subset of the model's weights. The model was guided by a "System Prompt" that established its persona as a neutrino physicist, providing specific criteria for classification (e.g., identifying "fuzzy electron showers" for $\nu_e$ CC events). To ensure structured output, constrained decoding was used to force the model to lead its responses with a standardized classification prefix.

Experimental Setup

The study utilized a simulated dataset from a Liquid Argon Time Projection Chamber (LArTPC), consisting of 190,000 events across three categories: Electron neutrino ($\nu_e$) Charged Current, Muon neutrino ($\nu_\mu$) Charged Current, and Neutral Current (NC) interactions. The input data consisted of dual 512x512 grayscale pixel maps representing orthogonal XZ and YZ views of the particle tracks. A Siamese CNN architecture served as the baseline for comparison.

Results and Discussion

The fine-tuned VLM demonstrated significant improvements over traditional methods in both performance and interpretability.

Performance and Robustness

LLaMA 3.2 Vision achieved an accuracy of 87%, significantly higher than the Siamese CNN's 68%. More importantly, the VLM exhibited remarkable robustness. When tested on downsampled 256x256 images, the CNN's accuracy dropped precipitously to 49%, whereas the VLM maintained an accuracy of 85%. This suggests that the VLM leverages high-level semantic understanding of particle topologies rather than relying solely on local pixel patterns.

Explainability

A key advantage of the VLM approach is its ability to generate natural language justifications for its classifications. By articulating the features it identifies—such as the length and narrowness of a muon track—the model allows physicists to verify the reasoning behind a prediction, bridging the gap between raw data and scientific insight.

An example of the fine-tuned model's natural language explanation for a Muon neutrino classification.

Conclusion

While the VLM offers superior accuracy and robustness, it introduces a significant computational trade-off, requiring substantially more VRAM and longer inference times than CNNs. Future work will focus on model distillation and the development of domain-specific foundation models to bring these advanced reasoning capabilities into real-time experimental triggers.

Summary updated

Abstract

Recent advances in Large Language Models (LLMs) have demonstrated their remarkable capacity to process and reason over structured and unstructured data modalities beyond natural language. In this work, we explore the applications of Vision Language Models (VLMs), specifically a fine-tuned variant of LLaMA 3.2 to the task of identifying neutrino interactions in pixelated detector data from high-energy physics (HEP) experiments. We benchmark this model against a state-of-the-art convolutional neural network (CNN) architecture, similar to those used in major neutrino experiments, which have achieved high efficiency and purity in classifying electron and muon neutrino events, and also a Vision Transformer (ViT-h/14), which is the same architecture inside the VLM's vision encoder. Our evaluation considers both classification performance and interpretability of the model predictions, comparing a VLM with a vision-only transformer (ViT) and a convolutional neural network (CNN) baseline. We find that transformer-based architectures outperform conventional CNNs in classification accuracy and robustness, with the VLM providing additional flexibility through the integration of auxiliary textual or semantic information and enabling more interpretable, reasoning-based predictions. These results highlight the potential of large transformer models, particularly vision-language models, as general-purpose backbones for physics event classification, combining strong performance, robustness, and interpretability, and opening new avenues for multimodal reasoning in experimental neutrino physics.
BibTeX
@misc{sagar2025adapting,
  author = {Dikshant Sagar and Kaiwen Yu and Alejandro Yankelevich and Jianming Bian and Pierre Baldi},
  title = {{Adapting Vision-Language Models for Neutrino Event Classification in High-Energy Physics}},
  year = {2025},
  eprint = {2509.08461},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.08461},
}
Figure