Improving Large Vision-Language Models' Understanding for Field Data

Xiaomei Zhang, Hanyu Zheng, Xiangyu Zhu, Jinghuan Wei, Junhong Zou, Zhen Lei, Zhaoxiang ZhangarXiv 2025 · DOI · arXiv

Introduction

Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities in general-purpose image understanding and reasoning. However, their performance significantly degrades when applied to scientific "field data"—high-dimensional matrices representing physical properties like velocity, pressure, or temperature across spatial domains. The authors identify two critical bottlenecks: the Data Gap, characterized by a lack of large-scale, high-quality scientific datasets with descriptive text, and the Token Limit, where the sheer density of physical data exceeds the input capacity of standard LLM decoders, leading to information loss.

Methodology

To address these limitations, the researchers introduced FieldLVLM, a framework that bridges the gap between natural and scientific vision through a two-pronged strategy.

Field-Aware Language Generation

To overcome the data shortage, FieldLVLM utilizes a pipeline where specialized expert models (e.g., for flow classification or vortex detection) act as "teachers." These experts analyze raw physical data to generate structured, highly accurate textual descriptions. These descriptions are then processed by a Large Language Model to create a consistent, natural-language dataset for training, effectively transferring domain-specific precision to the general-purpose model.

FieldLVLM framework utilizing expert models for data labeling and feature extraction.

Data-Compressed Multimodal Tuning

To handle the high dimensionality of field data, the framework treats physical fields as RGB images by mapping horizontal velocity ($u$), vertical velocity ($v$), and pressure ($p$) to the Red, Green, and Blue channels, respectively. A VQGAN (Vector Quantized Generative Adversarial Network) is then employed to compress these 256x256 "physics images" into just 256 discrete tokens—a 99.6% reduction in dimensionality. This allows the model to leverage powerful pre-trained visual encoders while staying within token limits. The final architecture is based on the Qwen2.5-VL backbone, fine-tuned using LoRA for efficiency.

Results

FieldLVLM was evaluated against state-of-the-art baselines, including LLaVA-v1.6 and Llama-3.2-Vision, using over 70,000 samples from the FlowBench and CFDBench datasets.

Comparison of caption quality and quantitative performance across different models.

The results demonstrate a stark contrast in performance. While general-purpose models often failed entirely (scoring 0/NA) on complex tasks like Reynolds number estimation, FieldLVLM achieved near-perfect accuracy across several domains:

  • Flow Categorization: 100% accuracy.
  • Reynolds Number Calculation: 99.79% accuracy (within 10% error tolerance).
  • Vortex Identification: 97.23% accuracy in detecting count, position, and rotation.

Qualitative analysis reveals that while standard models provide vague or non-committal descriptions, FieldLVLM identifies specific physical structures, such as Karman vortex streets and bifurcated stagnation points, grounding its reasoning in quantitative reality.

Conclusion

FieldLVLM represents a significant advancement in the application of LVLMs to scientific domains. By integrating expert-guided data generation with aggressive VQGAN-based compression, the framework enables precise physical reasoning that was previously unattainable for general-purpose models. Future research will focus on extending this methodology to 3D volumetric data and broader physical disciplines beyond fluid dynamics.

Summary updated

Abstract

Large Vision-Language Models (LVLMs) have shown impressive capabilities across a range of tasks that integrate visual and textual understanding, such as image captioning and visual question answering. These models are trained on large-scale image and video datasets paired with text, enabling them to bridge visual perception and natural language processing. However, their application to scientific domains, especially in interpreting complex field data commonly used in the natural sciences, remains underexplored. In this work, we introduce FieldLVLM, a novel framework designed to improve large vision-language models' understanding of field data. FieldLVLM consists of two main components: a field-aware language generation strategy and a data-compressed multimodal model tuning. The field-aware language generation strategy leverages a special-purpose machine learning pipeline to extract key physical features from field data, such as flow classification, Reynolds number, and vortex patterns. This information is then converted into structured textual descriptions that serve as a dataset. The data-compressed multimodal model tuning focuses on LVLMs with these generated datasets, using a data compression strategy to reduce the complexity of field inputs and retain only the most informative values. This ensures compatibility with the models language decoder and guides its learning more effectively. Experimental results on newly proposed benchmark datasets demonstrate that FieldLVLM significantly outperforms existing methods in tasks involving scientific field data. Our findings suggest that this approach opens up new possibilities for applying large vision-language models to scientific research, helping bridge the gap between large models and domain-specific discovery.
BibTeX
@misc{zhang2025improving,
  author = {Xiaomei Zhang and Hanyu Zheng and Xiangyu Zhu and Jinghuan Wei and Junhong Zou and Zhen Lei and Zhaoxiang Zhang},
  title = {{Improving Large Vision-Language Models' Understanding for Field Data}},
  year = {2025},
  eprint = {2507.18311},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.18311},
}
Figure