Proceedings of the First Workshop of Evaluation of Multi-Modal Generation

Wei Emma Zhang, Xiang Dai, Desmond Elliott, Byron Fang, Mong Yuan Sim, Haojie Zhuang, Weitong ChenWorkshops @ COLING 2025

Introduction

The rapid advancement of multimodal AI—systems capable of processing and generating text, images, and video—has outpaced the development of rigorous frameworks to evaluate them. The inaugural EvalMG 2025 workshop proceedings address this critical gap, marking a transition from the "Wild West" of ad-hoc metrics toward standardized, culturally aware, and domain-specific evaluation methodologies. By convening researchers from both Natural Language Processing (NLP) and Computer Vision (CV), the workshop highlights seven key papers that redefine how we measure the faithfulness and utility of generative models.

The Multimodal Evaluation Crisis

Traditional metrics like BLEU or ROUGE, originally designed for text translation, are increasingly insufficient for capturing the nuance of multimodal understanding. Current evaluation faces several fundamental questions: How much does a specific modality (e.g., an image) actually contribute to a model's answer? Is the model truly "seeing" the input, or is it relying on linguistic biases? The EvalMG proceedings suggest that evaluation must move beyond simple accuracy to account for the complex interplay between visual and textual data.

Methodological Innovations

The proceedings showcase diverse approaches to improving both model efficiency and evaluation accuracy:

  • Architectural Efficiency: To address the computational burden of dense video captioning, researchers introduced CVT5. This model utilizes compressed video encoders paired with UMT5, demonstrating that high-quality temporal understanding can be achieved without prohibitive resource costs.
  • Automated Relevancy Judging: The LLaVA-RE framework proposes using Multimodal Large Language Models (MLLMs) as autonomous evaluators. By implementing a binary image-text relevancy system, LLaVA-RE can judge the alignment between visual and textual data, potentially reducing the reliance on expensive and slow human annotation.
  • Strategic Model Selection: Research by Sinha et al. provides a framework for "Guiding Vision-Language Model Selection." This logic helps practitioners navigate the crowded landscape of available models by matching specific tasks and domains with the models best suited to handle the required knowledge types.

Expanding the Benchmark Horizon

A significant portion of the workshop focused on creating "real-world" benchmarks that move beyond sanitized, Western-centric datasets. These new resources test models in specialized and underrepresented contexts:

  • Cultural Grounding: The TaiwanVQA benchmark focuses on Visual Question Answering related to Taiwanese daily life, while a new suite of Persian Multi-Modal Tasks evaluates Vision-Language Models (VLMs) in the Persian language. These benchmarks reveal that state-of-the-art models often suffer significant performance drops when moved outside of English-centric contexts.
  • Specialized Domains: A new dataset for Programming-based Instructional Videos challenges models to classify and answer questions about technical, code-heavy content, a task that requires high precision and domain-specific logic.
  • Modality Utility: New datasets for Complementary Information Tasks specifically test whether a model can synthesize unique information found only in an image with unique information found only in text, forcing the model to prove it is utilizing both modalities effectively.

Key Insights and Results

The workshop's selective process (accepting only 7 out of 21 submissions) underscored several emerging truths in the field:

  1. The Utility of Individual Modalities: A recurring theme is the need to verify that models are not ignoring visual inputs in favor of text-based shortcuts. True multimodal AI must demonstrate that the visual component is essential to the output.
  2. Repurposing Models as Evaluators: Results from LLaVA-RE suggest that the same models being evaluated can, if prompted correctly, serve as effective judges for image-text alignment, offering a path toward scalable evaluation.
  3. The Performance Gap: Benchmarks in Persian and Taiwanese contexts provide a necessary "report card" for global models like GPT-4o, highlighting that "general" intelligence is often geographically and linguistically limited.

Limitations and Future Directions

Despite these advancements, the community faces ongoing hurdles. Many proposed evaluation methods remain computationally expensive or dependent on closed-source API models, which limits transparency. Furthermore, creating robust metrics for "fairness" and "faithfulness" remains an open challenge.

The EvalMG 2025 proceedings conclude that evaluation should no longer be treated as a secondary step in research papers. Instead, it must become a standalone discipline focused on creating robust, transparent, and culturally inclusive methodologies that can keep pace with the generative explosion.

Summary updated

BibTeX
@inproceedings{zhang2025proceedings,
  author = {Wei Emma Zhang and Xiang Dai and Desmond Elliott and Byron Fang and Mong Yuan Sim and Haojie Zhuang and Weitong Chen},
  title = {{Proceedings of the First Workshop of Evaluation of Multi-Modal Generation}},
  booktitle = {Workshops @ COLING},
  year = {2025},
}
Figure