TL;DR
This paper introduces F2TEval, a novel reference-free evaluation framework for Figure-to-Text (F2T) tasks that goes beyond simple similarity metrics to assess generated text across five human-aligned dimensions: faithfulness, completeness, conciseness, logicality, and analysis. They also release F2TBenchmark, a large-scale dataset for training and evaluating F2T models, demonstrating F2TEval's superior performance and efficiency compared to existing methods.
What Problem Does This Paper Address?
The Figure-to-Text (F2T) task, which involves converting structured figures into natural language descriptions, is crucial for making complex information accessible. However, evaluating the quality of these generated texts is challenging. Existing methods fall into two categories:
- Reference-based methods (e.g., BLEU, ROUGE) rely on human-written "golden" summaries. These are costly to create, especially for specialized domains, and often capture only shallow semantic similarities, missing factual errors or logical flaws.
- Reference-free methods (often using large language models or LLMs) are more flexible but can be sensitive to prompts, slow, and expensive due to reliance on large, often proprietary models. Furthermore, most existing methods provide a single "overall score," lacking the fine-grained, multi-dimensional assessment that humans naturally perform. This gap hinders interpretability and alignment with expert judgment.
The authors aim to bridge this gap by developing an evaluation method that is efficient, interpretable, and aligns with human expert criteria.
Methods
The core of F2TEval is a Mixture-of-Experts (MoE) architecture designed for multi-dimensional scoring. Key components include:
- Dimension-Specific Experts: For each of the five evaluation dimensions (Faithfulness, Completeness, Conciseness, Logicality, Analysis), a dedicated expert module is trained. These experts use a pre-trained CLIP encoder to process image and text inputs and a projection layer to generate a score. They are trained using a combination of Mean Squared Error (MSE) and a negative alignment correlation function to ensure accuracy and rank consistency.
- Shared Expert: A central expert module is trained to learn generalized scoring patterns across all dimensions. This module helps to correct and re-weight the scores from the dimension-specific experts, improving overall generalization.
- Hilbert-Schmidt Independence Criterion (HSIC): This novel optimization technique is applied to the shared expert. HSIC encourages statistical independence between the representations learned by different scoring heads. This "disentanglement" ensures that each expert focuses on distinct semantic signals, preventing redundancy and improving the interpretability of the multi-dimensional scores.
- Dynamic Weight Scoring: The final score for each dimension is a weighted combination of the dimension-specific expert's score and the shared expert's score, with learnable gating parameters.
Datasets
The authors introduce F2TBenchmark, a new, large-scale dataset for F2T evaluation.
- Collection: It aggregates data from 12 existing F2T datasets, covering 21 figure types and 35 domains.
- Generation: Figure summaries were generated by 10 different multimodal large language models (MLLMs) to capture a wide range of generation qualities and styles.
- Annotation: Each summary was manually annotated by 8 trained human annotators across the five F2TEval dimensions (Faithfulness, Completeness, Conciseness, Logicality, Analysis) on a 0-2 scale. This rigorous annotation process, including quality control and adjudication, ensures high-quality ground truth data.
Key Insights and Ideas
- Multi-dimensional evaluation is crucial: Simple similarity metrics (like BLEU/ROUGE) are insufficient for F2T tasks. Human evaluation naturally breaks down quality into multiple facets, and F2TEval mimics this by scoring across five distinct dimensions.
- MoE architecture for disentanglement: The MoE structure, combined with HSIC, is key to achieving independent scoring for each dimension. This prevents "gradient interference" and allows each expert to specialize, leading to more robust and interpretable evaluations.
- HSIC as a powerful regularization: HSIC's ability to measure statistical dependence in a reproducing kernel Hilbert space is more effective than traditional orthogonality constraints for promoting disentanglement in complex models like MoEs.
- Lightweight models can outperform giants: F2TEval, with a small parameter count (0.9B), achieves superior performance to much larger, state-of-the-art proprietary models like Gemini-2.0 and Claude-3.5, highlighting the effectiveness of their architectural design and evaluation strategy.
Key Results
- Superior Performance: F2TEval consistently outperforms all reference-based and reference-free baselines across all five evaluation dimensions and overall metrics (Pearson Correlation, Spearman Correlation, MAE, MSE). For instance, it achieved a Pearson Correlation of 0.7481 for Faithfulness, significantly higher than the next best (ChartX at 0.5965).
- Efficiency: F2TEval is remarkably efficient, with only 0.9B parameters and a fast inference time (31 seconds for the entire test set), making it over 50x faster than ChartX. This efficiency is attributed to its lightweight design and optimized architecture.
- Human Alignment: The model demonstrates strong alignment with human judgment, as evidenced by its high scores on metrics like Pearson Correlation and Spearman Correlation.
- Ablation Studies: The results confirm the importance of each component: the dimension-specific experts, the shared expert, and the HSIC optimization all contribute significantly to F2TEval's overall effectiveness.
Limitations and Future Work
- F2T Specificity: The current F2TEval model is specifically designed for F2T tasks.
- Future Directions: The authors plan to extend F2TEval to more complex multimodal tasks, such as multimodal multi-turn dialogue and multimodal chain-of-thought (MCOT) evaluation. This will likely require larger backbone models and potentially more advanced training techniques like reinforcement learning, which might trade some efficiency for improved performance.
Summary updated