FG-GraphBench: Benchmarking and Enhancing Fine-Grained Visual Understanding with Multimodal Scene Graphs

Sicheng Liu, Xunkai Li, Ziyu Han, Xingnan Zhang, Ru Zhang, Rong-Hua LiarXiv 2026 · arXiv

TL;DR

FG-GraphBench introduces a high-fidelity benchmark and a novel integration method, FiGRA, to fix the "small object blindness" in Multimodal Large Language Models (MLLMs). By converting fine-grained scene graphs into continuous "graph tokens," the authors enable models to perceive tiny entities and complex relationships that standard vision encoders often overlook.

What Problem Does This Paper Address?

While MLLMs like LLaVA or GPT-4o are impressive, they suffer from a resolution gap: they often miss small objects and struggle to reason about the intricate relationships between them, leading to "hallucinations."

Scene graphs—which represent images as nodes (objects) and edges (relationships)—are a natural solution. However, the authors identify two major roadblocks:

  1. Data Scarcity: Existing scene graph benchmarks (like GQA) focus on salient, large objects, leaving fine-grained details unannotated.
  2. Integration Gap: We don't have a standard, effective way to "feed" these structured graphs into MLLMs without either losing structural information or overwhelming the model with long text descriptions.

Methods

The authors tackle these challenges with a two-pronged approach: a data construction pipeline and a new model architecture.

1. The FG-GraphBench Pipeline

To build a better benchmark, they developed a two-stage pipeline:

  • Stage 1 (Entity Discovery): They use a "Two-Round VLM Discovery" strategy. A VLM (Qwen3.7-Plus) scans the image to find sub-parts of known objects and missing independent micro-objects. These are then grounded using GroundingDINO and segmented with SAM.
  • Stage 2 (Relation Construction): This is the clever part. To find "dependency" relations (like a button on a shirt), they use Depth Anything V2 to create a depth map. They build a "Dependency Candidate Tree" where objects are linked based on spatial overlap and depth consistency.
Figure 1. Overview of the FG-GraphBench construction pipeline.

2. FiGRA (Fine-grained Graph Representation Augmentation)

FiGRA is the method for injecting these graphs into an MLLM.

  • Encoding: It uses a Graph Transformer to encode multimodal features (CLIP visual features + text labels) for every node and edge.
  • Graph Tokens: These embeddings are projected into the MLLM’s hidden space as continuous tokens, which are then interleaved with standard visual and text tokens.
  • Two-Stage Training: First, the model undergoes Pre-training on "graph-native" tasks like Shortest-Path Prediction (teaching the model to navigate the graph structure). Second, it undergoes Supervised Fine-Tuning (SFT) for specific downstream tasks.
Figure 2. Overview of FiGRA and its two-stage training strategy.

Datasets

The authors introduced FG-GraphBench, which significantly expands three existing datasets by adding fine-grained multimodal scene graphs (MSGs):

  • FG-QA: Built from GQA, focusing on relational reasoning.
  • FG-Caption: Built from MS COCO, for concise descriptions.
  • FG-Description: Built from Action Genome, for highly detailed image descriptions.

The scale of improvement is massive: in the FG-Description set, the average number of nodes per graph jumped from 3.75 to 16.19 (a 332% increase).

Key Insights and Ideas

  • Depth as a Structural Prior: Using depth maps to establish "part-of" hierarchies is a powerful way to automate scene graph construction. It moves beyond simple 2D bounding box overlaps which are often ambiguous.
  • The "Graph-Native" Pre-training: Simply giving a model a graph isn't enough. By forcing the model to predict the shortest path between nodes during pre-training, the authors "teach" the MLLM the underlying topology of the scene graph.
  • Tokens > Text: Linearizing a graph into a text string (e.g., "man has shirt, shirt has button") is computationally expensive and hits context window limits. Continuous graph tokens are more compact and preserve the structural dependencies better.

Key Results

FiGRA consistently outperformed both general-purpose MLLMs and specialized fine-grained vision methods.

  • FG-QA: FiGRA achieved 59.59% overall accuracy, outperforming the strongest baseline (PinPoint) by 9.32 percentage points.
  • Entity Recognition: On the FG-Description task, FiGRA showed a massive lead in "Entity-Acc," proving it actually "sees" the fine-grained objects it describes.
  • Ablation Success: Removing the Graph Encoder or the Pre-training stage led to significant performance drops, confirming that the structural modeling is doing the heavy lifting.
Effectiveness of FG-GraphBench compared to original graphs

Limitations and Future Work

  • Pipeline Dependency: The quality of the benchmark depends on the "teacher" models (Qwen, SAM, GroundingDINO). Errors in the initial discovery phase can propagate.
  • Inference Cost: While more efficient than text linearization, adding graph tokens still increases the sequence length and memory overhead of the MLLM.
  • Future Direction: The authors suggest moving toward end-to-end fine-grained scene graph generation, where the model learns to build the graph and reason over it simultaneously, rather than relying on a pre-constructed MSG.

Summary updated

Abstract

Fine-grained visual understanding requires multimodal large language models (MLLMs) to recognize small visual entities and understand their relationships. Scene graphs offer a natural way to represent this information, organizing visual entities as nodes and their relationships as edges. However, we identify two challenges: existing scene graph benchmarks provide insufficient coverage of fine-grained entities and the relationships among them, and existing methods for fine-grained visual understanding have yet to effectively leverage such graph-structured information in MLLMs. To address these challenges, we introduce Fine-Grained Multimodal Scene Graph Benchmark (FG-GraphBench) and FiGRA. FG-GraphBench is a benchmark comprising three complementary tasks: fine-grained visual question answering (FG-QA), image captioning (FG-Caption), and image description generation (FG-Description). We develop a two-stage construction pipeline that discovers and grounds fine-grained entities and establishes their relations to construct the fine-grained multimodal scene graphs (MSGs) in FG-GraphBench. Building on these MSGs, we propose FiGRA, a Fine-grained Graph Representation Augmentation method for MLLMs. FiGRA adopts a two-stage training strategy. It encodes multimodal entity and relation features as continuous graph tokens, and integrates them into the MLLM. Experiments show that FiGRA significantly outperforms strong baselines across three tasks of FG-GraphBench. Further analyses confirm the effectiveness of FG-GraphBench and highlight the importance of preserving fine-grained entity and relation information for visual understanding.
BibTeX
@misc{liu2026fg,
  author = {Sicheng Liu and Xunkai Li and Ziyu Han and Xingnan Zhang and Ru Zhang and Rong-Hua Li},
  title = {{FG-GraphBench: Benchmarking and Enhancing Fine-Grained Visual Understanding with Multimodal Scene Graphs}},
  year = {2026},
  eprint = {2611.25408},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25408},
}
Figure