TL;DR
FG-GraphBench introduces a high-fidelity benchmark and a novel integration method, FiGRA, to fix the "small object blindness" in Multimodal Large Language Models (MLLMs). By converting fine-grained scene graphs into continuous "graph tokens," the authors enable models to perceive tiny entities and complex relationships that standard vision encoders often overlook.
What Problem Does This Paper Address?
While MLLMs like LLaVA or GPT-4o are impressive, they suffer from a resolution gap: they often miss small objects and struggle to reason about the intricate relationships between them, leading to "hallucinations."
Scene graphs—which represent images as nodes (objects) and edges (relationships)—are a natural solution. However, the authors identify two major roadblocks:
- Data Scarcity: Existing scene graph benchmarks (like GQA) focus on salient, large objects, leaving fine-grained details unannotated.
- Integration Gap: We don't have a standard, effective way to "feed" these structured graphs into MLLMs without either losing structural information or overwhelming the model with long text descriptions.
Methods
The authors tackle these challenges with a two-pronged approach: a data construction pipeline and a new model architecture.
1. The FG-GraphBench Pipeline
To build a better benchmark, they developed a two-stage pipeline:
- Stage 1 (Entity Discovery): They use a "Two-Round VLM Discovery" strategy. A VLM (Qwen3.7-Plus) scans the image to find sub-parts of known objects and missing independent micro-objects. These are then grounded using GroundingDINO and segmented with SAM.
- Stage 2 (Relation Construction): This is the clever part. To find "dependency" relations (like a button on a shirt), they use Depth Anything V2 to create a depth map. They build a "Dependency Candidate Tree" where objects are linked based on spatial overlap and depth consistency.
2. FiGRA (Fine-grained Graph Representation Augmentation)
FiGRA is the method for injecting these graphs into an MLLM.
- Encoding: It uses a Graph Transformer to encode multimodal features (CLIP visual features + text labels) for every node and edge.
- Graph Tokens: These embeddings are projected into the MLLM’s hidden space as continuous tokens, which are then interleaved with standard visual and text tokens.
- Two-Stage Training: First, the model undergoes Pre-training on "graph-native" tasks like Shortest-Path Prediction (teaching the model to navigate the graph structure). Second, it undergoes Supervised Fine-Tuning (SFT) for specific downstream tasks.
Datasets
The authors introduced FG-GraphBench, which significantly expands three existing datasets by adding fine-grained multimodal scene graphs (MSGs):
- FG-QA: Built from GQA, focusing on relational reasoning.
- FG-Caption: Built from MS COCO, for concise descriptions.
- FG-Description: Built from Action Genome, for highly detailed image descriptions.
The scale of improvement is massive: in the FG-Description set, the average number of nodes per graph jumped from 3.75 to 16.19 (a 332% increase).
Key Insights and Ideas
- Depth as a Structural Prior: Using depth maps to establish "part-of" hierarchies is a powerful way to automate scene graph construction. It moves beyond simple 2D bounding box overlaps which are often ambiguous.
- The "Graph-Native" Pre-training: Simply giving a model a graph isn't enough. By forcing the model to predict the shortest path between nodes during pre-training, the authors "teach" the MLLM the underlying topology of the scene graph.
- Tokens > Text: Linearizing a graph into a text string (e.g., "man has shirt, shirt has button") is computationally expensive and hits context window limits. Continuous graph tokens are more compact and preserve the structural dependencies better.
Key Results
FiGRA consistently outperformed both general-purpose MLLMs and specialized fine-grained vision methods.
- FG-QA: FiGRA achieved 59.59% overall accuracy, outperforming the strongest baseline (PinPoint) by 9.32 percentage points.
- Entity Recognition: On the FG-Description task, FiGRA showed a massive lead in "Entity-Acc," proving it actually "sees" the fine-grained objects it describes.
- Ablation Success: Removing the Graph Encoder or the Pre-training stage led to significant performance drops, confirming that the structural modeling is doing the heavy lifting.
Limitations and Future Work
- Pipeline Dependency: The quality of the benchmark depends on the "teacher" models (Qwen, SAM, GroundingDINO). Errors in the initial discovery phase can propagate.
- Inference Cost: While more efficient than text linearization, adding graph tokens still increases the sequence length and memory overhead of the MLLM.
- Future Direction: The authors suggest moving toward end-to-end fine-grained scene graph generation, where the model learns to build the graph and reason over it simultaneously, rather than relying on a pre-constructed MSG.
Summary updated