EgoFact: A Benchmark for Multi-Hop Multimodal Retrieval-Augmented Generation

Ping Zhang, Abinaya Kumar, Shabnam Ghaffarzadegan, Wei-Lun Chao, Chaithanya Kumar MummadiOpenReview 2026 · Submission to ICLR 2026 · Withdrawn · OpenReview

Its authors withdrew this paper, so there is no full text to read or summarize.

Abstract

Retrieval-Augmented Generation (RAG) has emerged as a powerful approach to improve large language models (LLMs) by grounding their outputs in external knowledge. However, progress in the multimodal domain remains limited, largely due to the lack of suitable benchmarks. Existing multimodal corpora are often built by merging unimodal datasets, which rarely support queries requiring multi-hop reasoning and thus reduce most tasks to single-modality, one-hop retrieval. To address this gap, we introduce $\textit{EgoFact}$, the first benchmark explicitly designed for multi-hop reasoning across visual and textual corpora. Success on $\textit{EgoFact}$ requires models to retrieve and integrate evidence spanning multiple modalities. We systematically evaluate existing RAG systems and uncover fundamental limitations in multimodal evidence integration and reasoning. Motivated by these findings, we propose a localization-first framework for cross-modal video reasoning that enables more precise evidence grounding and substantially improves reasoning accuracy. Extensive experiments demonstrate the effectiveness of our approach, establishing new state-of-the-art results on multimodal RAG tasks. Together, the benchmark and framework lay a foundation for advancing research in this emerging area and for building more reliable multimodal reasoning systems.
BibTeX
@misc{zhang2026egofact,
  author = {Ping Zhang and Abinaya Kumar and Shabnam Ghaffarzadegan and Wei-Lun Chao and Chaithanya Kumar Mummadi},
  title = {{EgoFact: A Benchmark for Multi-Hop Multimodal Retrieval-Augmented Generation}},
  year = {2026},
  url = {https://openreview.net/forum?id=6WS1wfLmc6},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2026},
}

Loading related content...

Figure