TL;DR
This paper introduces MaGiX, a novel framework that enhances Retrieval-Augmented Generation (RAG) for cross-lingual question answering, specifically between English and Vietnamese. By building a multi-granular knowledge graph and employing a sophisticated semantic-aware reranking strategy, MaGiX significantly improves retrieval accuracy and generation quality compared to existing methods.
What Problem Does This Paper Address?
Large Language Models (LLMs) are powerful, but they often struggle with integrating external knowledge, especially in multilingual or low-resource settings. Retrieval-Augmented Generation (RAG) aims to solve this by grounding LLM outputs in retrieved documents. While Graph-based RAG (GRAG) frameworks have shown promise by leveraging knowledge graphs (KGs) for better reasoning and coherence, they have been underexplored in cross-lingual scenarios. Existing GRAG methods also face limitations in:
- Cross-lingual applicability: Most GRAG systems are designed for monolingual tasks.
- Semantic granularity: They often use simplified representations of entities and relations, losing important contextual details.
- Entity alignment: Accurately matching entities across different languages is challenging, especially when surface forms differ.
- Retrieval bias: Traditional GRAGs can overemphasize structural connectivity, potentially missing semantically relevant but less connected information.
This paper tackles these issues by proposing MaGiX, a framework specifically designed for English-Vietnamese cross-lingual question answering.
Methods
MaGiX introduces several key innovations:
- Multi-Granular Cross-Lingual Knowledge Graph Construction:
- Fine-grained attribute descriptions: Instead of just entity names, MaGiX incorporates detailed attribute descriptions extracted from text chunks.
- Cross-synonym edges: It explicitly links equivalent entities across English and Vietnamese using a custom multilingual embedding model trained with contrastive learning. This goes beyond simple name matching and leverages contextual information.
- Custom Multilingual Embedding Model: A fine-tuned model using contrastive learning (SimCSE) and a multi-stage training strategy (domain adaptation and supervised cross-lingual alignment) is developed to capture domain-specific semantics and align embeddings across languages.
- Semantic-Aware Reranking Strategy:
- Granular Graph Reasoning: MaGiX considers all contextual descriptions for entities and relations, improving text chunk localization.
- Cross-Synonym Traversal: It leverages the cross-synonym edges to explore related entities in different languages.
- Multi-Aspect Scoring: A composite score combines chunk similarity, attribute similarity, and triple similarity, with a "Cross-Lingual Score Boosting" mechanism that prioritizes synonymous entities across languages.
Datasets
The authors evaluated MaGiX on several datasets:
- Simple QA:
- ZaloWikipediaQA (Vietnamese)
- ZaloLegal2021 (Vietnamese)
- NaturalQuestions (NQ) (English)
- PopQA (English)
- Multi-Hop QA:
- MuSiQue (English)
For cross-lingual evaluation, they translated the English datasets into Vietnamese and vice-versa using Gemini 2.0 Flash.
Key Insights and Ideas
- The power of fine-grained semantics: Simply concatenating or summarizing entity descriptions (as in some prior GRAGs) loses crucial detail. MaGiX's approach of using multiple, fine-grained attribute descriptions per entity significantly boosts retrieval performance.
- Cross-lingual alignment is key: Explicitly modeling cross-synonym relationships between entities in different languages, rather than relying solely on general-purpose multilingual embeddings, is vital for effective cross-lingual retrieval.
- A balanced scoring mechanism is crucial: Combining multiple semantic signals (chunk, attribute, and triple similarity) with a cross-lingual boosting mechanism leads to more robust and accurate ranking, avoiding over-reliance on any single factor.
- GRAG has untapped potential in low-resource cross-lingual settings: This work demonstrates that GRAG can be successfully adapted and improved for languages like Vietnamese, opening doors for similar research in other low-resource language pairs.
Key Results
- Retrieval Performance: MaGiX consistently outperformed prior GRAG systems (GraphRAG, LightRAG, HippoRAG 2) and dense retrieval baselines (BGE-M3) across all tested cross-lingual QA datasets. For example, on ZaloWikipediaQA (en-vi), MaGiX achieved 85.45%, outperforming HippoRAG 2 and BGE-M3 by nearly 10%. On the multi-hop MuSiQue dataset, MaGiX achieved scores of 60.37% (vi-en) and 58.43% (en-vi), significantly outperforming the second-best baseline.
- Generation Quality: MaGiX also demonstrated substantial improvements in generation quality, achieving high win rates across various evaluation dimensions (Accuracy, Comprehensiveness, Diversity, Logicality, Coherence) in LLM-based judgments.
- Efficiency: Despite its advanced representation, MaGiX achieved competitive inference times (3.77s on NQ) and graph construction times, striking a good balance between performance and scalability.
Limitations and Future Work
- Language Pair Specificity: The current evaluation is limited to English-Vietnamese. The authors acknowledge that the generalizability to other languages, especially those with significantly different morphological structures or weaker translation models, needs further investigation.
- Synthetic Data Reliance: While synthetic data and bilingual embeddings are effective, their performance might vary across domains or languages with less robust translation models.
- Future Directions: The authors suggest broader multilingual evaluation and further refinement of domain adaptation strategies as future work. They also highlight that while MaGiX excels, challenges remain for queries involving multiple dispersed entities where evidence is sparse and split across document chunks.
Summary updated